nanoVLM: The simplest repository to train your VLM in pure PyTorch

Source: HuggingFace Date: 2025-05-21 URL: https://huggingface.co/blog/nanovlm

Summary

Library release: nanoVLM, a minimal pure-PyTorch VLM training repo in the nanoGPT tradition. 222M parameters (SigLIP-base + SmolLM2-135M + modality projector), trains on free-tier Colab, pretrained checkpoint available after ~6 hours on one H100 on 1.7M Cauldron samples. Intentionally educational rather than SOTA — demystifies VLM architecture.

Implications

Thread: open-weights ecosystem health. nanoGPT-style minimal implementations have outsized pedagogical impact relative to their benchmark scores. nanoVLM puts VLM training within reach of anyone with Colab access, which accelerates the community’s ability to experiment with multimodal architectures. The SigLIP + SmolLM2 component choice is interesting as a minimal stack — both are HF-native models, making the whole pipeline a self-contained HF ecosystem demonstration. Educational tools like this expand the pool of researchers who can iterate on multimodal architecture ideas without needing a cluster.