Building an vision language model from scratch (poonai.xyz)

A developer built a toy vision-language model (VLM) that generates image captions by combining off-the-shelf components instead of training everything from scratch. They used a ViT image encoder (512-dim output) and a GPT-2 text generator (expects 768-dim token embeddings), and trained a lightweight Projection module to map vision embeddings into GPT-2’s embedding space. The Projection is a small MLP (Linear → GELU → Linear) that expands the 512 features to 512*3 then projects to 768. The system was trained on an image–caption dataset and produces captions such as “a boy holding a fish in the woods.” This is a clear, practical demonstration of a common VLM pattern: reuse pretrained unimodal backbones and learn a cross-modal connector. It’s significant because it shows how a simple projection layer can align visual and textual embedding spaces, enabling a language model to decode visual semantics without full multimodal pretraining. For practitioners this approach offers a fast, low-cost path for prototyping captioning or document-processing VLMs; limitations include dataset scale, evaluation rigor, and likely performance gaps vs. large joint-trained models. The project provides intuition and code that can be extended toward more ambitious document VLMs or fine-tuning strategies to approach SOTA.