Moondream 3 Preview: Frontier-level reasoning at a blazing speed (moondream.ai)

Moondream announced a preview of Moondream 3, a new visual-language model architecture that packs 9B parameters in a sparse Mixture-of-Experts (MoE) design but uses only ~2B active parameters at inference. The team positions it as a frontier-capable visual reasoner that remains fast and inexpensive: it extends context length from 2k to 32k tokens, improves OCR, supports native pointing and structured (JSON) outputs, and demonstrates strong object-detection and grounding behavior across real-world prompts. The release targets four priorities—visual reasoning, trainability, latency, and cost—so developers can fine-tune or RL-post-train the model for applied tasks like inspection, robotics, or high-volume image pipelines. Technically, Moondream 3 is a fine-grained sparse MoE with 64 experts and 8 experts activated per token, initialized from a 2B dense model via “drop upcycling.” Long-context capability was developed by interleaving long-context samples during pretraining and adding learned position-dependent attention temperature scaling, improving long-range modeling without a separate context-extension phase. Training used load-balancing and router orthogonality losses (then disabled in post-training) and attention tweaks such as LSE suppression; reinforcement learning in post-training significantly boosted grounded visual reasoning. Caveats: inference code isn’t fully optimized yet (slower than expected), benchmarks are preliminary, and the team plans quantized/distilled variants and more thorough latency-inclusive evaluations.