ChatGPT/Gemini can now draw on your screen to help you navigate complex software (sketchvlm.github.io)

A groundbreaking development has emerged in the realm of vision-language models (VLMs) with the introduction of SketchVLM, a new framework that allows models like Gemini-3-Pro and GPT-5 to generate editable SVG overlays directly on images. This innovation addresses a critical gap in current VLM capabilities, as traditional models typically respond to visual queries solely with text, making it challenging for users to verify their answers. SketchVLM enables these models to visually explain their reasoning by producing clear, annotated sketches, enhancing user interaction and understanding. This advancement is significant for the AI/ML community as it enhances the accuracy of visual reasoning tasks—such as maze navigation and object counting—by an impressive 28.5 points. Additionally, the quality of the generated sketches has improved by 48.3% compared to previous image-editing and fine-tuned approaches. The framework's ability to perform single-turn sketch generation with strong accuracy suggests immediate applicability, while the potential of multi-turn interactions could foster deeper human-AI collaboration. Overall, SketchVLM marks a notable leap forward in making AI visualizations more intuitive and reliable.