What if your AI looked at an image, answered, then doubted itself? (github.com)

Microsoft's MMCTAgent is an open-source, research-focused multimodal framework (NeurIPS OWA-2024 paper + GitHub repo) that brings human-like critical thinking to image and video reasoning. It introduces a planner–critic loop: the Planner composes tool-driven analyses (object detection, OCR, recognition, vision LLMs) to form answers, and a vision-based Critic evaluates and refines those answers against task-specific criteria. The result is iterative self-reflection and verification that improves accuracy and robustness on complex visual queries, with example ImageAgent and VideoAgent workflows and ready-to-run Python quickstarts. Technically, MMCTAgent is modular and vendor-agnostic: it plugs in multimodal LLMs, CLIP embeddings for frame similarity, transcription services (Whisper/Azure), search backends (Azure AI Search/FAISS), and storage providers without code changes. The VideoAgent uses a fixed toolchain (GET_VIDEO_ANALYSIS, GET_CONTEXT, GET_RELEVANT_FRAMES, QUERY_FRAME) to retrieve relevant videos, transcripts, keyframes and detailed frame queries; the ImageAgent exposes configurable ImageQnaTools (object_detection, ocr, recog, vit) and a use_critic_agent flag. The repo includes ingestion, indexing and FASTAPI components, hardware recommendations for CLIP/PyTorch (GPU/mixed precision advised), and an MIT license—making it practical for research and prototyping of more reliable, tool-enabled multimodal reasoning systems.