Teaching Clip, Whisper, and Gemini to Trust Nothing (mixpeek.com)

FakeCheck is a proof-of-concept "trust-nothing" deepfake detector that caught a viral kangaroo video (generated by Google’s Veo3) by fusing off‑the‑shelf models and heuristics instead of training a monolith. The pipeline uses CLIP with prompt-engineered semantic similarity to flag visual oddities, Whisper for transcription (with a safety "no_speech_prob" cutoff to avoid hallucination cascades), and Google’s Gemini LLM to run artifact, lip‑sync, blink and OCR checks. Additional detectors (optical-flow spikes, motion throttling, etc.) feed a hand‑tuned fusion score—CLIP, Gemini visual artifacts, lip‑sync, blink and OCR are weighted to produce a final LIKELY_FAKE verdict. The author also documents practical engineering workarounds (Gemini prompt “load‑bearing” injection, a protobuf async/sync fallback) and reports ~70% accuracy on short clips, ~30s processing per 30s video on GPU, and a current 30‑second clip length limit. For the AI/ML community this is a useful, pragmatic demonstration: multimodal ensembles of pretrained models can detect fakes more robustly than single models, but are brittle in predictable ways—LLM hallucinations, SDK bugs, hand‑tuned fusion, and scaling limits. The project highlights key research and production needs: reliable hallucination detection, automated fusion weight optimization, larger labeled corpora, model versioning, and distributed pipelines for longer or live streams. FakeCheck’s repo and call for contributions make it a practical starting point for improving detection pipelines as the deepfake arms race continues.