Show HN: Jevcrawl – A web crawler for AI agents with no LLM in the crawl loop (github.com)

Jevcrawl is a newly announced web crawler specifically designed for AI agents that do not rely on a large language model (LLM) in the crawling process. Instead of the LLM reading every fetched page—which can lead to inefficiencies and high costs—Jevcrawl employs a small decision model to select pertinent links and passages. This approach has shown to be significantly more effective, achieving a 54% success rate in finding answer pages compared to only 33% with Claude's LLM, and is approximately 8 times cheaper per task. The technical implications of Jevcrawl are noteworthy for the AI/ML community. It operates at a fraction of the cost—around a cent per crawl—and is twice as fast as traditional LLM-driven crawling methods. Importantly, it retrieves verbatim content with citations, mitigating issues related to misinformation since the decision model only scores text and does not generate it. By working seamlessly with multiple interfaces and providing precise cost tracking, Jevcrawl represents a significant leap in the efficiency of web crawling for AI applications, enhancing both speed and accuracy while remaining cost-effective.