Grep beats LSP? Why coding agents ignore your fancier tools (www.agentconnect.md)

A recent study highlights the surprising preference of coding agents for grep over LSP-backed semantic navigation in code retrieval tasks. The research explored the effectiveness of these two tools in terms of code finding and editing, revealing that while semantic navigation produced more precise results in some scenarios, agents favored grep 94-100% of the time in simpler tasks. The critical insight is that LLM-friendliness—how well tools integrate with the workflow and provide contextual information—may outweigh pure accuracy. Semantic tools often returned only limited data, necessitating further actions from the model, which reduced efficiency. The implications for the AI/ML community are significant, particularly for those developing coding agents. This study underscores the importance of considering the entire agent operational framework, or "harness," when evaluating new tools. It demonstrated that understanding how tools efficiently deliver context—rather than just measuring their precision—can greatly influence the overall performance of coding tasks. These findings suggest that adopting LSP or other sophisticated tools should be accompanied by thorough testing in real-world scenarios, ensuring that development maintains user-friendly interfaces that align with agent capabilities.