Agents turn simple keyword search into compelling search experiences (softwaredoug.com)

A developer experimented with replacing a traditional “thick” RAG search stack with a deliberately simple keyword search (BM25 on product name/description with basic snowball tokenization) and explicitly told an LLM exactly how that tool behaves by embedding the tool docstring in the prompt. The agent used its reasoning to iteratively reformulate queries (e.g., “vampire couch,” “ugliest chair”), issued multiple targeted searches, and judged results itself—saving each tool query, a quality label and reasoning. Semantic caching / vector lookups over past queries let the agent recall which expansions worked (similarity scores used to match past queries) so its query strategy improved over time, producing surprisingly relevant, human-friendly results despite the dumb backend. This approach highlights a trade-off: agents can compensate for simple retrieval backends by planning, reformulating, and learning a “mental model” of the search tool, turning sparse keyword search into compelling experiences. Key technical implications: docstrings-as-prompts, LLM-as-judge for per-call evaluation, stored interaction logs for semantic caching, and vector similarity for query reuse. Important caveats remain—agents need rich metadata to evaluate results, must possess domain judgment, and can diverge from real user preferences since clickstream engagement signals are noisy and often unavailable. Consequently, some ranking optimizations (engagement-driven signals) may be safer kept outside agents, lest an LLM discard results that actually perform best for users.