Benchmark for Agent Context Engineering (2025) (www.tarasyarema.com)

Author released a practical benchmark comparing three agent context-engineering strategies—raw append-only history, periodic summarization, and a deterministic “intent” prompt compressor—on a non-trivial multi-step analytics task (NYC taxi parquet data, DuckDB SQL, file read/write/update tools). The agents were tested across several modern models (Anthropic Claude Sonnet 4.5, OpenAI GPT-4.1 and GPT-4.1-mini, Google Gemini 2.5 Pro) with identical tools and iteration limits. Evaluation tracked success, accuracy, context/token usage, latencies and cost; full code and report are on GitHub. Key finding: the intent agent (a single system prompt that deterministically compresses history with dynamic placeholders) achieved 100% success across models and consistently higher accuracy, using far fewer steps (<25) at the expense of modestly higher latency and similar or slightly higher cost than summarization. The raw agent consumed the most context and was noisier but cheaper; summarization reduced token use but sometimes lost critical analytical detail and had poorer reliability. Takeaway for builders: orchestration frameworks (LangChain, OpenAI/Google agent stacks) matter less than mastering context engineering—design deterministic, task-aware context compression to improve reliability for complex agentic workflows, accepting small latency/cost tradeoffs for consistent deliverables.