The Case for Separating Thinking (GPU) and Compute (CPU) (www.gojiberries.io)

Researchers propose redesigning LLM runtimes to separate "Thinking" (model-internal work) from "Acting" (external I/O and tooling) and "Evaluating" (checks that draw on both). Instead of a single opaque process, user requests would be a short policy (objectives, limits, preferences) and the runtime would generate a plan that allocates discrete budgets: token budgets for internal reasoning and judge-model passes (Think), CPU/IO/network budgets and request counts for retrieval or tool runs (Act), and a bounded number of evaluation/edit cycles (Evaluate). Execution proceeds in multiple turns with checkpoints to re-estimate remaining spend and clear telemetry (tokens used, external requests, CPU seconds, egress, latency, eval cycles). Users can express splits (e.g., Explore‑heavy, Evaluate‑heavy) or set macro budgets/deadlines; the planner enforces limits and stops when marginal value is low. Significance: this control surface makes costs and quality trade-offs legible and tunable, letting teams optimize where to invest compute (GPU vs CPU) for different tasks. Instrumentation yields per-request datasets—task tags, allocation, realized spend, and outcome signals—that let systems learn spend‑vs‑quality curves by task family and phase (e.g., code synthesis favors Evaluate, literature reviews favor Act). The result is predictable billing, grounded defaults, accountable planning, and better workflow design for research and production ML systems.