Compiling an SOP into a 3B Model (arpitbhayani.me)

A recent experiment demonstrated the potential of fine-tuning a 3-billion-parameter model to manage multi-turn workflows without relying on extensive prompt instructions. By converting a 3,500-token Standard Operating Procedure (SOP) for incident triage into the model's weights, the fine-tuned model achieved a remarkable 93.0% end-to-end accuracy on test scenarios, outperforming Gemini 2.5 Pro, which scored 92.0%. The approach drastically reduced input tokens from 5,200 to just 450 per incident, while cutting operational costs by 8.4 times, from $1.85 to $0.22 per 1,000 incidents. This experiment is significant for the AI/ML community as it showcases the feasibility of compiling complex, branching decision trees directly into model weights, eliminating the need for continuous prompt updates. The technical innovation involved a deterministic simulator for training data and precise loss masking during model fine-tuning to focus only on the assistant's actions. The results suggest that smaller models can effectively handle nuanced workflows, offering a high-speed, cost-effective alternative to traditional, prompt-driven setups, while also simplifying maintenance compared to changing prompts as procedural rules evolve.