If the LLM Is Stuck, Ask It to Write a Diagnosis Script (davepotts.software)

When an LLM-powered assistant like GitHub Copilot stalls trying to directly fix a bug, a more productive pattern is to ask it to generate a standalone diagnostic script. The script should be small, easy to run, and systematically test different configurations (e.g., network options, auth tokens, HTTP headers, retries). It must produce detailed, structured output about what succeeds and fails so you get concrete facts instead of speculative edits. In practice this looks like a script that iterates header combinations, logs responses and status codes, and isolates the failing condition — for example, a diagnose_403.py that tried multiple header permutations and revealed missing headers were causing 403 responses. This approach matters for AI/ML engineers because it turns the LLM from a guess-and-patch tool into an automation-driven investigator: diagnostic scripts create reproducible evidence, speed up root-cause analysis, and serve as documentation of the investigation. Technically, the pattern encourages building small test harnesses, deterministic checks, and verbose logging that feed back into model prompts or CI pipelines, reducing blind trial-and-error and enabling faster, more reliable fixes in data pipelines, scrapers, model infra, and other systems.