Thought Engineering (pranavc28.github.io)

A new blog introduces "thought engineering" and a concrete technique called Automated Confidence Refinement: prompting LLMs to report numerical, natural-language confidence (0.0–1.0) at intermediate reasoning steps and using those scores to decide when to fetch more context or default to a conservative label. The author argues this metacognitive layer — treating confidence as a first‑class signal — improves multi‑class classification (especially detecting NOINFO) across model families by revealing where a chain of thought is weak and triggering targeted retrieval instead of blind aggregation or costly overthinking. Technically, the study compares three strategies on 200 SciFact claims (all NOINFO ground truth) against a 5,000+ abstract corpus using a shared term‑overlap retriever: naive query generation, "overthinking" pre‑reasoning that shapes search breadth by initial confidence, and the proposed two‑stage Automated Confidence Refinement that retrieves, asks the model to assess sufficiency, and conditionally issues refined queries when confidence < model‑specific thresholds (example threshold: 0.7). Primary evaluation is NOINFO F1, since converting low‑confidence SUPPORT/CONTRADICT to NOINFO both measures calibration and prevents overconfident errors. The approach promises better trust and efficiency by localizing failures (e.g., low confidence at a specific reasoning step) and only expending extra retrieval or prompting when the model recognizes gaps, extending recent metacognition and selective‑retrieval work.