LLM-generated skills work, if you generate them afterwards (www.seangoedecke.com)

A recent study highlights a crucial insight into the effectiveness of "skills" generated by large language models (LLMs) for task execution. Researchers found that LLM-authored skills, which serve as procedural knowledge prompts for tasks, are not beneficial when generated before the task is attempted. This suggests that LLMs have difficulty producing useful procedural guidelines without prior experience with the problem. Instead, the paper advocates for a new approach, where LLMs should generate skills after completing the task, allowing them to distill the valuable insights gained through iterative problem-solving rather than relying solely on pre-existing knowledge. This finding is significant for the AI/ML community as it challenges current prompting strategies that encourage LLMs to "think step by step" before task execution. By emphasizing the importance of post-task skill generation, the research encourages developers to optimize LLM applications in real-world scenarios where iterative learning can lead to better outcomes. Additionally, the study hints at the broader implications for LLM training and usage, suggesting that current reasoning models could substantially benefit from structured reflection on the complexities they encounter during task execution.