Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation (arxiv.org)

Recent research highlights that execution feedback is more critical than pipeline topology when optimizing code generation tasks in small language models (1-3B parameters). The study examines various code generation pipelines and finds that applying execution feedback significantly enhances performance – with self-refinement methods yielding improvements greater than four standard deviations on benchmarks like HumanEval and sanitized MBPP. Interestingly, the research shows that while refinement with execution feedback effectively resolves many common runtime errors, it struggles to address logic errors, indicating a gap in current capabilities. Significantly, the results suggest that the identity of the models used in the pipeline is less important than their ability to execute and refine code. A pairing of a 1.5B generator with a 3B refiner performed on par with a single 3B model handling both tasks. The findings challenge conventional approaches to model architecture, asserting that model specialization surpasses the potential benefits of more complex pipeline structures. This shift could lead AI/ML researchers to prioritize execution feedback mechanisms over intricate model compositions, ultimately improving code generation efficiency and accuracy in practical applications.