Prompt Fiddling Considered Harmful (tomsilver.github.io)

The article discusses the risks associated with "prompt fiddling" in machine learning, particularly in the realm of natural language processing. Researchers often manipulate test set prompts in a bid to improve model performance, a practice akin to hyperparameter optimization that leads to 'leakage'—where insights from the test set impact the model's training unfairly. Unlike traditional hyperparameters, prompt manipulation is a free-form endeavor, making it more challenging to identify when and how much fiddling has occurred. This undermines the integrity of results reported in academic research. To combat these challenges, the author advocates for a renewed adherence to established research practices: using validation sets instead of test sets for prompt optimization, documenting the optimization processes transparently, and participating in benchmarks where test sets remain hidden. Emphasizing the importance of robust methodologies, the article encourages researchers to shift toward constructing prompts with stable performance across semantically similar variations, rather than relying on luck or specific phrasing. Overall, the call is for the AI/ML community to uphold the foundational rule of keeping test sets "locked away" to ensure genuine model evaluation and avoid the pitfall of false progress through prompt manipulation.