The two types of LLM preferences (newsletter.danielpaleka.com)

Researchers tested LLM “preferences” by posing binary-choice tasks (do whichever you prefer) and statistically analyzing which task the model selects. They found results are often not robust: small, reasonable prompt variations—reformatting with XML tags, changing wording or task order—can flip outcomes. Citing prior work (Khan et al., 2025), the author frames this as two qualitatively different kinds of preferences. “Weak” preferences are statistical tendencies (like Alice preferring chocolate) that appear when averaging many trials but shift under alternate phrasings or contexts. “Strong” preferences are stable, resist reasonable prompt changes, and usually reflect post-training safety or behavioral constraints (e.g., refusal to produce NSFW or harmful content, analogous to Bob’s vegetarianism). The distinction matters for alignment and evaluation: most binary-choice experiments will reveal weak, prompt-sensitive trends unless evaluators explicitly test robustness across transformations, shuffles, and formatting. Strong preferences—products of targeted fine-tuning or safety training—behave differently and are harder to overturn at deployment. Practical takeaways: don’t overinterpret single-prompt results, report robustness to prompt transformations, and design benchmarks that separate statistical tendencies from durable, training-embedded behaviors.