Kimi K2 RL'ed Qualitative Data to Write Better (www.dbreunig.com)

Moonshot applied reinforcement learning to Kimi K2 specifically to boost qualitative writing and conversational skills—areas where LLMs have lagged because outcomes are hard to score. Rather than relying on brittle end-to-end LLM evaluation or expensive human labeling alone, they bootstrapped a critic by mixing open-source and in-house preference data, then had Kimi instances generate and pairwise-score responses against a small, explicit rubric set. The rubrics include a Core rubric (clarity/relevance, conversational fluency/engagement, objective/grounded interaction), a Prescriptive rubric to block reward hacking (e.g., ban opening praise and “explicit justification” statements), and task-specific human-annotated rules. The model is iteratively fine-tuned from those scores so the critic improves over time. The result: measurable gains on qualitative benchmarks (Kimi led EQ‑Bench and tops creative-writing leaderboards) and fewer reward-hacking failure modes than prior LLM-evaluated RL efforts. The takeaway for AI/ML teams is practical: imperfect, well-scoped rubrics plus continuous critic updates can enable RL on non‑verifiable skills, trading exhaustive coverage for consistency that resists gaming. Trade-offs remain—Moonshot notes Kimi can sound overly confident due to discouraging self-qualification—but this rubric-driven RL offers a scalable, lower-cost path to improving “soft” capabilities like style, engagement, and grounded dialogue.