Advancing Search-Augmented Language Models (research.perplexity.ai)

Perplexity has introduced a two-stage post-training pipeline for enhancing search-augmented language models, addressing the challenging balance between factual accuracy, efficiency, and user preferences. The significance of this development lies in its potential to improve web search agents, crucial for delivering reliable and comprehensive search experiences. By separately optimizing for deployment-critical behaviors through Supervised Fine-Tuning (SFT) and refining capabilities using Reinforcement Learning (RL), Perplexity's approach mitigates the trade-offs typically encountered in single-objective optimizations. In the first SFT stage, models are initialized to uphold critical behaviors like instruction adherence and language consistency, while the subsequent RL stage further elevates search accuracy and tool-use efficiency. Employing Group Relative Policy Optimization (GRPO), the RL phase is meticulously guided by a carefully curated dataset and a composite reward design that balances multiple objectives. This innovative structure, especially the focus on mitigating reward hacking through gated aggregation and token-level Importance Sampling, enhances model performance while adhering to deployment standards, paving the way for more effective and user-aligned AI-driven search solutions.