Step 5 Preview: Architecture and Training for Agentic Work (platform.stepfun.ai)

The introduction of Step 5 Preview marks a significant advancement in AI model architecture, tailored specifically for agentic workloads that demand robust reasoning capabilities over lengthy contexts. This new model employs a "narrow but deep" design, comprising 92 transformer blocks and implementing a combination of Gated Sliding Window Attention alongside Sparse GQA to facilitate efficient training and inference on extended sequences up to 1 million tokens. By leveraging sparse attention mechanisms and optimizing the indexing process, Step 5 Preview aims to enhance long-horizon reasoning without the excessive computational costs typically associated with large models. In addition to its architectural innovations, Step 5 Preview adopts advanced strategies for training stability, including the use of mixed precision techniques and finegrained gradient clamping to address issues like numerical instability. The model introduces an agent-native data construction system, enabling seamless coordination among agents to synthesize data efficiently while incorporating expert-defined quality controls. This holistic approach not only enhances the model's ability to perform complex tasks but also optimizes resource utilization, positioning Step 5 Preview as a promising tool for future AI applications in diverse domains.