Serving GLM-5.3-Flash on one Trainium2 box with NKI layer megakernels (runanywhere.ai)

A recent breakthrough in serving the GLM-5.3-Flash model, which boasts 320 billion parameters and a million-token context, was achieved using a single AWS Trainium2 trn2.48xlarge instance. This marks a significant advancement for the AI/ML community, as it demonstrates a dramatic improvement in processing capabilities, particularly for coding tasks with real-world agent interactions, as evaluated on the SemiAnalysis AgentX benchmark. The team successfully restructured the model's decoding process by utilizing a Neuron Kernel Interface (NKI) to create optimized megakernels, which enhanced the model's response time for lengthy cached prompts and agent requests. The results reveal that the system can handle up to 28.9 agent requests per minute, showcasing a decoding speed of 3,720 tokens per second per Trainium2 chip. Notably, a single instance was able to process a massive 1,039,912-token prompt in just 190.8 seconds—8.7 times faster than previous benchmarks—highlighting both efficiency and cost-effectiveness at $0.021 per request. Despite notable achievements in throughput and initial response times, the article notes that further optimization efforts are required to address latency issues during the decoding process, which currently takes longer than the theoretical bandwidth limits suggest. This work sets a promising foundation for future advancements in serving highly complex models efficiently within the AI space.