Zebra-Llama – Towards efficient hybrid models (arxiv.org)

A new hybrid modeling approach called Zebra-Llama has been proposed to enhance the efficiency of large language models (LLMs) in response to the rising demand for sustainable AI solutions. By integrating existing pre-trained models, Zebra-Llama combines State Space Models (SSMs) and Multi-head Latent Attention (MLA) layers, creating 1B, 3B, and 8B model variants that maintain Transformer-level accuracy while significantly reducing resource requirements. This innovative technique utilizes only 7-11 billion training tokens—compared to the trillions typically needed for pre-training—thereby offering a more eco-friendly alternative to retraining. Zebra-Llama’s designs notably minimize the key-value cache size to a mere fraction (3.9%, 2%, and 2.73% of the original) for its variants, without compromising performance—achieving over 97% of average zero-shot accuracy on benchmark tasks. The model outperforms existing solutions like Minitron and MambaInLlama, showcasing up to 7% improved few-shot accuracy with 8x fewer training tokens and over 12x smaller cache memory. Additionally, it boasts a throughput increase of 2.6x-3.8x at a context length of up to 32k tokens. This research promises to democratize access to powerful AI models by streamlining their deployment and resource requirements, with the authors planning to release code and model checkpoints for community use.