Show HN: Thaw – Git branch for a running LLM (fork agents, skip prefill) (github.com)

A new tool called "Thaw" has been announced that enables real-time forking of AI agents from a running state, significantly improving efficiency in exploring multiple hypotheses simultaneously. By allowing agents to fork "N" ways without the need for cold pre-filling of model weights, Thaw facilitates parallel processing while sharing a common memory context. This innovation, using a snapshot mechanism that captures the entire state of a running session—including weights, KV cache, and scheduler state—has shown to reduce the time required to branch computations from over 340 seconds to just under 1 second, marking a substantial leap in performance. The Thaw framework is especially impactful for the AI/ML community engaged in reinforcement learning (RL) and real-time coding applications. With traditional methods incurring heavy costs for each fork due to the necessity of resetting model state, Thaw's capability to maintain divergent paths from a shared state allows for a higher throughput with less computational overhead. It supports session migration across different hardware setups, enabling seamless transitions without loss of data, making it an attractive option for developers looking for rapid prototyping and effective resource management in AI model training and execution. The open-source nature of Thaw means it is accessible for integration with existing frameworks like vLLM and SGLang, paving the way for broader adoption.