FlashPack: Fast Model Loading for PyTorch (blog.fal.ai)

FlashPack is a new, pure‑Python file format and loader for PyTorch checkpoints that dramatically reduces model startup time by treating a model’s entire state_dict as one contiguous, indexed byte stream. In benchmarks the team reports 2–6× faster loads than common methods (load_state_dict, accelerate) even on systems without GPU Direct Storage (GDS). That matters because checkpoint I/O — not just GPU compute — commonly becomes the bottleneck when spinning up or reloading models, and faster loading directly increases GPU utilization and reduces latency for real-world ML deployments. Technically, FlashPack flattens all tensors into a single file with a compact weight map (key, shape, offset), memory‑maps that file, and reads it into a few mid‑sized CPU buffers (≤64 MB) in a round‑robin pattern. Each buffer is paired with a CUDA stream so disk, CPU and GPU work in parallel; tensors are reconstituted on the GPU as views into the flat block (reshaping is O(1)), avoiding copies. It converts .pt/.safetensors/diffusers/transformers checkpoints into .flashpack via a one‑command converter and integrates via mixins or direct API calls. Current limitations: all weights must share one dtype, no device‑map/pipeline‑parallel loading, and state_dict transformations aren’t supported. If your model fits those constraints, FlashPack offers a simple, dependency‑light way to cut load times substantially.