Spiral (spiraldb.com)

Spiral is a groundbreaking data platform designed from the ground up to meet the demands of the "Third Age" of data systems, where machines—not humans—are the primary consumers of data. Traditional data architectures, built for human-scale inputs and outputs or the big-data-era focus on dashboards and aggregate reports, struggle with AI workloads that require rapid, large-scale, low-latency access to diverse data types like embeddings, images, and videos. Spiral tackles this mismatch by combining a new state-of-the-art columnar file format called Vortex—with contributions from major industry players and academic validation—and a native object storage architecture optimized for machine-scale throughput and unified governance. Technically, Vortex delivers compression ratios comparable to Parquet but with 10-20x faster scans, 5-10x faster writes, and orders-of-magnitude faster random reads, enabling GPUs like NVIDIA’s H100 to be fully saturated by feeding millions of data items per second directly from S3 storage while bypassing costly CPU bottlenecks. Spiral’s unified system eliminates the artificial trade-offs in handling small and large data blobs, providing seamless and secure access to microsecond-level embeddings and multi-gigabyte videos alike, all under a coherent, security-first permissioning model. This evolution promises to end inefficiencies that cost AI teams precious time and overhead, positioning Spiral as critical infrastructure for AI-first enterprises ready to scale next-generation models without the performance and security compromises inherent in legacy systems.