Better data infrastructure is needed for the AI era (tracto.ai)

TractoAI — drawing on a decade building big-data and AI infrastructure for firms like Yandex and Nebius — warns that current AI-era data tooling isn’t ready for scale. Modern AI workloads create complex, interdependent data flows (raw crawls, refined/train sets, synthetic data, agent trajectories, evaluation logs) that often live in separate storage systems (S3, OLAP/OLTP databases, event brokers). That fragmentation makes simple reuse (e.g., adding production failures into training) expensive, increases operational overhead, and forces brittle client-side workarounds — problems that amplify at petabyte scale and slow iteration for ML engineers and data scientists. Technically, TractoAI argues datasets should be first-class, table-like objects with schemas that span primitives, containers and multimodal types (images, audio, video, tensors, vectors), plus “blob/json/any” for unstructured fields. Treating datasets as transactional facades enables snapshot isolation, safe distributed writes, versioning and lineage, and avoids costly IO from ad-hoc file/metadata hacks. They point out S3/POSIX lack atomic multi-object transactions, motivating storage with built-in transactions (e.g., YTsaurus). Finally, better exploration tooling — fast stats, SQL-like ad-hoc queries, provenance discovery and structured viewers — is essential for debugging biases and accelerating reuse. The takeaway: unified namespaces, transactional scalable storage and structured multimodal schema + exploration tooling are foundational for productive, large-scale AI systems.