Engineering message appends for AI transport: three vignettes (ably.com)

Ably has unveiled AI Transport, a new transport layer designed for streaming large language model (LLM) output across Ably channels, while ensuring smooth user experiences like resumability and multi-device continuity. This technology addresses the challenge of treating gradually emitted model outputs as unified messages, preventing clutter from fragmented tokens in channel history. The key innovation here is the introduction of "appends," a new type of pub/sub message that allows incremental updates to existing messages, ensuring subscribers receive a coherent view of responses as they unfold. This development is significant for the AI/ML community as it streamlines interaction with LLMs, enabling real-time communication while maintaining clarity in historical message storage. Ably's design decision to store each append in two forms—both as a fragment and as a comprehensive update—enhances efficiency for subscribers connecting mid-stream. Moreover, the conflation of appends has been treated as a core semantic property, negating the need for users to set up explicit conflation rules and thereby simplifying the API experience. By building upon generic primitives rather than bespoke features, Ably positions itself for greater adaptability, potentially paving the way for future innovations in AI messaging and interaction frameworks.