Jepsen: NATS 2.12.1 (jepsen.io)

The recent announcement of NATS version 2.12.1 highlights significant issues in the JetStream subsystem, particularly regarding data durability and consistency. JetStream, which uses the Raft consensus algorithm for ensuring "at least once" message delivery, faced critical failures during testing conducted by Jepsen. These included instances of total data loss due to process crashes in prior versions and profound issues with file corruption affecting acknowledged writes. Notably, single-bit errors in JetStream's data files led to substantial message loss, with reported figures indicating a disturbing loss of upwards of 679,000 acknowledged writes under certain conditions. Moreover, the testing revealed that the default settings for data persistence in NATS allowed for substantial risk of data loss during power failures. By acknowledging messages without ensuring their immediate persistence to disk, NATS's design could result in the loss of a significant number of writes during a catastrophic event, contradicting its promises of durability. With these findings, the NATS team is urged to refine JetStream's error handling and data persistence mechanisms, particularly to consider adjustments to the fsync intervals to bolster reliability. These developments are crucial for the AI/ML community, as NATS is often leveraged in real-time data processing applications where consistency and durability are paramount.