S-Expression Array for Lightweight Immutable Representation (SealIR) (github.com)

A new library titled SealIR has been introduced, offering an innovative data structure for the efficient and immutable representation of S-expressions. S-expressions, which are critical in various AI applications, are stored as a flat array of integers—this design significantly enhances memory efficiency and access speed. Each integer in the array either denotes a token or references another S-expression node, with tokens mapped to negative indices in a dedicated dictionary. This immutable representation prevents accidental modifications, ensuring the integrity of data structures. The technical features of SealIR stand out, particularly its non-recursive processing approach, which simplifies the codebase and enhances performance. The core Tape class manages the heap and token dictionary, allowing for efficient tree traversal from the bottom up. As new nodes are added in an append-only manner, the design guarantees that parent nodes appear later in the heap than their children, streamlining traversal. This advancement is significant for the AI/ML community as it not only improves the efficiency of working with S-expressions but also opens doors for more complex and higher-performing applications in AI models.