Building Code-Chunk: AST Aware Code Chunking (supermemory.ai)

Supermemory has unveiled code-chunk, an advanced AI infrastructure tool designed to enhance code ingestion and searchability through superior code chunking techniques. Unlike traditional methods that indiscriminately split code into fixed character counts—often resulting in meaningless fragments—code-chunk leverages Abstract Syntax Trees (AST) to create semantic-aware code chunks. This innovative approach identifies natural boundaries within the code structure, such as functions and classes, significantly improving the relevance of search results for AI applications. The technical backbone of code-chunk includes the use of tree-sitter for accurate code parsing, coupled with rich context extraction that captures essential metadata like function signatures and import relationships. This rich contextualization allows embeddings to convey deeper semantic meaning than raw code alone. Benchmark results indicate that code-chunk outperforms existing solutions, achieving 70.1% Recall@5 and a robust IoU of 0.43, demonstrating its effectiveness in producing relevant chunks aligned with the code's semantic architecture. This advancement is set to streamline AI interaction with code, paving the way for more accurate and efficient development processes in AI and machine learning applications.