Diving into Rama: A Clojure LSH Vector Search Experiment (shtanglitza.ai)

A developer experimented with building a complete vector-search pipeline inside Rama (a distributed Clojure backend) by implementing classic Locality-Sensitive Hashing (LSH) instead of more common graph methods like HNSW. The single Rama module handles ingestion, indexing, and querying: embeddings and LSH buckets are stored in two PStates (embeddings and $$lsh-tables with {:subindex? true} for performant large buckets), a stream-topology ingests vectors and hashes them into buckets, and a two-phase query topology first gathers candidate IDs from probe buckets (via a defgenerator gather-unique-ids) then re-ranks candidates with exact cosine-distance to return the top-k results. The prototype is tested locally using Rama’s InProcessCluster and a small Wikipedia embedding set. Key technical choices and implications: embeddings use dims=384 (all-MiniLM-L6-v2), nbits=8 (256 buckets per table), num-tables=4 (trade-off recall vs lookups), and top-k=50. Work distribution is controlled with Rama’s |hash partitioning; ops/explode, local-select, and aggs/+top-monotonic implement candidate scatter/gather and top-k aggregation. A practical compiler pitfall—embedding large, static hyperplane-sets inline at compile time—was solved by exposing them via defonce + a getter to avoid JVM method-size limits. The code is experimental (not production-optimized); next steps are full-scale performance benchmarks to evaluate Rama’s scalability for real-time vector workloads.