Building Fast Vector Search for Legal Documents (huggingface.co)

I built a fast semantic search over 143,485 chunks of Australian High Court decisions to test whether vector search can handle dense, sensitive legal text at scale. Along the way I compared embedding providers (Isaacus, OpenAI, Voyage AI, Google Gemini) and a small local model (fine-tuned BAAI/bge-small-en on the legal corpus). A major non-technical finding: read embedding API terms. Some providers claim rights to train on your data or require opt-outs; Isaacus stood out for not training on user data by default (author notes a working relationship). Local embeddings avoid data-leak risk and rate limits but may trade off embedding quality vs proprietary API models. Technically, the local 384‑dim model dominated on speed (924 texts/sec batch; ~7 ms single-query p95 ~15 ms) while API models ran 14–184 texts/sec and 500–1,700 ms single-query latencies (OpenAI 184/sec, avg 1,114 ms; Isaacus 102/sec, avg 1,532 ms; Google 19.8/sec, avg 501 ms; Voyage 14/sec). Isaacus’ 1,792‑d embeddings have PCA-like structure so truncating to 256 dims cut memory 7× and sped searches 8.6× (459 q/s vs 53 q/s) at the cost of recall (61% recall@10 relative to full-dim self-matching). For CPU-first deployment I used USearch (SIMD-optimized) and HNSW: building an index (1792d) took ~214s and yielded ~993 q/s with 98.6% recall@10, demonstrating that CPU SIMD + HNSW can give sub-millisecond, cost‑effective legal search without GPUs. The takeaway: choose embeddings based on legal/privacy terms and run empirical trade-offs between dimensionality, latency, accuracy, and infra cost for RAG or production search.