My trick for getting consistent classification from LLMs (verdik.substack.com)

A developer published a practical technique and Golang package (consistent-classifier) for turning stochastic LLM outputs into deterministic class labels when your label space is unconstrained. The core insight: LLMs produce lexicographically inconsistent but semantically consistent labels for the same input. The pipeline embeds both content and generated labels, stores them in a vector index (Pinecone), uses cosine-similarity search to find near-duplicates, and applies a Disjoint-Set Union (DSU) “path compression” to map variant labels to a single root label. On a cache miss the system falls back to an LLM, stores embeddings, then clusters labels for future reuse. Benchmarks on ~10K tweets used gpt-4.1-mini for classification and voyage-3.5-lite (1,024-d) for embeddings. With a 0.80 cosine threshold the approach reduced unique labels from ~6,520 to ~1,381 (≈1/5), and cache-hit rate rose to ~94% by 10K items. Cost-wise vectorization is ~15% pricier initially but becomes ~10× cheaper per item by 10K as cache hits avoid LLM calls; latency is ~130% slower at first (embedding + search overhead) but improves as hits increase. Repo structure, config, and defaults are provided; the author recommends tuning similarity thresholds for different content lengths. This method offers a scalable, low-friction way to produce consistent labels for large unlabeled corpora without predefining label schemas.