Teaching a small embedding model with LLMs to deliver GPT-like semantics in 10ms (instantdomainsearch.com)

A team revisited their domain search tool to combine the deep contextual understanding of large language models (LLMs) like GPT-4 with the speed of small embedding models. Traditional methods using fastText embeddings lacked domain-specific nuance—misinterpreting terms like “mint” in a business context versus general language. While GPT-4 provided accurate semantic matches, its high latency (~1.7 seconds) made it impractical for real-time search, where sub-10 ms responses are critical. To solve this, they used GPT-4 offline to generate millions of domain-specific training examples and fine-tuned a lightweight embedding model (all-MiniLM-L6-v2) with triplet loss to mimic GPT-4’s semantic understanding. Optimized via Rust and ONNX Runtime for efficient CPU inference and paired with hierarchical navigable small world (HNSW) indexing, the resulting model delivers semantic search results in under 10 ms with a 0.87 correlation to GPT-4’s grasp of meaning. This approach distills trillion-parameter LLM knowledge into a compact 22.7M-parameter model, balancing rich domain context with production-grade latency, enabling instant, semantically relevant domain suggestions. This innovation highlights a broader AI design principle: leveraging large models offline to train specialized embeddings that bring LLM-level semantics into real-time applications—bridging the gap between powerful, slow LLMs and fast, lightweight embedding models for domain-specific semantic search at scale.