Cross-Lingual News Dedup at $100/Month – Embeddings, Pgvector, and UnionFind (yingjiezhao.com)

A new AI-powered news aggregation service, 3mins.news, has implemented a cost-effective solution for cross-lingual news deduplication, processing articles from over 180 RSS sources in 17 languages for just $100 a month. The main challenge is deduplicating articles that report the same event in different languages, where traditional techniques like MinHash and Locality Sensitive Hashing (LSH) fail due to lack of token overlap. Instead, the service uses modern multilingual embeddings that map similar text into close proximity within vector space, allowing high accuracy in identifying semantically equivalent articles regardless of language. The system employs a two-pass clustering approach, utilizing PostgreSQL's pgvector extension for efficient similarity searches. In the first pass, new articles are matched against existing story embeddings to ensure that ongoing events are updated rather than duplicated. The second pass uses a UnionFind algorithm to cluster remaining articles into new stories by analyzing their similarities, ensuring that articles about the same event are grouped effectively. By leveraging this pipeline, 3mins.news minimizes costs and maximizes the efficiency of processing multilingual content, underscoring the significance of advanced embedding techniques in the evolving landscape of AI-driven news aggregation.