Generalized K-Means Clustering for Apache Spark with Bregman Divergences (github.com)

Massive Data Science released version 0.6.0 of its Generalized K-Means library, promoting a new DataFrame-native Spark ML API (RDD legacy archived) and adding broad support for multiple Bregman divergences and advanced k‑means variants. The update makes the library production-ready for Spark ML pipelines (Estimator/Model pattern), includes a PySpark wrapper, and targets Scala 2.13 / Spark 3.5.x (backwards-tested on Spark 3.4 and Scala 2.12). It’s significant because it extends clustering beyond squared‑Euclidean to KL, Itakura–Saito, L1/Manhattan (K‑Medians), Generalized‑I and logistic losses, while delivering Bisecting, X‑Means (BIC/AIC auto‑k), Soft K‑Means, streaming updates, and K‑Medoids (PAM/CLARA)—useful for audio, count data, heavy‑tailed and robust clustering regimes. Key technical details and production implications: the DataFrame API is the recommended path with model persistence (cross-version save/load), deterministic behavior, and extensive CI (≈740 tests including divergence/gradient checks and performance sanity). Assignment strategies balance scale and exactness: a Squared‑Euclidean fast path uses a cross‑join + groupBy(min) with a RowId provider, while general Bregman losses use broadcasted centers + tight JVM UDFs subject to a k×dim broadcast ceiling. Input transforms (log1p, epsilon shift) support positive-domain divergences and centers are learned in transformed space (invert for reporting). Streaming supports micro‑batch snapshots and pretrained init. Overall, 0.6.0 is a comprehensive, scalable, and well‑tested toolkit for non‑Euclidean clustering in Spark ML.