Spark 4.2 has a feature that could retire your vector database (thenewstack.io)

Apache Spark 4.2 has launched, introducing features that enhance its role in enterprise data processing and have the potential to disrupt the use of standalone vector databases. Key updates include native vector search capabilities, governed metrics for consistent data reporting across teams, improved Python interoperability, and enhanced real-time streaming functionalities. These upgrades allow developers to perform more tasks within the Spark ecosystem, reducing the need for additional data management systems. The addition of native vector search with distance and similarity functions streamlines the retrieval process by eliminating data movement between Spark and external databases. Furthermore, governed metric views ensure a unified definition of business metrics, minimizing discrepancies in AI applications that rely on consistent data. With real-time streaming updates like Auto CDC, Spark 4.2 positions itself as a vital tool for continuously updated AI workloads, moving beyond its traditional role of data preparation to become a component of the data serving layer. This evolution signifies a broader trend within the AI/ML community toward integrated solutions that simplify workflows and enhance efficiency.