StreamLens – Visual Kafka Lineage and topology viewer with AI navigation (github.com)

StreamLens has launched a full-stack application designed to visualize Apache Kafka topologies, allowing users to interactively explore topics, producers, consumers, schemas, connectors, and ACLs. The tool features live topology visualization through React Flow and includes capabilities like auto-discovery of Kafka components, consumer lag monitoring, and detailed insights into topics and connectors. Notably, it allows users to produce messages from the UI and offers an AI assistant named StreamPilot, which leverages AI technologies from providers like OpenAI, Gemini, and Anthropic to enhance navigation and understanding of the Kafka environment. This development is significant for the AI/ML community as it bridges the gap between data streaming and AI, enabling engineers to better manage and optimize their Kafka clusters. By incorporating AI-driven navigation, StreamLens not only simplifies the visualization process but also empowers users to query their data pipelines more intuitively. The application supports both PLAINTEXT and SSL protocols, making it flexible for diverse deployment scenarios, and is accessible via user-friendly front-end and back-end structures built with modern technologies such as Vite, TypeScript, and FastAPI. With simplified cluster management and an engaging UI, StreamLens positions itself as a valuable tool for professionals working in data engineering and real-time analytics.