Nukitori is a Ruby gem for HTML data extraction (github.com)

Nukitori has launched as a new Ruby gem designed for efficient HTML data extraction. By leveraging a large language model (LLM) to create reusable XPath schemas, it enables users to specify data they need to extract once, and then perform ongoing data extraction rapidly without AI assistance, utilizing the Nokogiri library. This process enhances robustness by avoiding brittle page-specific identifiers and rather relying on generalized scraping logic, which is particularly beneficial for users scraping similarly structured HTML pages. The significance of Nukitori lies in its ability to streamline the web scraping process while maintaining transparency and flexibility. The generated schemas are output as plain JSON, allowing for easy inspection and versioning. Additionally, it supports various LLM providers, such as OpenAI and Anthropic, which enables developers to customize their scraping experience. With features like token optimization to minimize unnecessary data sent to the LLM, and options for LLM-only extraction when needed for complex data transformations, Nukitori balances performance and adaptability in web scraping use cases. This innovation not only simplifies the extraction logic for developers but also promises faster and more reliable data retrieval, making it a valuable tool in the AI/ML toolkit for data-driven applications.