Show HN: Markdownee – scrapes web into Markdown, own Trafilatura typescript port (www.markdownee.com)

🤖 AI Summary
Markdownee is a newly unveiled web scraping tool that converts website content into Markdown, HTML, or plain text formats, specifically tailored for machine learning applications like LLMs and data retrieval systems. It allows users to customize their scraping parameters such as link selection, depth limits, and content preservation, while also offering granular control over how elements like tables, links, and images are handled. The tool is built on Trafilatura Core, an open-source extraction engine available in both TypeScript and Python, which specializes in removing boilerplate content to enhance the quality of extracted data. This development is significant for the AI/ML community as it introduces a user-friendly CLI interface that simplifies the scraping process, making it accessible for researchers and developers who may not have extensive experience with web scraping. Furthermore, Markdownee's approach contrasts with traditional methods like Mozilla Readability, leveraging advanced structural and content heuristics to improve extraction accuracy. Users can get started with a simple command via Node.js, highlighting its ease of use and immediate practical application in building datasets for machine learning purposes. The capability to download images alongside text further enhances its versatility for different research needs.
Loading comments...
loading comments...