New agents.txt file found on DreamHost (journal.kvibber.com)

DreamHost has introduced a new `agents.txt` file to the root directories of all hosted websites, retroactively applying it to existing sites as of May 7. This default file serves a purpose similar to the well-known `robots.txt`, providing directives that govern how AI models can interact with hosted content. Notably, the file discourages the use of training data from the sites while permitting on-the-fly AI-generated summaries, alongside restrictions on accessing sensitive folders. This development is significant for the AI/ML community as it reflects growing concerns over how websites are utilized by large language models (LLMs). While the measures taken are intended to protect site owners from unauthorized data scraping, the retroactive implementation has sparked frustration among users, particularly due to its inclusion of explicit permission for retrieval-augmented generation (RAG). Additionally, discrepancies in the syntax and purpose for the `agents.txt` file versus its initial proposal, renamed to `agent-manifest.txt`, highlight potential inconsistencies in compliance and communication within hosting services.