Show HN: I built "AI Wattpad" to eval LLMs on fiction (narrator.sh)

A new project dubbed "AI Wattpad" has emerged, designed to evaluate large language models (LLMs) specifically for creative writing. This initiative aims to identify which AI models can effectively engage readers by providing insights based on real data rather than relying on traditional synthetic benchmarks. It employs a unique architecture that combines three specialized models: a Brainstorming Model that generates concepts and plots, a Writer Model that crafts chapters and narratives, and a Memory Model that maintains story context and recalls essential details throughout the writing process. This project is significant for the AI/ML community as it addresses the challenge of assessing LLMs in a creative context, where standard evaluation methods might fall short. By benchmarking these models on their ability to produce compelling fiction, AI Wattpad could influence the development of more advanced and context-aware AI writing assistants. The collaboration feature also invites users to suggest models for benchmarking, fostering community engagement and driving innovation in the field of AI-driven storytelling.