Show HN: Doom Compiled into an LLM (github.com)

A recent project has successfully compiled the classic video game DOOM into a transformer model using the torchwright framework. This innovative approach builds a computation graph that renders the game autoregressively, with each discrete token representing pixel information processed in a similar manner to modern chat models. Notably, the entire rendering logic, including wall selection and texture lookup, is contained within the transformer, allowing the host system to focus solely on feeding tokens and displaying pixels. This development is significant for the AI/ML community as it demonstrates a novel application of transformer architectures beyond traditional text generation, showcasing their potential in rendering complex visual scenes. The production artifact is a stock Hugging Face Phi3ForCausalLM model, comprised of sharded fp32 safetensors that do not require custom tokenizer code. The project emphasizes a minimalist "dumb-host principle" whereby the host only interfaces with the model during input and output, enabling efficient utilization of computational resources. With varying checkpoint sizes accommodating different hardware configurations, this project highlights the intersection of gaming and AI/ML, paving the way for future advancements in generative models capable of real-time rendering.