LLMs and Shaders (amitp.blogspot.com)

A recent post draws an intriguing parallel between cloud-based LLMs (large language models) and fixed-function graphics pipelines, highlighting how both present limitations to user creativity. While graphics pipelines initially restricted features like lighting models and texture lookups, the introduction of shaders transformed the graphics landscape, enabling more advanced and customizable visual effects. Similarly, today's LLMs operate within a rigid structure—processing a system prompt, generating tokens, and appending them to a transcript—without offering users the flexibility to explore multiple narrative paths or control token selection beyond basic parameters. This comparison is significant for the AI/ML community as it calls attention to the potential evolution of LLMs. The author advocates for a future where users could customize LLM pipelines akin to using shaders in graphics, allowing dynamic editing of transcripts and diverse token-generating strategies. Enabling such features could broaden the applications of LLMs far beyond conversational interfaces, driving innovation in fields like scientific research, creative writing, and beyond. By pushing for greater flexibility and programmability in LLMs, the community may unlock new functionalities and uses for these powerful models, much like the expanded roles of GPUs in various non-graphical computations.