Show HN: MicroLLM lab – try small LMs in browser, see tok/s and accuracy (github.com)

MicroLLM Lab has launched a platform that allows users to run 135M-class language models directly in their web browser without any need for API keys or complex setups like Python and CUDA. Utilizing WebGPU for hardware acceleration, the platform maintains model weights locally in the browser’s IndexedDB, ensuring that user prompts remain private. With impressive performance, a modern Mac mini can achieve over a hundred tokens per second, while older GPUs like the GTX 1060 remain capable. The platform prioritizes a user-friendly experience, enabling evaluation of speed and accuracy trade-offs for small models in a straightforward way. This development is significant for the AI/ML community as it democratizes access to language models, emphasizing privacy and cost-effectiveness. Users can experiment with model performance without incurring costs for API calls, making it ideal for on-device features, drafts, or sensitive information. The lab provides various small models from notable sources like Hugging Face and the creator yangqi0, with built-in comparison metrics to facilitate experimentation. This tool not only enhances understanding of small model capabilities but also underscores the trend towards local processing for better privacy and reduced latency in AI interactions.