Show HN: Run Jev-style models locally on Mac with 0.74 GB RAM (github.com)

A new tool called Laya allows users to run Jev-style typed decision models locally on Apple Silicon Macs with as little as 0.74 GB of RAM. This innovative approach provides a median response latency of about 32 milliseconds using roughly 2.1 GiB of RAM on an M5 Pro device, significantly optimizing resource usage while maintaining fast performance. Laya is designed for specific applications, such as customer service and incident management, rather than being a general-purpose language model, making it particularly valuable for organizations looking to streamline decision-making processes. Laya employs Metal Performance Shaders (MPS) through PyTorch to efficiently utilize the Mac's GPU, thereby enhancing computational performance. The software features several memory settings, allowing users to balance RAM consumption and latency according to their needs. For instance, the "minimal" setting requires only 0.74 GiB but comes with longer response times compared to the "reduced" or "full" settings. This flexibility and efficiency could empower developers and businesses by enabling more resource-efficient AI applications, fostering broader adoption of AI solutions in environments with limited hardware capabilities.