Tradeoff considerations while running LLM models locally (swapniltalekar.substack.com)

A recent exploration into running large language models (LLMs) locally has revealed significant challenges and constraints, particularly when employing agent frameworks like Hermes. The author shares their experience with Hermes, which requires models with a context window of at least 64,000 tokens. This limitation effectively rules out many small models, as agent frameworks inject extensive system prompts and additional information into each interaction, resulting in exorbitantly high token usage. For instance, a simple "hi" message to an LLM can cost over 17,000 tokens, significantly impacting the feasibility of local model deployment for cost-sensitive users. In search of more efficient solutions, the author transitioned to OpenRouter, a multi-model gateway that simplifies integration and management of AI models. OpenRouter offers advantages such as automatic failover, response caching, and cost-effective session management, proving to be a more suitable choice for dynamic use cases. Through Hermes' compatibility with OpenRouter, users can leverage cheaper and efficient models without the overhead associated with local running. The findings emphasize that while local models could be viable under specific conditions, utilizing a versatile platform like OpenRouter may be the best option for most developers looking to optimize both performance and costs in their AI applications.