Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama (patrickmccanna.net)

A recent discussion has emerged around the risks of using frontier AI providers like OpenAI and Anthropic, especially concerning user privacy and data protection during interactions with language models. The author highlights concerns that these companies could be exploiting user session data to enhance their models, raising ethical questions about their practices. As a result, those looking to safeguard their insights and ideas may find it imperative to migrate to self-hosted AI environments, where they can maintain control over their data and prompts. The technical challenges involved in this migration process are significant. The author shares firsthand experiences of transitioning 35KB prompts from a frontier provider to a self-hosted Ollama model, noting that local systems often have much smaller context windows, which can lead to inefficiencies and "thrashing" behavior when processing prompts. Key strategies suggested for successful migration include tuning prompts to fit within context limitations, splitting complex instructions into simpler units, and employing logging to manage session state. By addressing these issues, developers can maintain high levels of functionality while ensuring their data remains private and secure.