Injecting Spotify API Data into the Gemini AI Context Window (jessewaites.com)

I built a real-time voice agent on my website using Google’s Gemini 2.0 multimodal live API and injected dynamic Spotify data into the model’s system instruction so the assistant can answer with up-to-the-minute info about what I’m listening to, my top artists, and recent tracks. The architecture uses a Node.js WebSocket relay between the browser and Gemini: the browser streams PCM audio (captured and converted via Web Audio API), the server fetches Spotify data (using an OAuth refresh token to get short‑lived access tokens) in parallel, formats that data into natural language, and injects it into Gemini’s context before the session starts. Audio frames are base64‑wrapped JSON over WebSocket, Gemini returns audio (24 kHz PCM) and text responses, and the server proxies everything so API keys never hit the client. This pattern is significant because it shows a practical way to give LLMs live, personal state without retraining — simply place fresh API outputs into the context window. Key technical takeaways: use parallel API calls and caching to avoid latency and rate limits, be mindful of context window size and selectivity, enforce server-side OAuth and API key security, and handle audio format conversions and WebSocket reliability for real‑time UX. The approach is reusable for calendars, location, or sensors, but requires tradeoffs around privacy, scaling, and cost.