Our agent found a bug with WireGuard in Google Kubernetes Engine (lovable.dev)

Last week, Lovable faced persistent errors in their Google Kubernetes Engine (GKE) setup, prompting a thorough investigation by their infrastructure team, which discovered a significant bug in the integration of WireGuard within Google's anetd networking daemon. The AI debugging agents played a crucial role in pinpointing the issue, which revolved around excessive pod restarts due to a concurrency issue within the WireGuard module, where multiple goroutines accessed a shared data structure incorrectly. This bug led to networking instability—crucial for Lovable, which rapidly spins up over 50 sandboxes per second. Through collaboration with Google's support team, Lovable temporarily mitigated the issue by disabling node-to-node encryption, although this posed some security trade-offs. Further troubleshooting revealed an MTU mismatch affecting network communications, exacerbated by mixed configurations in the cluster. Ultimately, the team successfully resolved these interconnected issues, reinforcing the importance of layered failure awareness in distributed systems and the power of AI in debugging complex infrastructures. Google has since patched the original concurrency bug, benefiting all users of the GKE service.