Building an infinitely learning geolocation game with RL (blog.sdan.io)

The author built Geospot Infinity, a photo-to-GPS demo that re-scores 10 candidate coordinates from a frozen GeoCLIP backbone (ViT‑L/14 + small MLP) using a tiny 3‑layer MLP re‑ranker whose 1026‑dim input concatenates image and location embeddings, similarity, and base probability. They tried online RL: sampling permutations with a Plackett‑Luce model and training with REINFORCE plus KL regularization against a Polyak‑averaged behavior policy (EMA baseline). In practice over 65% of users clicked the first guess regardless of accuracy, KL stayed <0.01 (the policy barely changed), and the REINFORCE policy actually degraded geolocation error to 2,878 km (+17%, 414 km worse) versus the baseline top‑1 GeoCLIP at 2,464 km; a best‑of‑10 oracle reduces error to 1,276 km (−48%). The outcome highlights a key lesson for online RL in applied systems: reward signal and user distribution matter more than quick policy tweaks. Preference bias (users habitually accepting the top option) turned learning into “preference tuning” rather than improving true geodesic accuracy. The author pivoted to Direct Preference Optimization (DPO) — pairwise winner/loser training against a frozen reference with a KL constraint — and released the demo and code. More broadly, the author cites recent work showing RL scaling has an asymptotic ceiling you raise with bigger models, longer context, larger batches and verifiable rewards; you can’t learn past your data distribution but you can change the distribution or reward design to lift that ceiling.