Discovering Reinforcement Learning Interfaces with Large Language Models (akshat-sj.github.io)

A new framework called LIMEN has been proposed to enhance the evolution of reinforcement learning (RL) interfaces using large language models (LLMs). Traditionally, crafting the environment interfaces that define observations and reward functions for RL tasks has required extensive manual input. LIMEN automates this process by generating candidate interfaces as executable programs and refining them iteratively through policy training feedback. This framework operates on a bilevel optimization strategy, where the outer loop searches for optimal pairs of observations and rewards to maximize task success, while the inner loop employs Proximal Policy Optimization (PPO) for training. The significance of LIMEN lies in its ability to jointly evolve observations and rewards, thereby achieving better overall performance across a range of tasks, including complex continuous control and discrete gridworld scenarios. Notably, joint evolution consistently outperformed reward-only and observation-only approaches, showcasing its capability to discover effective RL interfaces without requiring extensive pre-defined parameters. This breakthrough not only highlights the potential of LLMs in automating RL processes but also points to a new paradigm in task interface synthesis that could streamline future AI/ML developments significantly.