Conant and Ashby's "Good Regulator Theorem" (2021) (gokererdogan.github.io)

The "Good Regulator Theorem" proposed by Conant and Ashby emphasizes that an optimal controller of a system should be a model of that system, suggesting the necessity for model-based techniques in AI and machine learning. This theorem, however, has been reassessed in the context of reinforcement learning (RL) and found to have limited relevance. The main distinction lies in the notion of "model"; the theorem's definition of a model aligns more closely with what we refer to as a policy in RL, rather than the typical model-based definitions used within the field. The theorem shows that optimal policies must map distinct system states to different actions, but lacks practical implications for model-free versus model-based RL approaches. Moreover, the objective of minimizing the entropy of the final state distribution, as proposed by the theorem, may lead to trivial solutions that do not effectively address actual RL challenges. The analysis highlights that while the theorem provides insight into policies, it is not fundamentally different from the problems model-free RL techniques solve. The connection to latent MDPs and other advanced techniques suggests that while there are conceptual similarities, the practical applications within the broader RL landscape remain more nuanced, necessitating further exploration to better align the theorem’s principles with contemporary RL methodologies.