Incompressible Knowledge Probes: Measuring what frontier models know (01.me)

A new benchmark called Incompressible Knowledge Probes (IKP) has been introduced to assess the factual knowledge across frontier AI models, comprising 1,400 questions divided into seven tiers of obscurity. This benchmark highlights that factual knowledge is fundamentally incompressible, meaning it cannot be generated through reasoning or improvements in model architecture, thus serving as a reliable indicator of a model's parametric storage capacity. The benchmark aims to create a clearer understanding of how these models store knowledge relative to their size. This development is significant for the AI/ML community as it establishes a direct relationship between model accuracy and parameter count. An analysis of 89 open-weight models demonstrates that for every tenfold increase in parameters, accuracy increases by approximately 15 percentage points. Proprietary estimates of several closed models were also derived from a calibration curve, revealing enormous variances in their effective knowledge storage capacities. The findings underscore the intricate balance between model size and knowledge retention, ultimately providing key insights for future AI architects looking to enhance model performance in a more efficient and effective manner.