Floor plan area micro-eval (kerrickstaley.com)

A recent evaluation tested the capabilities of two language model agents, GPT-5.5 and Claude Opus 4.8, in accurately computing the square footage of apartments from floor plans. The evaluation involved assessing their ability to infer the scale from room dimensions and correctly identify areas contributing to floor space. Over 15 floor plans, GPT-5.5 recorded an average error of 23.2%, while Claude Opus 4.8 demonstrated greater accuracy with an average error of 13.2%. The errors primarily stemmed from mis-estimating scale rather than incorrectly marking interior spaces, indicating that both models are more competent in identifying marked areas than performing accurate scaling. This evaluation holds significant implications for the AI/ML community as it highlights the potential applications of language models in real estate, architecture, and construction. Despite the relatively impressive accuracy displayed, the results suggest that current models like GPT-5.5 and Claude Opus 4.8 are still not reliable enough for critical tasks requiring precise measurements. The study also showcased the variability in performance across runs, particularly with GPT-5.5, which struggled with certain floor plans due to errors in dimension recognition. This variability emphasizes the need for continued refinement of AI systems to ensure consistent reliability and accuracy in practical applications.