He asked AI to count carbs 27000 times. It couldn't give the same answer twice (www.diabettech.com)

In a recent study, an investigation into AI models' accuracy in estimating carbohydrate content from food images revealed alarming inconsistencies. Researchers submitted 13 food photos—each sent over 500 times to OpenAI GPT-5.4, Anthropic Claude Sonnet 4.6, Google Gemini 2.5 Pro, and Gemini 3.1 Pro—to assess their reliability in carbohydrate counting for diabetes management. The findings showed significant variability in estimates, with some models producing counts that could dangerously impact insulin dosing. For example, Gemini 2.5 Pro's estimates for the same photo ranged from 55g to 484g, leading to potential insulin overdoses of up to 42.9 units, underscoring the risks of relying on these models without oversight. This study highlights critical implications for the AI and ML community, particularly in healthcare applications. The results reveal that while using AI for carb counting appears convenient, it poses significant risks due to systematic bias and stochastic variability; for instance, all models tended to overestimate carbs, which could lead to unnecessary insulin dosing. Moreover, the confidence scores reported by the models proved unhelpful for gauging accuracy. Instead, the study advocates for a careful approach when utilizing AI in medical contexts: querying multiple times and analyzing the spread of estimates would offer a better indication of uncertainty. This research provides crucial insights for developers and users of diabetes applications, emphasizing the necessity for human oversight in AI-driven health solutions.