Show HN: What Apple's OS updates silently change in the on-device AI model (umer9538.github.io)

A recent project unveiled by developer Umer9538 highlights how Apple's OS updates affect the AI models used on devices, changing them without explicit user notifications or documentation. By capturing the performance of AI models across different OS builds, the project demonstrates that Apple and Google update these models silently, making it impossible to benchmark previous versions once they are superseded. The data shows that on macOS, the models provide consistent outputs, but variations in performance were noted on the iOS simulator and iPhone models, suggesting discrepancies in how updates are implemented across platforms. This development is significant for the AI/ML community as it emphasizes the need for transparency in AI model changes, which can drastically alter app behaviors and user experiences. The project provides a structured methodology for capturing and analyzing model performance changes, revealing that some output refusals and behaviors stem not just from the AI model itself, but also from the operating system's guardrails. The implications of this work suggest that developers need to continually validate app performance against potential silent updates, reinforcing the importance of maintaining a clear understanding of AI model dependencies with each OS revision.