We tested 6 AI assistants on the same solar data (heliopeak.app)

HelioPeak has tested six AI assistants using a comprehensive export file of solar production data to evaluate their analytical capabilities. This experiment, which included a synthetic dataset from a Belgian solar installation, aimed to inform the development of a new “Export for AI Analysis” feature. The concept allows users to generate a Markdown file containing their data and tailored instructions, which they can then input into AI chatbots for analysis without intermediaries. The findings revealed significant discrepancies in the outputs provided by the AI assistants, highlighting a troubling variability in their reliability and accuracy. The disparity in performance, from AI models that fabricated errors or produced inaccurate data to those that generated competent reports, underscores a critical challenge within the AI/ML community: the need for improved consistency and accuracy in AI outputs, especially when processing complex datasets. For example, while Google’s Gemini Pro excelled with thorough and accurate analyses including contextual judgments, Microsoft Copilot presented misleading information due to imagined data truncation. As developers refine their prompts and strategies for AI interactions, such testing not only serves as a benchmark but also underscores the necessity of developing systems that prioritize integrity in their analyses over expediency.