LLMs Still Excels at Synthesis, Not Discovery (www.zbeegnew.dev)

In a recent evaluation, an R&D engineer tested three powerful AI systems—Claude, ChatGPT, and Gemini—using a complex financial analysis task. While Claude and ChatGPT produced consistent and structured reports on the valuation of a private company, Gemini failed to deliver usable output. Notably, both successful models converged on similar financial estimates despite employing different analytical approaches, demonstrating their ability to synthesize existing knowledge effectively. This convergence highlights the strength of current state-of-the-art AI models in synthesizing information rather than discovering new insights. The AI systems excelled at pulling together existing data, employing established frameworks such as discounted cash flow analysis and probabilistic scenario modeling, ultimately saving significant time on analysis. However, the engineer emphasized that while such models are valuable for synthesizing knowledge, they fall short in innovation and creative reasoning—essentially, they cannot generate original questions or hypotheses. The engineer's experience underscores the need for human oversight in guiding AI tasks and discerning between synthesis and genuine discovery, where human insight remains crucial.