How well do LLMs generate R code? (skaltman-model-eval-app.share.connect.posit.cloud)

A new app has been released that evaluates the effectiveness of various large language models (LLMs) in generating R code. The evaluation utilizes the "An R Eval" dataset, which consists of complex R coding problems paired with their correct solutions. By leveraging the vitals package, the app scores each model's output, categorizing the results as Incorrect, Partially Correct, or Correct, based on the assessments conducted by Claude 3.7 Sonnet. This initiative is significant for the AI/ML community as it sheds light on the practical capabilities of LLMs in the realm of programming, particularly in R, a language widely used for statistical analysis and data visualization. The findings from this evaluation could guide developers in selecting the most effective LLMs for coding tasks, potentially influencing future improvements in model training and application. As LLMs continue to evolve, understanding their proficiency in generating accurate and functional code is crucial for both researchers and practitioners in the field.