Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites (arxiv.org)

A recent study delves into the significant shortcomings of Large Language Models (LLMs) in terms of moral competence, which is essential for effective AI alignment with human values. Researchers argue that for AI systems to align with human norms, they must exhibit coherent policies that map situations to moral verdicts consistently and reliably. The study introduces four structural conditions—verdict stability, monotonicity, decisiveness, and Pareto viability—that serve as a foundation for assessing moral competence based on behavior rather than moral benchmarks. Through extensive simulations involving LLM-based agents faced with moral dilemmas, the research reveals a concerning lack of coherence in their moral judgments. The findings indicate that these models can show dramatic shifts in their responses based on minor changes in scenario wording or context, with verdict-rate variances reaching up to 99 percentage points. This inconsistency raises crucial questions about LLMs' suitability for alignment tasks, suggesting that they currently lack the necessary moral framework to engage meaningfully in alignment efforts. This study highlights the need for a deeper understanding of moral competence in AI, signaling a vital area for future research in the AI/ML community.