What's the Role of Trust in AI? (algorithmictradeoff.substack.com)

The piece argues that our long-evolved heuristics for trust—how we shortcut vetting when persuaded by others—fit well for human-to-human relations, and, later, for deterministic software whose behavior is predictable and verifiable. The Internet blurred that contract by hiding which humans sit behind digital interfaces, but people could still reason in human-to-human terms. Generative AI, however, collapses these categories: it looks like a tool, behaves like a black‑box model trained on human data, and is marketed like an autonomous conversational agent. That mismatch means typical users apply the wrong trust equation (treating probabilistic, opaque models as deterministic tools), which the author sees as a root cause of many societal harms from large-scale consumer AI. For the AI/ML community this diagnosis has practical consequences. Technically, GenAI’s stochastic, non-reproducible outputs and opaque training provenance undermine traditional vetting and reliability assumptions; debugability, provenance tracing, and update stability become central concerns. The essay implies a need for new trust frameworks—better provenance/attribution, model accountability, verifiable evaluation, interface design that signals uncertainty, and institutional certifications—to align user mental models with how these systems actually work. In short: without explicit design, governance, and literacy changes, generative models will continue to break the trust heuristics people rely on, amplifying risks even when underlying intelligence is “subhuman.”