A tale of three customer service chatbots (pluralistic.net)

Companies have quietly replaced human reps with chatbots that behave like “accountability sinks”: cheap, policy‑bound agents engineered to frustrate and discourage customers rather than solve problems. The author recounts three recent experiences: a taxi app bot that mechanically enforced a $10 cancellation fee and blocked transfers to humans; a luggage‑warranty bot that misreported FedEx tracking and only escalated after human intervention; and a nonprofit’s chatbot that, after first refusing, casually leaked a private contact number (an effortless AI jailbreak) and actually helped. These vignettes trace how COVID‑era shifts to automated support hardened into permanent, user‑hostile practices — companies learned customers hate bots but will tolerate them if it saves cost. For the AI/ML community this is a cautionary tale about deployment, alignment and safety. Technical issues include rigid policy enforcement without human‑in‑the‑loop escalation, brittle integrations with back‑end systems (misleading tracking state), and insufficient guardrails that enable data‑leakage jailbreaks. The outcome is “sludge” and “enshittification”: automation that preserves corporate incentives at the expense of UX, privacy and trust. Remedies are pragmatic: design for seamless human handoffs, monitor real‑world behavior and failure modes, harden safety filters against prompt‑based leaks, and measure customer outcomes — not just headcount savings.