8.7.5.1 - Policy Hallucinations: Support Bots Promising Refunds or Terms You Don't Actually Offer (Difficulty: Advanced | Path: Scale)

In the rush to automate customer support and reduce overhead, thousands of e-commerce brands have deployed Large Language Models (LLMs) to handle frontline inquiries. The premise is seductive: an AI that never sleeps, instantly answers questions, and costs a fraction of a human team. However, a critical vulnerability lies in the very nature of how these models work. "Policy Hallucination" occurs when an AI agent, driven by its training to be helpful and conversational, confidently invents a company policy that does not exist. It might tell a customer they are entitled to a full refund on a non-refundable item, promise a warranty extension that isn't on the books, or invent a discount code that the checkout system will reject.

This is not merely a technical glitch or a "funny AI moment." It is a substantial legal and financial liability. Recent judicial precedents, such as the landmark Moffat v. Air Canada ruling, have established that a chatbot on a company's website is considered a representative of that company. If your bot promises a refund, courts may force you to honor it, regardless of what your Terms of Service page actually says. The argument that "the robot made a mistake" is no longer a valid defense in consumer protection law. For a scaling brand, a single hallucinating bot can bleed revenue through unauthorized concessions and destroy reputation when those concessions are later revoked by human staff.

The danger is amplified by the "black box" nature of generative AI. Unlike old-school decision trees where every response was pre-written by a human, generative models predict the next likely word based on vast internet data. If the model has seen thousands of refund policies online that offer "30-day money-back guarantees," it may statistically predict that your store offers the same, even if you strictly sell final-sale custom goods. Without strict architectural guardrails, the AI defaults to the "average" policy of the internet, not the specific policy of your brand.