The Silent Majority

When an AI system fails and the user complains, the failure is visible. It appears in feedback logs, complaint tickets, thumbs-down ratings. The system's developers can find it, categorize it, and fix it. Visible failures drive improvement.

When an AI system fails and the user says nothing, the failure is invisible. The user moves on, works around the error, abandons the task, or doesn't realize the output was wrong. No signal reaches the developer. The failure persists.

A large-scale analysis of the WildChat dataset (arXiv:2603.15423) finds that 78% of AI failures are invisible. In more than three out of four cases where something went wrong, the user gave no overt indication that there was a problem.

The taxonomy identifies eight failure archetypes. The most striking finding: 91% of invisible failures involve interactional dynamics — how the system and user communicate — rather than capability limitations. The model can do the task. It fails at the exchange. It misunderstands the request, provides technically correct but practically useless responses, or answers a question the user didn't ask. The capability is present. The interaction is broken.

The second finding is worse: 94% of invisible failures would persist even with more advanced models. Better capability doesn't fix interactional failures because the failures aren't caused by insufficient capability. They are caused by the structure of the interaction itself — ambiguous requests that the model resolves incorrectly, implicit expectations that the model doesn't share, conversational context that the model loses or never had.

The implication for evaluation: standard benchmarks measure capability. They measure whether the model can do the task under ideal conditions. But the failures that matter most in deployment — the ones that persist, that users don't report, that more powerful models won't fix — are failures of interaction, not capability. Measuring the right thing (can it do the task?) tells you almost nothing about the dominant failure mode (does it do the right task in the right way for this specific user at this specific moment?).

The silence is not consent. It is data — the most important kind, and the kind that current evaluation frameworks are designed not to hear.