AI features need failure design

A model that is right 95% of the time sounds like a strong feature. On a surface used twenty times a day it is also a feature that is confidently wrong in front of a user every single day, and the user has no way to tell which time it was.

That asymmetry — high accuracy, no signal about which case you are in — is the actual design problem. It is not solved by a better prompt.

Show the workings, not the confidence score

A percentage next to an answer is not provenance. It is a number the user has no basis to calibrate against, and it tends to increase trust rather than inform it.

Make review cheaper than redoing it

The value of an extraction feature is not the extraction. It is the time saved versus doing it manually. If verifying the output takes as long as producing it, the feature has moved work rather than removed it.

• Put the source next to the answer, not one click away.
• Let a correction be one keystroke, and remember it.
• Make "I am not sure" a first-class output, not a low score.
• Never let an unreviewed answer silently become an input to something else.

Design the refusal

Every AI feature needs a defined behaviour for the case where it should not answer: input out of scope, source missing, permission absent, confidence genuinely low. Products that skip this do not stop answering — they answer anyway, which is the failure mode with the highest cost and the lowest visibility.

The question to ask before building is not "how good is the model?" It is "what does this do on the day it is wrong, and who finds out?"