Designing AI Products People Trust

Models are now good enough to do real work and still wrong often enough to cause real damage. The products that last are designed around both facts at once. Users decide whether to trust you by watching what your product does when it fails, what it costs them, where their data goes, and whether it admits to being a machine.

Product or Feature

Start by deciding what role AI plays. If users pay for the model's output, AI is the product; if the model speeds up a workflow they already have, it is a feature. Each calls for a different design.

When AI is the product, output quality is your whole value, so evaluation, failure handling, and model choice are core engineering work. When AI is a feature, the surrounding workflow carries most of the value, and users should be able to ignore the AI when it gets something wrong.

To tell which you are building, imagine removing the model. If nothing useful remains, you are building an AI product and have to be very good at it. If a solid tool remains, keep the AI in a supporting role and resist putting it in front of every interaction.

Also ask whether the problem needs a model at all. Rules, search, and plain forms cost less, respond faster, and behave predictably. Use AI where the input is messy, written in natural language, or too varied for rules to cover.

Design for Being Wrong

Every model will sometimes produce output that is confident, plausible, and wrong. NIST's 2024 Generative AI Profile calls this confabulation, "confidently stated but erroneous or false content," and describes it as a natural result of how generative models work. Your product's job is to make those failures cheap to catch and easy to see.

Courts will not treat the model as someone else's problem. In 2024 a Canadian tribunal held Air Canada liable for its website chatbot's wrong answer about bereavement fares, rejecting the airline's argument that the chatbot was responsible for its own actions. Whatever your product tells a customer, you told them.

Match the safeguards to the cost of a mistake:

• Low stakes (drafting, summarizing, suggesting): show the output, make it easy to edit, and let users regenerate.
• Medium stakes (sending messages, changing records): require a review step before anything leaves the product, and offer undo.
• High stakes (money, legal, medical, irreversible actions): keep a human in the loop on every decision and show the sources or reasoning behind the output.

Some patterns help at every level. Show where an answer came from, so users can check it. Mark clearly which content the model generated and which came from the user's own data. Make editing an output as easy as accepting it. Microsoft's 18 Guidelines for Human-AI Interaction, tested against 20 AI products in a 2019 study, include these: make clear how well the system can do what it does, support efficient dismissal, support efficient correction, and make clear why the system did what it did. Undo is among the most trust-building features you can ship, because it lowers the cost of letting the system try.

Be careful with confidence indicators. A model's stated confidence is not always well calibrated, and a 2025 paper by Kalai and colleagues argues that standard training and benchmarks reward models for guessing rather than admitting uncertainty. Google's People + AI Guidebook notes that confidence displays can help users calibrate trust but should be tested, since numeric ones assume users understand probability. Prefer signals you can check, such as whether an answer is grounded in a cited source.

Evals Before Prompts

Most teams start by tuning prompts against a handful of examples they remember. Each change fixes one case and quietly breaks another. Build an evaluation set from real cases before you start tuning.

A practical approach:

1. Collect real inputs from users, support tickets, or your own manual work. A few dozen is enough to start.
2. For each, write down what a good output looks like, or the rules a good output must follow.
3. Add hard cases deliberately: ambiguous requests, missing data, hostile inputs, unusual formats.
4. Score outputs automatically where you can, using code checks for structure and a model-based grader for judgment calls, then spot-check by hand.
5. Run the set every time you change a prompt, a model, or a retrieval step.

When something fails in production, add that case to the set. Over time it becomes the most valuable asset in your AI stack, because it lets you change models or prompts and know whether you made things worse.

Cost, Latency, and Lock-In

Every inference request costs money and takes noticeable time. Both belong in the product design from the start, alongside features and flows.

Constraint	Design responses
Cost per request	Use smaller models for simple steps, cache repeated work, limit context to what is needed
Latency	Stream output, do slow work in the background, show progress that reflects real steps
Unpredictable usage	Set per-user limits, price so heavy users do not lose you money

Know your cost per core action and compare it with what a user pays. Free tiers with unlimited AI usage can make growth expensive. OWASP lists unbounded consumption among its top ten LLM risks, including "denial of wallet" attacks that run up your inference bill, and its first recommendation is rate limits and per-user quotas. Route easy tasks to cheaper, faster models and save the most capable ones for the steps that need them. Your eval set tells you when a cheaper model is good enough.

Model prices and quality change quickly. Stanford's 2025 AI Index found the inference cost of a system performing at the level of GPT-3.5 dropped over 280-fold between November 2022 and October 2024. The provider that suits your task today may not suit it next year, so design for switching:

• Put model calls behind one internal interface instead of calling a provider's SDK from all over the codebase.
• Keep prompts in versioned files rather than scattered through code.
• Store inputs and outputs, with consent, so you can replay them against a new model.
• Avoid depending on features only one provider offers unless they are central to your product.

The eval set is what makes a switch safe: you can measure a new model against it before any user sees the change.

Agents That Take Actions

Once your AI can act as well as suggest, the stakes rise. An agent that sends email, edits records, spends money, or runs code needs the controls you would give a new employee, and usually more. The OWASP Top 10 for LLM Applications names this risk excessive agency: too many functions, too many permissions, or too much autonomy.

• Least privilege: give the agent only the permissions the task requires, scoped to the user it acts for. Enforce those permissions in your own systems; never ask the model to decide what is allowed.
• Confirmation for consequential steps: show what the agent is about to do and let the user approve it, especially for anything external or irreversible.
• Audit logs: record every action, the input that led to it, and who approved it, in a form both you and your users can read.
• Reversibility: prefer drafts, staging, and soft deletes, so most actions can be rolled back.
• Limits: cap spending, volume, and run time, so a loop or a bad instruction cannot run away.

Treat any content the agent reads, such as web pages, emails, or uploaded documents, as untrusted. Instructions hidden in that content can try to redirect the agent. OWASP calls this indirect prompt injection, ranks it first on its 2025 list, and says it is unclear whether any method prevents it completely. Design for containment instead: read access should never silently grant write access, and privileged actions should wait for human approval.

Privacy and Honesty

Users trust you with data that you then pass to a model provider, so know where it goes. Read your providers' terms on data retention and training, choose settings that match what you promise users, and state plainly in your privacy policy which data goes to third-party models. Strip or minimize personal data the task does not need. Enterprise buyers ask about this early, and their caution has precedent: in May 2023 Samsung barred staff in one of its biggest divisions from using ChatGPT and similar tools after what appeared to be an accidental leak of internal source code. Have your answers ready before the first security questionnaire arrives.

Be honest about what is automated. People should know when they are reading machine output or talking to a bot, and should be able to reach a human when it matters. Label AI-generated content, do not present automated replies as personal ones, and do not overstate what the system can do. Some jurisdictions now require this. The EU AI Act's transparency rules, which the European Commission says came into effect in August 2026, require that people be told when they are interacting with an AI system unless it is obvious, and that generated content be marked as artificial in a machine-readable way. Check the disclosure and automated-decision rules in each market you serve. Disclosure also protects you: users who know a draft came from a model check it, while users who find out they were misled stop trusting the rest of your product.