Designing around the failure modes

You cannot make the model reliable. You can build a system that is reliable while containing it — the same way engineering handles every other component with a known failure rate.

1. Constrain the output

An open-ended answer can be anything. An answer that must fit a schema can only be a few things.

Ask for a fixed JSON shape, an item from a list you supplied, or a value in a range — then validate it in code and reject what does not fit. You have converted "might be subtly wrong" into "is either valid or caught".

This is the highest-value change in most applications and it is nearly free.

2. Supply the facts

If the answer depends on your data, put your data in the prompt. Then instruct it to answer only from what you supplied, cite which part, and say it does not know when the documents do not say.

"Say you don't know" works reasonably well when the alternative is visibly absent from the context. It works poorly as a general instruction, because the model has no reliable sense of what it does not know.

3. Let it call things that actually know

For anything factual, current or computed — a price, a stock level, a sum, today's date — give it a tool rather than expecting recall.

The division: the model chooses what to do; your code does it. That boundary is where correctness lives, and it is also where your permissions live. A tool that reads is safe to expose. A tool that writes needs the same scrutiny as any endpoint.

4. Check it with something that is not it

Validate in code where you can. Where you cannot, a second call with a different, narrower question catches some errors — but be honest that correlated failures exist, and that two calls to the same model are not two independent opinions.

For anything that matters, keep a test set of real inputs with known-good outputs and run it when you change the prompt. A prompt is code. It regresses like code and deserves tests like code.

What does not work

• "Be accurate." "Do not hallucinate." "Be careful." These add tokens and change nothing. The model is not withholding accuracy pending your request.
• Asking it to grade its own answer. You get the shape of a confident self-assessment, which is what you would have got anyway.
• Threats, urgency, or promised rewards. These circulate widely. Test them against a real evaluation set before believing any of it.

The habit

When something goes wrong, the instinct is to add a sentence to the prompt. Ask first whether the problem should be handled by a schema, a retrieved document, a tool, or a check — because those hold, and a sentence usually just moves the failure somewhere you are not looking.