Choosing what to point it at
A test for whether a task is a good fit, and the categories that reliably are and are not.
Most failed AI projects were not built badly. They were pointed at the wrong task, and no amount of prompt engineering rescues that.
The test
Ask two questions about the task:
- Can a competent person verify the output quickly?
- What is the cost of a wrong one getting through?
Fast to verify and cheap to be wrong → excellent fit, ship it. Fast to verify and expensive to be wrong → good fit with a human approving. Slow to verify and cheap to be wrong → workable, but measure it. Slow to verify and expensive to be wrong → do not build this.
That last quadrant is where the disasters live. It is also, unfortunately, where the most impressive demos live, because nobody in a demo verifies anything.
Reliably good
- Transformation. Summarising, reformatting, translating, extracting structure from prose. The source is right there, so checking is fast.
- Drafting. First versions a person will edit. The bar is "better than a blank page", which is a low bar honestly met.
- Classification and routing. With clear categories and examples, and a measurable accuracy you can actually track.
- Explaining. Code, documents, error messages — into simpler words, at a level you specify.
- Interfaces to your own data. Turning a question into a query against a system that holds the truth.
Reliably bad
- Being the source of a fact. Anything where the answer is not in the input.
- Arithmetic at any scale. Give it a calculator as a tool instead.
- Anything irreversible and unattended. Sending, paying, deleting, publishing.
- Being the only check on its own work. Asking it "is that correct?" gets you a prediction of what a confirmation looks like.
The most common mistake
Building the impressive version first.
The unglamorous version — extract these six fields from this document, into this JSON shape — ships, works, and is verifiable. The autonomous agent that handles the whole workflow demos beautifully and then spends six months failing in ways nobody can reproduce.
Ship the boring one. It is usually most of the value, and it teaches you what the ambitious one would actually need.