Clients ask us for "AI" the way they used to ask for "an app" — as a category rather than a solution to a stated problem. The useful conversation starts one question later: what is the task, how often does it happen, and what does being wrong cost?
Those three answers decide almost everything. Below is roughly how we sort it, based on where it has genuinely paid off in our own delivery work and where it has not.
The question that decides it
What does a wrong answer cost, and who notices?
If a wrong answer is caught immediately by the person who asked, cheaply, then speed matters more than accuracy and AI is usually a win. If a wrong answer is silently absorbed into a system and discovered three weeks later by a customer, the economics invert completely — you now need review, audit trails and a fallback, and those often cost more than the automation saves.
Automate the tasks where being wrong is obvious and cheap. Be very careful about the ones where being wrong is invisible.
Where it consistently pays
First drafts of structured things
Test scaffolding, migration scripts, type definitions from a sample payload, boilerplate for a component that follows an existing pattern. The output is verifiable in seconds — it compiles or it does not, the test passes or it does not — and the reviewer is the person who asked for it.
This is the clearest win we have measured. Not because the generated code is especially good, but because the cost of checking it is near zero and the cost of starting from an empty file is not.
Reading, more than writing
The genuinely underrated use. Handed an unfamiliar 40,000-line codebase during an audit, asking questions of it — where is authentication handled, what touches this table, which of these two modules is dead — collapses the orientation phase from days to hours.
Every answer still gets verified against the actual code. But verifying a specific claim is far faster than forming the hypothesis in the first place, and that is the part the model does well.
Classification at volume, with a threshold
Routing support tickets, tagging content, flagging anomalies. These work when two conditions hold: you have a test set drawn from real data, and you have agreed a confidence threshold below which a human decides instead.
On the triage system we built for a support desk, roughly 72% of tickets clear the threshold and route automatically. The remaining 28% go to a person — which was the point. The value was never "no humans"; it was "humans only on the hard ones".
Build the threshold before the automation The order matters. Teams that ship the automation first and add confidence handling after an incident end up retrofitting it around a system that assumed it was always right.
Where it quietly costs you
Architectural decisions
Ask for an architecture and you will get a plausible, conventional one that ignores the three constraints that actually make your situation specific — the team's existing skills, the system you cannot replace, and the deadline that is not moving. It reads well in a document and falls apart in the first sprint.
Useful as a checklist of options you might not have considered. Not useful as the decision.
Debugging anything stateful
A race condition, a cache invalidation bug, an intermittent failure that depends on request ordering. These require holding the actual runtime state in your head, and a confident wrong hypothesis is worse than no hypothesis — it sends you down a path and costs you the hour you would have spent reading the log properly.
Code that is 95% right
The dangerous zone. Generated code that is obviously wrong is free — you discard it. Generated code that is subtly wrong in one branch, in a file you skimmed because the rest looked fine, is expensive. It passes review precisely because it looks like the code around it.
This is why we do not relax code review for generated code. If anything, it gets read more carefully, because the usual signal that something was hard to write — and therefore deserves attention — is absent.
How we scope an AI feature
When a client asks for one, the first two weeks are not building. They are:
- Assemble a test set from real data. A hundred genuine examples with known-correct answers. If this cannot be produced, that is itself the finding — you cannot evaluate what you cannot measure.
- Measure the boring baseline. Keyword matching, a rules table, a lookup. Surprisingly often it scores within a few points of the model at a fraction of the cost and complexity.
- Measure the model against the same set. Same data, same scoring, agreed in advance.
- Project the cost at real volume. Per-request pricing multiplied by actual throughput, not by demo throughput. This has killed more features at this stage than accuracy has.
- Decide the fallback. What the product does when the model is unavailable, slow, or below threshold. If there is no acceptable answer, the feature does not ship.
Roughly a third of the feasibility studies we have run ended with a recommendation not to build the AI feature. Clients have generally been happier with that outcome than with a working prototype whose running cost only became visible in month four.
The honest summary
AI has changed how fast our team gets oriented and how quickly the first draft of something structured appears. It has not changed how long it takes to understand a problem properly, and it has not reduced the need for review — it has increased it, because more code arrives per hour and it all still has to be read.
The teams getting the most from it are the ones treating it as a fast, tireless, slightly unreliable colleague whose work you check. The ones getting the least are treating it as an oracle, and paying for that in defects they find late.