The demo always goes well. Someone types the question everyone rehearsed, the answer arrives, the room nods. Three weeks later the feature is behind a flag because nobody can tell whether a change made it better or worse.
Build the evaluation set before the feature
Fifty real questions with agreed-upon good answers will tell you more than any amount of manual clicking. Collect them from support tickets, sales calls, the messages people actually send. Run them on every change.
Without this, prompt engineering is superstition. You change a line, it feels better, you keep it. An evaluation set turns that into a number that moves.
Route models by task, not by loyalty
- Classification and extraction rarely need your most expensive model
- Reasoning over long context does — send only those requests there
- Cache aggressively; identical questions should not be paid for twice
- Measure cost per resolved request, not cost per token
Most bills we are asked to reduce shrink by half from routing alone, with no measurable quality loss. The expensive model was doing work a cheap one handled identically.
Design for the failure case
The model will time out, refuse, or return something malformed. Decide now what the user sees when that happens. A feature that degrades to a clear message is fine; one that spins forever is not.
Assume the model is a flaky third-party API, because that is exactly what it is.
Log enough to investigate later
When someone reports a bad answer next month, you need the input, the retrieved context, the model version, and the output. Without those four you are guessing, and the fix becomes another prompt tweak nobody can verify.
None of this is glamorous. It is the difference between a feature that ships and a demo that gets remembered fondly.
