Key takeaways
- Pick by cost per completed task at your required quality, not by benchmark headlines. A model that is 20% cheaper per token but needs two tries is more expensive.
- Build a 100-example evaluation set from your real data before you compare anything. Without it every vendor demo looks good.
- For most Indian business workloads (support, extraction, classification, drafting) a mid-tier model is the sweet spot; reserve frontier models for reasoning-heavy steps and route between them.
- Open-weight models make sense when data must stay on-prem, volume is high and flat, or latency to a foreign region is unacceptable — and only if you have the GPU and ops skills to run them.
- Keep an exit: abstract the model behind one interface, log every prompt/response, and re-run the eval set quarterly.
1. Start from the task, not the model
Write down the task in one line with a measurable output: "Classify incoming WhatsApp messages into 6 intents with ≥95% accuracy", "Extract 12 fields from GST invoices with ≤2% field error", "Draft support replies that agents accept unedited ≥60% of the time".
Then collect 100 real examples with the correct answer. This is the evaluation set. Everything else is opinion.
2. The nine criteria
| # | Criterion | How to measure |
|---|---|---|
| 1 | Task quality | Score on your eval set (accuracy / human acceptance) |
| 2 | Cost per task | (input + output tokens × price) × retries, over 100 tasks |
| 3 | Latency | p50 and p95 time to first token and to completion, from Indian locations |
| 4 | Context window | Does your longest realistic input fit with room for instructions? |
| 5 | Tool / structured output | Does it reliably return valid JSON and call tools? Failure rate over 100 runs |
| 6 | Data handling | Training opt-out, retention period, region of processing, DPA available |
| 7 | Hosting options | API only, private deployment, open weights |
| 8 | Availability in India | Regional endpoints, billing in INR, support hours |
| 9 | Exit cost | How much code/prompts change if you switch? Logged data to re-evaluate? |
Weight them for your case (a hospital weights 6 and 7 heavily; a marketing tool weights 1 and 2).
3. Tiers, and when each is right
- Frontier models (highest reasoning): complex multi-step agents, code generation, legal/financial analysis. Expensive; use for the step that needs it, not the whole pipeline.
- Mid-tier models: support assistants, RAG answers, extraction, classification, drafting. Best price/quality for 80% of business tasks.
- Small models (including small open-weight): intent classification, routing, PII detection, on-device or high-volume batch. Fine-tune them on your eval set and they often beat bigger models on narrow tasks.
- Open-weight on your hardware: when data residency or volume demands it. Budget for GPUs (a single modern 24–80 GB GPU server in India runs ₹4–15 lakh), inference tooling, and someone to keep it patched.
4. Routing beats choosing
Most production systems we build use two or three models: a small one to classify and route, a mid-tier one to answer, a frontier one for the hard 5%. Cost drops 60–80% versus "frontier for everything" with no measurable quality loss on the eval set.
5. Things that go wrong
- Comparing models on your prompts that were tuned for one of them. Re-tune briefly per model.
- Ignoring retries and timeouts in cost.
- Testing from the US and deploying in India (latency doubles).
- No logs → no way to find out why last week's answers were wrong.
- Letting prompts live in ten places in the code.
6. Re-evaluate quarterly
Prices and models change fast. Keep the eval set, the harness and the logs; re-run every quarter; switch when a model wins on cost per task by a clear margin. The decision process is the asset, not the model name.
We run this evaluation as a two-week engagement and deliver the harness, the scorecard and a recommendation — see AI & development.