Choosing models: build, buy or fine-tune
How to pick models with evidence: selection criteria, hosted APIs versus cloud platforms versus self-hosted open-weight models, when fine-tuning pays off, total cost of ownership, and avoiding lock-in.
🎯 By the end of this lesson you will be able to:
- Compare models on quality for your task, cost, speed, capabilities and data terms
- Choose between hosted APIs, cloud-platform models and self-hosting
- Decide when to prompt, retrieve, fine-tune or train, and limit lock-in
In this lesson
There is no "best model"
Public leaderboards are a starting point, not an answer. The right model is the one that meets your quality thresholds on your evaluation set at an acceptable cost, speed and risk. Often that means different models for different jobs: a small, fast model for classification, a capable model for complex reasoning, and an embedding model for search.
Selection criteria
| Criterion | Questions to ask |
|---|---|
| Quality on your tasks | How does it score on our evaluation set, per slice? |
| Cost | Price per input and output token at our volume; any discounts for batch or caching? |
| Speed | Time to first token and throughput for our typical request size |
| Capabilities | Context window, tool calling, structured output, images or audio, languages we need |
| Data and compliance | Data retention, use for training, regions where data is processed, certifications |
| Reliability and support | Uptime record, rate limits, deprecation policy, enterprise support |
| Licence (open-weight models) | Is commercial use allowed for our case? Any usage restrictions? |
Where the model runs
| Option | Pros | Cons |
|---|---|---|
| Provider API | Fastest start, latest models, no infrastructure | Data leaves your environment; less control over versions and limits |
| Cloud AI platform | Models available in your existing cloud account, region and security setup | Model choice and features can lag direct APIs; cloud lock-in |
| Self-hosted open-weight model | Full control of data and versions; can be cheaper at very high steady volume | You run GPUs, scaling, security and updates; often lower capability than top hosted models |
Prompt, retrieve, fine-tune or train?
- Prompting: clear instructions, examples and structured output. Always start here.
- Retrieval (RAG): when the model needs your knowledge or current facts.
- Fine-tuning: when you need consistent behaviour that prompting can't achieve, such as a specific format or style, a narrow classification task, or a smaller, cheaper model trained to match a larger one on your task.
- Training from scratch: almost never for typical organisations. It needs vast data, money and expertise.
| Fine-tuning is a good fit when… | Fine-tuning is the wrong tool when… |
|---|---|
| You have hundreds or more high-quality examples of the exact behaviour | You want the model to know facts that change (use RAG) |
| Prompting is close but inconsistent | You haven't tried good prompting and evaluation yet |
| Volume is high and a smaller tuned model would cut cost | You can't maintain the training data and re-tuning |
Avoiding lock-in
- Call models through your own gateway or abstraction layer, not provider SDKs scattered across the codebase.
- Keep prompts, evaluation sets and tool definitions provider-neutral where possible.
- Re-run your evaluation set on alternative models periodically, so switching is a measured decision, not a rewrite.
- Be careful with provider-specific features: use them when the value is clear, and document the dependency.
What this means for your role
Developer
Keep model choice in configuration behind a common interface, so swapping or routing models is a config change plus an evaluation run.
Tester / QA
Run model comparisons on the full evaluation set with per-slice results, cost and latency, and present them side by side.
Business Analyst / PO
Bring the data and compliance constraints, such as residency and retention, into model selection early; they can rule options out.
Tech Lead / Architect
Own the approved model catalogue, deployment choices, total-cost comparisons and the lock-in strategy, documented in ADRs.
Project / Delivery Manager
Expect model choices to change over the product's life. Budget for periodic re-evaluation and migrations, including deprecations.
Scrum Master / Agile
Encourage evidence-based model debates: "what do the evals say?" settles arguments faster than opinions.
Try it: run a mini model comparison
- Take 20 cases from an evaluation set (or write them) for one task.
- Run them on two different models and score with the same rubric; record cost and response time for each.
- Write a short recommendation: which model, for which task, and what would make you revisit the choice.
Key takeaways
- Choose models with your own evaluation set, cost, speed, capabilities and data terms, not leaderboards alone.
- Provider APIs, cloud platforms and self-hosting trade speed of start, control and operational burden.
- Climb the ladder: prompting, then retrieval, then fine-tuning for specific proven gaps; training from scratch rarely.
- Limit lock-in with a gateway, provider-neutral assets and periodic re-evaluation.
Key terms
- Open-weight model
- A model whose trained weights are published so it can be run on your own infrastructure, subject to its licence.
- Total cost of ownership
- All costs of an option over time, including infrastructure, people, support and upgrades.
- Lock-in
- Dependence on one vendor that makes switching difficult or expensive.
- Distillation
- Training a smaller model to imitate a larger model's behaviour on specific tasks.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. What is the most reliable way to choose a model for your feature?
2. The model must know company policies that change monthly. What should you use?
3. When is fine-tuning a good fit?
4. What is often underestimated when self-hosting an open-weight model?
5. Which practice reduces vendor lock-in?