Nishaglobal Education logo

Choosing models: build, buy or fine-tune

How to pick models with evidence: selection criteria, hosted APIs versus cloud platforms versus self-hosted open-weight models, when fine-tuning pays off, total cost of ownership, and avoiding lock-in.

Lesson 25 of 4417 min readQuiz: 5 questions

🎯 By the end of this lesson you will be able to:

  • Compare models on quality for your task, cost, speed, capabilities and data terms
  • Choose between hosted APIs, cloud-platform models and self-hosting
  • Decide when to prompt, retrieve, fine-tune or train, and limit lock-in
In this lesson
  1. There is no "best model"
  2. Selection criteria
  3. Where the model runs
  4. Prompt, retrieve, fine-tune or train?
  5. Avoiding lock-in
  6. What this means for your role
  7. Key takeaways
  8. Quiz

There is no "best model"

Public leaderboards are a starting point, not an answer. The right model is the one that meets your quality thresholds on your evaluation set at an acceptable cost, speed and risk. Often that means different models for different jobs: a small, fast model for classification, a capable model for complex reasoning, and an embedding model for search.

Selection criteria

CriterionQuestions to ask
Quality on your tasksHow does it score on our evaluation set, per slice?
CostPrice per input and output token at our volume; any discounts for batch or caching?
SpeedTime to first token and throughput for our typical request size
CapabilitiesContext window, tool calling, structured output, images or audio, languages we need
Data and complianceData retention, use for training, regions where data is processed, certifications
Reliability and supportUptime record, rate limits, deprecation policy, enterprise support
Licence (open-weight models)Is commercial use allowed for our case? Any usage restrictions?

Where the model runs

OptionProsCons
Provider APIFastest start, latest models, no infrastructureData leaves your environment; less control over versions and limits
Cloud AI platformModels available in your existing cloud account, region and security setupModel choice and features can lag direct APIs; cloud lock-in
Self-hosted open-weight modelFull control of data and versions; can be cheaper at very high steady volumeYou run GPUs, scaling, security and updates; often lower capability than top hosted models

Prompt, retrieve, fine-tune or train?

1. Promptingalways start herelowest cost and effort2. Retrieval (RAG)your knowledge, current factslow to medium3. Fine-tuningconsistent behaviour, cheaper small modelmedium to high4. Train from scratchalmost never for typical organisationsvery highcost, data and effort rise
Climb only as far as needed. Most business needs are met by good prompting plus retrieval; fine-tuning and training are for specific, proven gaps.
  1. Prompting: clear instructions, examples and structured output. Always start here.
  2. Retrieval (RAG): when the model needs your knowledge or current facts.
  3. Fine-tuning: when you need consistent behaviour that prompting can't achieve, such as a specific format or style, a narrow classification task, or a smaller, cheaper model trained to match a larger one on your task.
  4. Training from scratch: almost never for typical organisations. It needs vast data, money and expertise.
Fine-tuning is a good fit when…Fine-tuning is the wrong tool when…
You have hundreds or more high-quality examples of the exact behaviourYou want the model to know facts that change (use RAG)
Prompting is close but inconsistentYou haven't tried good prompting and evaluation yet
Volume is high and a smaller tuned model would cut costYou can't maintain the training data and re-tuning

Avoiding lock-in

  • Call models through your own gateway or abstraction layer, not provider SDKs scattered across the codebase.
  • Keep prompts, evaluation sets and tool definitions provider-neutral where possible.
  • Re-run your evaluation set on alternative models periodically, so switching is a measured decision, not a rewrite.
  • Be careful with provider-specific features: use them when the value is clear, and document the dependency.

What this means for your role

Developer

Keep model choice in configuration behind a common interface, so swapping or routing models is a config change plus an evaluation run.

Tester / QA

Run model comparisons on the full evaluation set with per-slice results, cost and latency, and present them side by side.

Business Analyst / PO

Bring the data and compliance constraints, such as residency and retention, into model selection early; they can rule options out.

Tech Lead / Architect

Own the approved model catalogue, deployment choices, total-cost comparisons and the lock-in strategy, documented in ADRs.

Project / Delivery Manager

Expect model choices to change over the product's life. Budget for periodic re-evaluation and migrations, including deprecations.

Scrum Master / Agile

Encourage evidence-based model debates: "what do the evals say?" settles arguments faster than opinions.

Try it: run a mini model comparison

  1. Take 20 cases from an evaluation set (or write them) for one task.
  2. Run them on two different models and score with the same rubric; record cost and response time for each.
  3. Write a short recommendation: which model, for which task, and what would make you revisit the choice.

Key takeaways

  • Choose models with your own evaluation set, cost, speed, capabilities and data terms, not leaderboards alone.
  • Provider APIs, cloud platforms and self-hosting trade speed of start, control and operational burden.
  • Climb the ladder: prompting, then retrieval, then fine-tuning for specific proven gaps; training from scratch rarely.
  • Limit lock-in with a gateway, provider-neutral assets and periodic re-evaluation.

Key terms

Open-weight model
A model whose trained weights are published so it can be run on your own infrastructure, subject to its licence.
Total cost of ownership
All costs of an option over time, including infrastructure, people, support and upgrades.
Lock-in
Dependence on one vendor that makes switching difficult or expensive.
Distillation
Training a smaller model to imitate a larger model's behaviour on specific tasks.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. What is the most reliable way to choose a model for your feature?

  2. 2. The model must know company policies that change monthly. What should you use?

  3. 3. When is fine-tuning a good fit?

  4. 4. What is often underestimated when self-hosting an open-weight model?

  5. 5. Which practice reduces vendor lock-in?

Answer all 5 questions to check.

Practise more