17th November 2026 · 18 min
RAG vs Fine Tuning: Cost, Accuracy and When to Use Each
The most repeated advice on rag vs fine tuning is "start with retrieval, fine-tune later". It sounds cautious. It is also the wrong default for a large share of UK products, because it treats the choice as an accuracy contest and ignores the bill, the audit trail, and what fails in production.
Retrieval-augmented generation (RAG) changes what the model sees at the moment of answering. Fine-tuning changes how the model behaves. Those are different jobs. A support assistant that invents a refund rule has a knowledge gap. An assistant that knows the rule but emits the wrong JSON has a behaviour gap. Training does not reliably fix the first, and a document index does not fix the second.
UK teams are already putting language systems into compliance, learning, fintech and support. A poor architecture shows up as stale answers, personal data that nobody can delete, or a retraining cycle every time a policy moves. The comparison below stays on mechanism, money, accuracy and the smallest auditable footprint, and it stays neutral until the decision order at the end.
Table of Contents
- How Each Works and Where They Differ
- Cost, Data Needs and Maintenance Over Time
- Accuracy and Why Retrieval Quality Often Decides the Result
- Data Governance and the Smallest Auditable Footprint
- Use Cases Where Each Approach Fits Best
- Choosing Your Starting Point and Confirming It
How Each Works and Where They Differ
RAG and fine-tuning are different engineering strategies, not interchangeable versions of one technique. RAG leaves a foundation model's learned parameters unchanged and supplies relevant external information at inference time, typically by searching a document collection and inserting selected passages into the prompt. Fine-tuning updates those parameters with task-specific examples. It is better suited to behaviour, formatting, classification and domain language, and worse suited to facts that keep moving. The Artificial Intelligence Sector Study published in 2024 identified more than 5,800 AI companies in the UK, an 85% increase over two years, with sector revenue of £23.9 billion and gross value added of £11.8 billion. That commercial base is why the architecture is a product decision, not a lab preference.

| Criterion | RAG | Fine-tuning |
|---|---|---|
| How it works | Retrieves passages from a document index and adds them to the prompt; model weights unchanged | Updates model weights using task-specific examples |
| Data needed | Existing documents that are cleaned and permission-controlled | Curated, labelled training examples |
| Handling changing facts | Update the index; no retraining needed | Retrain the model to reflect new facts |
| Controlling format and tone | Limited to prompt instructions | Strong, consistent behaviour across requests |
| Source traceability | Retrieved passages can be cited and inspected | Answers are not linked to a source by default |
What each criterion decides
Mechanism decides where the knowledge or the habit lives. In RAG it lives in the index. In fine-tuning it lives in the weights.
Data needed decides whether you can start from contracts and handbooks you already hold, or whether you must write hundreds of ideal input-output pairs first.
Changing facts decide the maintenance path. A policy edit is a re-index, or it is a new training run.
Format and tone decide whether a prompt is enough. A fixed JSON schema, a specialist intent taxonomy, or a house style that must hold on every request is a weights problem.
Source traceability decides whether an auditor can open the passage the model used. Retrieved text can be cited. A weight update cannot point at a clause by default.
Two pipelines, then a hybrid
The RAG path runs in a fixed order.
- The question is turned into a search query, often an embedding, sometimes mixed with keyword search for exact identifiers.
- The most relevant passages are pulled from an indexed collection.
- A reranker or a top-k cut narrows that set.
- The surviving passages go into the prompt, and the model answers from them.
The base model is unchanged. If Tuesday's handbook replaces Monday's, the next answer can follow the new text without a training job.
The fine-tuning cycle is a different loop.
- Assemble input-output examples that show the behaviour you want.
- Train, which updates model parameters, often through a small adapter rather than every weight.
- Evaluate the new version on held-out cases.
- Deploy it as a new model version, with version control and a rollback.
Learned behaviour then sits in the weights, so style, structure and classification hold more reliably than a long instruction. Facts absorbed there are hard to correct and hard to trace. You cannot open the model and delete one outdated clause.
Practical rule: If the failure is "the model does not know this week's document", retrieval is the tool. If the failure is "the model knows the document and still writes the wrong shape", training is the tool.
In practice the two are complementary. A hybrid keeps a fine-tuned model for format and terminology, and uses retrieval for current facts. For compliance, learning, fintech and support work, RAG is generally the more maintainable start when answers must reflect current policies, contracts, product documentation or regulatory material, because source documents update without retraining and passages can be inspected. That pattern is the one behind a custom AI chatbot grounded in approved business material. Fine-tuning becomes the lead when the requirement is consistent behaviour across many requests.
Before either build, answer three questions. How often does the knowledge change? Does every answer need a source? Must the output follow a fixed pattern?
Cost, Data Needs and Maintenance Over Time
The useful question is not which method is cheaper to build. It is which one stays cheapest after the data moves, people change jobs, and an auditor asks why an answer was allowed. For UK firms, the decision is a total-cost and governance decision: spend is spread across evaluation, data preparation, hosting, monitoring, failure handling and the effort needed to prove control.
Adoption is broad enough that this choice now reaches ordinary teams, but it is uneven. Government adoption research describes recurring barriers around trust, cost and expertise in the artificial intelligence adoption research report. The earlier activity study estimated that about 15% of businesses, approximately 432,000 companies, had adopted at least one AI technology, with adoption at 68% of large companies, 34% of medium-sized companies and 15% of small companies, while natural-language processing and generation had been adopted by 8% of UK firms, according to the AI activity in UK businesses executive summary. Many of those firms need language systems and do not hold a large labelled set.
| Cost item | RAG | Fine-tuning |
|---|---|---|
| Initial data preparation | Cleaning, chunking and permissioning documents | Assembling and labelling training examples |
| Model training | None; the base model is used as it is | Training runs on the chosen base model; cost varies with model and data volume |
| Recurring infrastructure | Vector storage, embedding refreshes, retrieval serving | Hosting a tuned model version and running inference |
| Cost of a knowledge change | Re-index the changed documents | Curate new examples and retrain |
| Evaluation | Score retrieval recall, groundedness and answer accuracy | Score task accuracy and check for memorised data |
| Prompt and context overhead | Retrieved passages add input tokens to each query | Often shorter prompts once behaviour is learned |
Where the bill actually lands
RAG avoids a training run. It does not avoid labour. Someone cleans documents, chooses a chunk size, embeds them, refreshes embeddings when the text changes, pays for vector storage, and re-scores retrieval when answers drift. Each query also carries extra input tokens for the passages, plus the latency of the lookup before generation starts.
Fine-tuning front-loads that effort. You curate and label examples, run training, keep versions, and check that the model has not memorised confidential rows. Once the behaviour is learned, prompts are often shorter, so per-request token cost can fall. The bill returns when the rules change, because you curate again and retrain.
The governance line can flip the answer. A poorly governed retrieval index can expose more than a tightly scoped fine-tune, because every connected share, stale permission and forgotten draft becomes answerable context. A narrow fine-tune on reviewed examples has fewer moving parts, provided confidential rows are excluded and each version can be rolled back.
A scorecard, then two sketches
Score a UK SME on six items, not on a public benchmark. Update frequency. Query volume. Data sensitivity. Whether citations are required. Workflow stability. Internal engineering capacity.
Say a policy team publishes about twenty guidance changes a month, and every answer must name the clause. RAG wins on the change line. Re-indexing those documents is cheaper than rebuilding a training set twenty times.
Say a triage system sorts about 50,000 tickets a month into a fixed set of categories and a strict schema, and the categories barely move. Fine-tuning can be the cheaper design. The behaviour is stable, volume makes short prompts matter, and retrieval would add tokens you do not need for facts.
A fine-tune-first design can therefore be cheaper and more reliable when knowledge changes slowly and the problem is behaviour rather than factual access. "Start with RAG" skips that arithmetic.
Accuracy and Why Retrieval Quality Often Decides the Result
Accuracy belongs to the whole system, not to the badge on the method. In a benchmark covering 10,195 chunks of UK legal material and 7,901 synthetic queries, the strongest dense embedding model, Voyage-3-Large, reached a mean reciprocal rank of 0.826 and retrieved the correct chunk within the top 15 results for 99.3% of queries, with dense embeddings substantially ahead of BM25, as set out in the UK legal retrieval study. That is a retrieval result. It is not a promise that the generated sentence was right.
A high hit rate still leaves the generator free to misread a passage, drop a condition, or answer from older training instead of the text in front of it. Score these separately: retrieval recall, groundedness, factual accuracy, completeness, refusal behaviour, latency and cost. One blended score hides which part regressed.
The UK Government's AI Playbook states that GOV.UK Chat chose RAG over fine-tuning because GOV.UK content is regularly edited and expanded. In its evaluation the system achieved 80% answer accuracy, and 69% of users rated it useful or very useful. Treat that 80% as a GOV.UK-specific pilot with expert assessment and a defined content domain, not as a universal RAG benchmark. Build an evaluation set from UK queries your users already send before you lock an architecture. A human-in-the-loop review is how those scores stay honest after launch.
Fine-tuning can tighten style and task execution. It cannot stand in for an index that reflects newly amended source documents. For a SaaS, compliance or support product whose answers must track current policies, product documentation or regulations, version the source corpus, enforce access controls at retrieval time, and measure groundedness, factual completeness, citation accuracy and answer rate.
Tune retrieval before you train
Change one layer at a time, in this order: chunking, metadata filters, the embedding model, reranking, then top-k. Only after those moves plateau should you spend on a behaviour fine-tune.
| Symptom | Likely cause | Fix |
|---|---|---|
| Answer uses an outdated policy | Index not refreshed or stale chunks | Re-index on change and version the source corpus |
| Correct passage retrieved but the answer is wrong | Model misreads or omits part of the passage | Improve the prompt, add reranking, and score completeness |
| Right facts but the wrong output format | Format behaviour not learned | Fine-tune for the format, keep retrieval for facts |
| Relevant passage never retrieved | Poor chunking, weak embeddings or missing metadata filters | Adjust chunk size, filters and the embedding model |
| Confident answer with no supporting source | Weak groundedness checks | Require citations and refusal when no passage matches |
The third row is the hybrid case in miniature. Retrieval supplied the facts. The format still failed. Training the format, while leaving the facts in the index, fixes the failure you measured. The legal benchmark is domain-specific and partly synthetic, so production validation still needs expert-labelled queries from your own users.
Data Governance and the Smallest Auditable Footprint
Most rag vs fine tuning write-ups stop at accuracy. For a UK controller the sharper question is which design leaves the smallest auditable data footprint.

Fine-tuning can embed personal or confidential information into model behaviour. Once it is in the weights, you cannot point at the row, and deletion is not a file operation. RAG can keep answers tied to a controlled, updateable source. That does not make the index harmless. You still store, chunk, embed, cache and retrieve material that may be personal data, and a prompt can be steered into revealing a passage the user should never have seen.
What the ICO expects, and what copyright does not excuse
The Information Commissioner's Office says organisations processing personal data to train foundation models must establish a valid lawful basis, assess necessity, evaluate impacts on individuals, and consider less intrusive alternatives such as synthetic data. The same discipline applies when you decide whether a training set is justified at all.
Copyright cuts both ways. Reproduction of protected works can arise in fine-tuning and in building a RAG corpus. The UK has no broad commercial text-and-data-mining exception, and it has no US-style fair-use rule. "We only embedded it" and "we only indexed it" are not licences.
A short briefing on AI agent security is worth reading beside the checklist below, because retrieval expands the attack surface as well as the knowledge.
Score both designs on the same eight items
- Lawful basis: Name the basis for each dataset before indexing or training, and record why a less intrusive option was rejected.
- Retention: Set how long chunks, embeddings, caches and training files are kept, and who can extend that period.
- Deletion: Prove you can remove a document from the index, and state what you cannot remove from a trained weight file.
- Subject access: Show how you would find one person's data in the corpus or admit you cannot find it inside a fine-tune.
- Source licensing: Confirm you may store and retrieve each work, not only that you possess a PDF.
- Tenant isolation: Keep one customer's passages out of another customer's retrieval results.
- Prompt-injection exposure: Assume retrieved text can contain instructions, and test that the model does not follow them.
- Auditability: Log which passages were retrieved, or accept that a fine-tuned answer has no source trail.
RAG is not automatically the compliant choice. A poorly governed retrieval index, with stale permissions and a shared cache, can create more operational exposure than a narrowly scoped fine-tune trained on licensed, minimised data.
Use Cases Where Each Approach Fits Best
Match the starting point to the gap, not to the trend. The table is a starting verdict for common UK work. Edge cases below it are allowed to overturn a row.
| Use case | Better starting point | Why |
|---|---|---|
| Policy and compliance Q&A from changing guidance | RAG | Documents change often and answers need citations |
| Internal knowledge assistant over product documentation | RAG | The index updates without retraining and access controls apply at retrieval |
| Support ticket triage into fixed categories | Fine-tuning | Consistent classification across many requests |
| Structured output in a fixed JSON schema | Fine-tuning | The format must hold across every request |
| Branded tone for customer messages | Prompting first, then fine-tuning | Style is behavioural and fairly stable |
| Legal or clinical drafting with specialist terminology | Hybrid | Fine-tune for terminology; retrieve current source texts |
| Regulatory reporting with frequent amendments | RAG | Amended sources can be re-indexed immediately |
Policy and product assistants fail when last month's PDF is still the one the model "remembers". Retrieval, with permissions checked at query time, is the fit. Triage and strict JSON fail in the opposite way. The categories and the schema are stable, and the model keeps drifting. That is a behaviour gap, so examples and a training run earn their cost. Tone sits in between. A tight prompt and a few examples often hold a brand voice. Move to fine-tuning only after the prompt still drifts. Drafting that needs both a house vocabulary and this week's statute or protocol is a hybrid. Terminology in the weights, current texts in the index.
The nature of the gap decides the approach. A knowledge gap points to retrieval. A behaviour gap points to fine-tuning. Both gaps, measured on the same questions, point to a hybrid.
Three edges flip a neat verdict. Stable knowledge plus a strict format pushes toward fine-tuning, because you would pay retrieval tokens for facts that do not move. A corpus that is permission-controlled per user pushes toward RAG, with the access check inside retrieval, not as a filter after the answer. A small team with no one to label failures should buy a managed retrieval service before any tuning work. An unevaluated fine-tune is an expensive guess.
You can also see a wrong choice after the fact. Repeated retraining whose only purpose is to fix facts means the knowledge was put in the weights. A retrieval system that cannot cite the passage it used means the audit trail was never built. Both are reasons to change the architecture, not to add a longer prompt.
Choosing Your Starting Point and Confirming It
Decide in order, and do not skip the cheap step.
- Test whether a base model with careful prompting already meets the requirement. Structured output modes and a handful of examples close many "we need a fine-tune" tickets.
- If the gap is knowledge that changes, start with RAG and build the index from current, permissioned sources.
- If the gap is consistent behaviour, such as format, classification or terminology, consider fine-tuning on a minimised, licensed set.
- If both gaps exist, plan a hybrid and build each layer separately so a regression has an owner.

The order is a cost order as much as a technical one. Prompting costs engineering time. Retrieval adds an index that needs permissions checked at query time and a refresh schedule. Fine-tuning adds a training pipeline, a dataset licence review and a retraining cadence. Each step up commits you to a larger maintenance bill, so the cheapest architecture that clears your evaluation bar is the right starting point. A retrieval index nobody governs can carry more risk than a tightly scoped fine-tune, because its failures arrive through stale or over-permissioned content rather than through a measurable drop in answer quality.
Use the use-case verdicts as the shortlist, then throw them out if your own questions disagree. Before you commit budget, build an evaluation set of 50 to 100 real queries from the people who will use the system. Have experts score groundedness, accuracy and refusal. Rerun that set after every chunking change, embedding swap or training run. A single demo conversation is not evidence.
Confirm before you build
- The update frequency is known, in changes per month, not as "it changes sometimes".
- Every answer that needs a citation has a source passage, or the design does not pretend it can cite.
- The data basis has been reviewed against lawful basis, necessity and less intrusive options.
- A named owner refreshes the index, or a named owner schedules retraining. An ownerless pipeline goes stale.
In the specification step, you explain the idea in writing, and we put together a specification suited to the work. That written scope is what stops a retrieval project turning into an unplanned training project, and it fixes who owns the index before the build starts.
This week, write the three answers from the opening. How often does the knowledge change, does every answer need a source, and must the output follow a fixed pattern? Send them with the evaluation set. The architecture follows those answers. It does not lead them.
Digital Souls Studios LTD designs and builds AI assistants, SaaS products and automation for UK teams that need current answers, traceable sources and behaviour that holds in production. If you are weighing retrieval, a tightly scoped fine-tune, or a hybrid for compliance, support or product knowledge, we can turn that choice into a specification and a working system. Visit Digital Souls Studios LTD to start that conversation.