Every week a client asks us some version of the same question: "Should we fine-tune a model or just use RAG?" It sounds like a technical question, but it's really a business question — and the wrong answer can cost you months of work and tens of thousands of dollars.

After building AI systems for clients across fintech, healthcare, legal, and e-commerce, we've developed a clear mental model for making this call. Here it is.

What Is RAG, Actually?

Retrieval-Augmented Generation (RAG) keeps the base model exactly as it is and teaches it to look things up before answering. You build a knowledge base — from PDFs, databases, product docs, Notion pages, whatever — index it into a vector database like Pinecone or Weaviate, and at query time the system retrieves the most relevant chunks and feeds them into the model's context window.

The model never changes. The knowledge does. That's the key insight.

What Is Fine-Tuning, Actually?

Fine-tuning modifies the model's weights by training it on a curated dataset of examples. You're literally changing what the model knows at a fundamental level — its tone, its reasoning patterns, its domain vocabulary. Think of it as the difference between giving someone a reference book (RAG) versus sending them back to university for three years (fine-tuning).

The Decision Framework We Use

We ask four questions before recommending either approach:

1. Is the knowledge dynamic or static?

If your data changes frequently — pricing, inventory, recent events, customer records — RAG wins every time. Fine-tuned models have a knowledge cutoff the moment training ends. RAG lets you update your knowledge base in minutes without retraining anything.

2. Do you need the model to behave differently, or just know more?

Fine-tuning is about behaviour. If you need the model to write in a very specific tone, follow a strict output schema, or reason in a domain-specific way (legal contracts, medical diagnoses), fine-tuning is often the right call. If you just need it to answer questions about your company's products accurately, RAG is almost certainly enough.

3. What's your data volume and quality?

Fine-tuning requires high-quality, labelled training examples — typically 500 to 10,000+ samples depending on the task complexity. Poor data produces a confidently wrong model, which is worse than no model at all. RAG only needs the raw source documents.

4. What are the cost and latency constraints?

RAG adds a retrieval step (typically 100–300ms) and increases the context window per request, raising token costs. Fine-tuning has a one-time training cost ($500–$5,000+ depending on model size) but results in faster, cheaper inference at scale. If you're running millions of queries per day, fine-tuning's economics can look very attractive.

The Real-World Answer

For most businesses we work with, RAG is the right starting point. It's faster to build (days, not weeks), easier to debug, and lets you update knowledge without engineering involvement. We've shipped production RAG systems in under two weeks.

Fine-tuning makes sense when: you've already validated RAG and hit its ceiling, you have a very specific output format or style requirement, or you're at a scale where per-query token savings justify the training cost.

And increasingly, the best solutions combine both — a fine-tuned model for reasoning and tone, augmented by a RAG layer for fresh knowledge. That's the architecture we reach for on complex enterprise AI builds.

A Quick Decision Table

Choose RAG if: knowledge changes frequently · you need sources/citations · fast time to market · lower upfront cost · domain knowledge lives in documents.

Choose Fine-Tuning if: style/tone is critical · very specific output schema · static knowledge domain · high query volume · you have quality labelled data.

Choose Both if: enterprise-grade accuracy requirements · knowledge changes but behaviour must also be specialised · you have the budget and timeline.

Our Honest Recommendation

Start with RAG. Ship something. Learn from real users. If you hit a ceiling that RAG can't break through — whether that's accuracy, latency, or behaviour consistency — then invest in fine-tuning. Don't skip to the expensive solution when the cheaper one might be good enough.

We've seen too many teams spend three months fine-tuning a model only to realise that a well-structured RAG pipeline would have solved the problem in two weeks. Build the feedback loop first. Then optimise.