RAG vs fine-tuning: which one for policy documents?
Retrieval-augmented generation and fine-tuning solve different problems, and policy documents expose the difference quickly. Retrieval changes what the model can see when it answers. Fine-tuning changes how the model behaves. Most teams asking for one actually need the other.
What each one does
A RAG system searches a document collection for the passages relevant to a question and gives them to the model alongside the question. The model answers from that text and can cite it. Nothing about the model changes; update a document and the next answer reflects it.
Fine-tuning continues training the model on your examples. It is good at teaching format, tone, terminology and task structure — how to write a briefing note in your department’s house style. It is poor at teaching facts reliably, and it cannot tell you where an answer came from.
Why policy work favours retrieval
- Policy changes. A fine-tuned model keeps answering from the version it was trained on until it is retrained.
- Answers need a source. An officer has to be able to check the clause, and an auditor has to be able to see which text an output relied on.
- Access differs by user. Retrieval can filter by the reader’s clearance before the model sees anything; weights cannot be filtered after training.
- Cost. Indexing a new document is minutes of compute; retraining is a project.
When fine-tuning earns its place
Fine-tune when the problem is behaviour rather than knowledge: a model that must produce a fixed document structure, classify correspondence into your categories, or handle language so specialised that a general model misreads it. In practice the strongest systems combine the two — a lightly fine-tuned model that knows the format, answering from retrieved text it can cite.
What to measure before deciding
Build a set of real questions with known correct answers and the passages that support them. Measure retrieval first: if the right passage is not in the top results, no model will answer correctly. Only once retrieval is reliable is it worth asking whether the model needs further training.