RAG vs fine tuning: how to add an LLM to your product

If you are weighing RAG vs fine tuning to add an LLM to your product, start with retrieval-augmented generation (RAG) in almost every case. RAG lets a model answer from your own documents and data at the moment a user asks, while fine tuning changes how the model behaves by training it further on examples. Most products need the first far more often than the second, and many need neither until the basics work.

What is the difference between RAG and fine tuning?

A large language model knows what it learned during training and nothing about your business. There are two main ways to close that gap. Retrieval-augmented generation keeps the model as it is and hands it the right information at request time: your system searches a store of your documents, tickets, product data or records, picks the most relevant pieces, and places them in the prompt next to the user's question. The model then answers from that context.

Fine tuning takes a different route. You train an existing model further on a set of example inputs and outputs, so the model itself changes. It learns a format, a tone, a labelling scheme or a narrow task well enough that you no longer need to explain it in every prompt. What fine tuning does badly is teach a model facts that change. Once training ends, that knowledge is frozen until you train again.

RAG changes what the model can see. Fine tuning changes how it behaves. Most products have a knowledge problem, not a behaviour problem.

When is RAG the right choice?

RAG fits when the answer lives in your data and that data moves. It is usually the right starting point when:

• Your content changes weekly or daily, such as product catalogues, policies, support articles or case notes.
• Users need answers they can check, with a reference back to the source document.
• Different users are allowed to see different data, so access rules must apply before anything reaches the model.
• You want the freedom to switch model providers later without retraining anything.
• You need a first release in weeks rather than months.

The provider point matters more than it looks. With RAG, the model is a replaceable part. On one of our projects, an AI-assisted project scheduling tool we rescued shortly before a customer demo, the chat feature failed during a model provider's capacity incident, and we switched the integration to another provider through LangChain in time for the demo. Swaps like that stay simple when your product's knowledge sits in your own retrieval layer instead of inside a custom-trained model.

RAG is also where most of the engineering effort goes, and it is not glamorous. Documents have to be split into sensible chunks, turned into embeddings, stored in a vector index such as pgvector, and refreshed when the source changes. Retrieval quality decides answer quality far more than the choice of model does. If the right passage never reaches the prompt, no model will give the right answer.

When does fine tuning actually make sense?

Fine tuning earns its place when the problem is how the model behaves. Good candidates include:

• A fixed output format the model keeps getting wrong, even with clear instructions and examples in the prompt.
• A classification or extraction task with a stable set of labels and plenty of labelled examples.
• A house style or tone that is hard to describe but easy to demonstrate.
• High request volumes, where a smaller tuned model could replace a larger general one and the saving justifies the training and evaluation work.

The cost of fine tuning sits less in the training run and more in what surrounds it. You need clean, representative training data, a held-back set to test against, and a plan for retraining when the base model is retired or your requirements shift. Without that discipline, a fine tuned model becomes a second codebase that nobody can safely change.

Before fine tuning anything, try better prompting. Specific instructions, a few worked examples and a clear output schema often fix "the model does not behave" complaints. On an AI medical scribe we rescued and rebuilt, clinicians needed notes shaped for their specialty, and we met that need with engineered, specialty-specific note-generation prompts on OpenAI.

RAG vs fine tuning: how do you decide for your product?

Ask three questions, in this order:

• Does the model need information it could not have learned, such as your own documents or live records? If yes, you need retrieval.
• With good retrieval and a well-written prompt, is the output still wrong in a consistent way? If yes, consider fine tuning for that specific behaviour.
• Does the feature work but cost too much or run too slowly at your volume? That is an optimisation question. The answer might be a smaller model, caching, fewer calls, or eventually a tuned model.

Many mature products end up using both: retrieval for knowledge, and a tuned model for one narrow step such as routing requests or formatting output. The order still matters. Start with retrieval and prompting, measure where it falls short, then fine tune only the part that still fails. Going the other way round usually means paying to train a model on a problem you have not yet understood.

The third question is often where the real savings hide. In one of our projects, an agency's internal AI platform, optimising a review-mining pipeline cut LLM calls by about 40% and database write load by about 75%. That came from reworking how the pipeline used the model, not from training a new one.

What does it take to add an LLM to your product in production?

The model call is the easy part. The work that decides whether an LLM feature survives real users sits around it:

• Data preparation: deciding which sources the assistant may use, cleaning them, and keeping the index in sync.
• Access control: filtering retrieved content by user or tenant before it goes into the prompt.
• Evaluation: a set of real questions with known good answers, run on every prompt or model change.
• Guardrails: handling empty retrieval, declining out-of-scope requests, and redacting personal data where it matters.
• Cost and speed: tracking tokens per request, caching repeated work, and choosing the smallest model that passes your evaluation.
• Fallbacks: retries, timeouts and a second provider for the day the first one struggles.

On an AI sales-outreach SaaS we built, an LLM agent writes and categorises emails using context retrieved from pgvector embeddings in a multi-tenant PostgreSQL design. Retrieval is what lets each customer's emails reflect their own context, and a design like that has to keep every tenant's data separate all the way into the prompt.

How we approach LLM features at Devflovv

Our AI & LLM Solutions service follows the same four stages as the rest of our work: discover, design, develop and deploy, then support and evolve. For LLM features, discovery means finding the questions users actually ask and the data that answers them. Design means choosing retrieval, prompting or fine tuning based on that evidence, not on what is fashionable this quarter. We build in agile sprints, and we recommend having an evaluation set from the first week, and after launch we stay with you to tune retrieval, prompts and cost as usage grows.

If you already run a product and want an assistant over your own documents, that is almost always a RAG project, and a first release usually does not need a custom model at all.

Key takeaways

• In the RAG vs fine tuning decision, start with RAG: it gives the model your current data without retraining.
• Fine tuning changes behaviour, not knowledge. Save it for stable formats, labels or tone that prompting cannot fix.
• Retrieval quality, access control and evaluation decide whether an LLM feature works, more than the model you pick.
• Keep the model replaceable, so a provider outage or price change is a configuration change rather than a rebuild.
• Measure first, then fine tune only the step that still fails.