Fine-tuning, RAG, and other ways to be confidently wrong

The model gives a bad answer. Fine-tune it. Still wrong? Add preference optimisation. Give it a retriever. Quantise the result so it can be wrong on a laptop as well.

There are plenty of ways to change an LLM system. SFT, PEFT, DPO, PPO, and RAG all do useful things. None of them means "make the product good", although that seems to be the unofficial expansion when the demo is on Friday.

The problem is usually more specific. Perhaps the model never saw the relevant document. Perhaps the examples taught it the wrong behaviour. Perhaps the evaluation rewards an impressive answer instead of a correct one. Those failures need different fixes.

I want to separate what these methods actually change from the things we hope they will somehow take care of while the GPUs are busy.

The acronyms do different jobs

Here is what these terms normally mean in LLM development. The links lead to the original papers or the relevant library documentation.

Term	The usual meaning
SFT	Supervised Fine-Tuning
PEFT	Parameter-Efficient Fine-Tuning
DPO	Direct Preference Optimization
PPO	Proximal Policy Optimization
RAG	Retrieval-Augmented Generation

SFT describes a training objective and supervision setup. PEFT describes ways to limit trainable parameters. DPO and PPO offer different approaches to changing behaviour using preferences or rewards. RAG changes what information the model receives when answering.

These are not five levels of model maturity. You can do SFT using a PEFT method, then serve that model inside a RAG application. You can also build a useful application without doing any of those training steps. The acronym count is a poor quality metric, even if it looks excellent in an architecture diagram.

SFT and PEFT: teach the task, then discuss the GPU bill

With supervised fine-tuning, you train on examples of the behaviour you want. In a typical instruction-tuning setup, the model learns to produce target responses from supplied inputs using a token prediction loss. TRL's SFT documentation describes both the objective and supported data formats.

For a hypothetical support-summary assistant, an example might contain a ticket conversation and a reviewed summary. The useful work is deciding what a faithful summary must preserve. Did the customer request a refund, or did support actually issue one? Was the root cause confirmed, or merely suggested? A model that reproduces the template beautifully while changing those distinctions has learned the stationery.

SFT can improve consistency and task behaviour. A lower training loss does not establish that the answers are correct. If the target summaries contain unsupported conclusions, those conclusions become part of what the model is rewarded for reproducing. More epochs give the mistake additional educational opportunities.

PEFT addresses a different concern: how much of the model to train. LoRA keeps the base weights frozen and trains low-rank updates to selected weight matrices. You can use those updates while doing SFT. "SFT or LoRA?" is therefore often the wrong question; one can describe the learning objective and the other how the trainable update is represented.

QLoRA adds a frozen, four-bit quantised base model while training LoRA adapters, reducing training memory requirements. That is useful. It does not automatically make the deployed model faster, smaller in every configuration, or better at the task. Those are separate measurements.

Ordinary prompt engineering changes instructions or examples in the input without updating model weights. Learned soft-prompt methods do optimise trainable prompt representations and belong to the PEFT family. Typing "You are a world-class expert" remains a string edit. The model has received an adjective, not a qualification.

Before training, I would test a clear prompt on a representative evaluation set. If the existing model already performs the task adequately, the absence of a training run is allowed. Nobody has to buy a GPU as a gesture of commitment.

DPO: preferred by whom, and for what reason?

Direct Preference Optimization uses preferred and rejected responses to the same input. In its standard formulation, it learns to favour the preferred answer relative to a reference policy, without separately fitting a reward model and running a PPO rollout loop. That is the simplification described in the DPO paper. The TRL implementation accepts preference datasets with chosen and rejected completions.

The part I would spend time on is the preference label.

Imagine two summaries. One accurately preserves uncertainty and says that a detail is not documented. The other supplies a neat, confident explanation that the source never established. If a reviewer rewards smooth prose and apparent completeness, the second answer could win. DPO will not stop the meeting to ask whether this was a sensible rubric.

I would have reviewers distinguish factual support from writing quality and record important disagreements. For code generation, I would also want execution-based checks where possible: an elegant explanation of a nonexistent API should lose to a working solution. Synthetic preferences or an LLM judge may help with scale, but agreement with that judge is a different measurement from correctness on the task.

DPO can be useful when you can reliably tell a better answer from a worse one. It cannot rescue an unclear definition of "better". If your reviewers consistently reward confidence, you may successfully train a model to sound more certain. That is an improvement only if certainty was the missing feature.

PPO: now the bad objective can have a learning curve

Proximal Policy Optimization is a reinforcement-learning algorithm. It alternates collecting samples with policy updates; its commonly used clipped objective discourages excessively large changes on those samples. The original paper explains the method. Clipping helps control the update; it does not determine whether the reward is worth optimising.

In a conventional PPO-based RLHF setup, generated responses receive reward scores, and the policy is updated to improve that reward, usually with a penalty for drifting from a reference model. This adds moving parts: generating training responses, evaluating rewards, estimating learning signals, and keeping training stable. The DPO paper's background describes this earlier setup.

I would ask why the task needs this machinery. Perhaps there is a useful reward signal that is difficult to capture with a fixed demonstration dataset. Fine. Show it, along with evidence that improving the reward improves the intended task.

Suppose a hypothetical reward favours answers that cover every requested field. A model could improve that score by filling unsupported fields instead of leaving them unknown. The dashboard goes up. Someone reading the record now has more work to undo.

PPO does not invent that mismatch, but optimisation can make it more visible. Choosing a more sophisticated optimiser is a peculiar response to not knowing what the score means.

There is also no requirement to run DPO and then PPO merely because both fit on the slide. Each extra training stage needs to earn its place against a simpler baseline.

RAG: the model now has access to the wrong PDF

Retrieval-Augmented Generation combines retrieved material with generation. The original RAG paper combines a generator with a retriever over an external document index. The practical attraction is clear: an answer can use supplied source material rather than depending entirely on information stored in model weights.

There is an awkward amount of work between "we added a vector database" and "the answer uses the right evidence".

For a product documentation assistant, I would first inspect the corpus. Which product version does each page describe? Has an API been deprecated? Can the person asking the question access the retrieved material? Returning a perfect explanation of version 2 to somebody running version 4 is still a retrieval failure, even if its vector is extremely close.

Then I would inspect what the search actually returns. A lexical baseline is useful, especially for exact terms; dense retrieval and reranking are candidates to test. Chunk boundaries need to preserve qualifications and context. An instruction separated from the paragraph explaining when it applies is an excellent way to manufacture a confident answer with a citation attached.

The generator needs evaluation too. Does the answer follow the supplied evidence? Does each citation support the claim beside it? What happens when documents disagree or the answer is absent? I would score retrieval relevance separately from answer support, rather than hiding both behind a single chatbot rating.

Fine-tuning might improve a retriever, a reranker, or the generator's handling of evidence. It is also possible to build a useful RAG baseline without changing the generator's weights. The failure analysis should tell you which component needs work.

RAG provides a route to relevant information. It does not guarantee the information is current or correctly applied. A citation is useful when it supports the answer. Otherwise it is decoration with a URL.

Clean the data. Try not to delete the meaning

Data cleaning sounds like the part you finish before the interesting work starts. In practice, it can decide whether the interesting work measures anything useful.

For collection, I would start with the task and sample representative inputs, including messy cases and failures. Record the source, check the permitted uses, and decide what information the model actually needs. Grabbing every available text file is a storage strategy, not a sampling strategy.

Then define the labels and review examples against that definition. For a support dataset, "the issue is resolved" and "the customer asked whether the issue is resolved" should not become interchangeable during normalisation. Negation, speaker attribution, and chronology carry meaning. Removing them can make the text shorter and the dataset worse.

I would preserve provenance and transformations so a suspicious example can be traced back to its authorised source. De-identification needs review, including contextual clues. Replacing names with placeholders does not establish that the remaining narrative cannot identify somebody. I discussed that problem in the article on LLM deanonymization.

The evaluation split deserves at least as much attention as the cleaning script. Slightly edited copies of a support conversation should not land on opposite sides of the split. If the claim is that a coding assistant generalises to unseen projects, hold out projects. If it must cope with future tickets, hold out a later period. The split should match the claim, rather than whichever random seed produces the most encouraging chart.

I would inspect errors by language, input length, and task category where relevant. An overall score can hide a subset on which the tool is unreliable. Refinement should follow those errors: correct bad labels, fill important coverage gaps, and keep the final test set separate from that iteration.

This is slower than calling dropna() and describing the result as high-quality data provisioning. It also produces more convincing evidence than a CSV with excellent posture.

Congratulations, the mistake is now in GGUF

Eventually the model needs to run outside the training notebook. This is where it is tempting to ask a file extension to finish the engineering.

Different artefacts solve different deployment problems:

• Safetensors stores tensors without relying on pickle's executable serialisation mechanism. Weights alone do not specify the entire application; the model configuration, tokenizer, and runtime still matter.
• GGUF packages model tensors and metadata for GGML-based inference engines, including llama.cpp. It supports different tensor encodings; converting to GGUF and choosing a quantisation scheme are related but distinct decisions.
• ONNX represents a computation graph and associated data for execution by compatible runtimes. Operator support and the exported model's behaviour still need checking.

There is no benefit in collecting formats like conference stickers. Pick the runtime and hardware you need to support, then produce the appropriate artefact.

I would compare the exported model with the training reference using the same inputs, tokenizer, chat template, and decoding settings. If adapters were trained, confirm that the correct base model and adapter are actually loaded or merged. Then evaluate the complete workflow again. A successful conversion command establishes that a conversion command succeeded.

Quantisation also needs a measured tradeoff. Memory use, latency, and task quality can change. The useful evidence is performance on the target machine and workload, including long inputs and concurrent requests, together with checks for important regressions. A smaller download is not evidence that the same answers remain reliable.

Operational readiness adds access controls, controlled logging, monitoring, and a workable rollback. I would also want to know what happens when retrieval fails, the request is too large, or a model update changes the output format. Somebody needs to own those failures. Renaming a checkpoint production_final_v7 does not assign the on-call rotation.

The missing method is choosing a task

I would start with something narrow enough to evaluate: summarising support conversations, extracting fields from documents, or answering questions about a particular product. Each has different failure modes and acceptable outputs. For some tasks, a smaller classifier or ordinary code may be enough; I covered that comparison in the article on modern classifiers.

Build a baseline. Examine its failures. If it needs better source material, work on retrieval. If it repeatedly mishandles the required output despite adequate instructions, investigate supervised adaptation. If reliable preference labels address a remaining behaviour problem, test preference optimisation. Select the deployment format when you know where the model must run.

The useful comparison is how much each change improves the intended workflow, what it breaks, and what it costs to maintain. I would want evidence of that before adding another training stage.

All of these methods can earn their place. Listing them together does not establish that they have.

Sometimes the best result is removing an unnecessary training stage and fixing the data. This produces fewer exciting boxes in the architecture diagram. You may have to settle for software that works.