When fine-tuning is actually the answer

The question always arrives the same way. Somebody senior has been reading, or has sat through a vendor meeting, and asks whether we should be training our own model. It is not a stupid question. It is usually the wrong one, and explaining why takes about ten minutes, which is longer than most people get.

I work on the machine learning end as well as the engineering end, so I have done this. I have curated the data, run the jobs, sat through the evaluation, and shipped a tuned model. That is why I am comfortable telling most teams not to. The honest answer is no in the large majority of cases I see, and the few where it is yes are specific enough to name.

Of everything in the model-training layer, this is the one decision a working engineer actually faces. The rest of it is a survey. This part is a choice somebody will ask you to make.

What fine-tuning actually does

Start with the mechanism, because most of the confusion here is people picturing something that does not happen.

A base model has already been trained on an enormous amount of text. That is where its knowledge of the world came from, and it cost somebody millions of pounds and a great deal of hardware. Fine-tuning does not repeat it. It takes the finished model and continues training on a much smaller set of examples you provide, nudging the weights toward the behaviour those examples demonstrate. Hundreds or thousands of them, not billions.

The clearest statement of what that buys you comes from the LIMA paper, published in 2023. They called it the Superficial Alignment Hypothesis: "A model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users." They tested it by fine-tuning a 65B LLaMa on exactly 1,000 curated prompt-response pairs, with no reinforcement learning and no preference modelling, and got responses that human raters found equivalent or better than GPT-4's in 43% of comparisons.

A thousand examples. That is the shape of the thing. Fine-tuning is how you tell a model what kind of output you want, and it is a poor way to tell it anything it does not already know.

Fine-tuning comes in two forms, and the difference matters for what you are signing up to.

Full fine-tuning updates every weight in the model. You end up with a complete second copy of a very large model, which you then have to store and serve. Parameter-efficient methods take a different route. The best known is LoRA, from Hu and colleagues at Microsoft in 2021, which freezes the original weights entirely and injects small trainable rank decomposition matrices into each layer. Their headline comparison against GPT-3 175B fine-tuned with Adam: 10,000 times fewer trainable parameters and a third of the GPU memory, with "no additional inference latency". QLoRA extended it two years later to fine-tuning a 65B model on a single 48GB GPU in 24 hours.

So the training run is genuinely affordable now, and that is what people mean when they say fine-tuning has got cheap. The training run was never the expensive part.

The trade-off inside that cheapness is worth knowing. A 2024 paper published in TMLR under the title "LoRA Learns Less and Forgets Less", compared the two directly on programming and mathematics. In standard low-rank settings LoRA "substantially underperforms full finetuning" on the target domain, while better preserving the base model's ability on everything outside it. Their explanation is that full fine-tuning learns perturbations with a rank 10 to 100 times greater than a typical LoRA configuration. You are buying cheapness with capacity, which is fine when the thing you are teaching is small, and it usually is.

The order to go in

Before any of that, there is an order, and skipping steps in it is where the money goes.

Prompt first. Properly, not one attempt. State the task, supply examples of what good output looks like, constrain the format explicitly. OpenAI's own fine-tuning documentation says it plainly: "The prompt engineering process may be all you need in order to get great results for your use case." That is a vendor telling you not to buy the thing they sell, which is the kind of vendor statement worth listening to.

Then try a different model. Teams skip this step more than any other, because it feels like giving up rather than engineering. A capability gap that looks like it needs training frequently closes when you move up a tier, and it closes today rather than in a fortnight.

Then retrieval, if the problem is that the model does not know something. I have written about that separately and will not repeat it here beyond the summary: retrieval is a search problem, the engineering happens before the model is involved, and it is the correct tool when the gap is knowledge rather than behaviour.

Then evals. OpenAI's guidance is a single sentence and I would print it on a wall: "Only invest in fine-tuning after setting up evals." Without a measurement you trust, you cannot tell whether the tuned model is better, and certainly cannot tell six months later whether it still is. An earlier post in this series covers building evals you actually believe.

Every step before fine-tuning is reversible on a Tuesday afternoon. Fine-tuning is not, because what comes out the other end is an artefact you now own and have to keep. That asymmetry is the whole argument for the ordering.

Knowledge is the wrong thing to teach it

The most common reason teams reach for fine-tuning is the one it handles worst. They want the model to know about their business, so they plan to train it on their documents.

Ovadia and colleagues at Microsoft measured what happens, and published it in December 2023. They built a deliberately hard case: 910 multiple-choice questions about events in the United States between August and November 2023, drawn from Wikipedia and therefore entirely after their base models' training cut-off. Genuinely new facts, in other words. They then compared unsupervised fine-tuning against retrieval on three 7B models.

Take Mistral 7B as the representative line. The base model scored 0.481, roughly chance on a multiple-choice set. Fine-tuning on the source documents took it to 0.504. Fine-tuning with paraphrase augmentation, meaning many rewordings of each fact, got to 0.588. Retrieval over the same documents, with no training at all, scored 0.875. Their own summary is that "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge".

Soudani, Kanoulas and Hasibi ran a wider version across twelve models, focused on low-frequency entities, which is the closest published analogue to your organisation's internal vocabulary. Same conclusion: tuning helps a bit across the board, and "RAG surpasses FT by a large margin particularly for least popular factual knowledge".

Both of these are academic benchmarks on 7B-class models using Wikipedia, and both sets of authors flag hyperparameter sensitivity and dataset dependence, so treat the direction as well established and the exact figures as conditions-of-that-experiment. I have never seen a production result that ran the other way, but I have not seen an independent replication on frontier-scale models either, and I would want you to know that.

The intuition underneath is simple. A fact you fine-tune in has been dissolved into weights. It cannot be updated when the policy changes, cannot be attributed to a source, cannot be deleted on request, and cannot be hidden from a user who is not entitled to see it. Retrieval keeps all four of those properties because the material stays outside the model.

When it genuinely is the answer

Now the useful half. There are cases where tuning is right, and they have a common shape.

In every one of them, you can demonstrate what you want and cannot adequately describe it, and you can measure whether you got it. If either half of that is missing, you are not ready.

Output format that must be identical every time, at volume. Not "usually valid JSON" but the same schema, the same field ordering, the same handling of the awkward edge case, across millions of calls. Prompting gets you most of the way and leaves a tail of failures no extra instruction will close. OpenAI lists "generating content in a specific format" and "correcting instruction-following failures" among the things supervised fine-tuning is for, and that matches what I have seen.

A house style or tone you can show but not specify. Every organisation that has tried to write a tone-of-voice prompt knows the failure: the description is either so vague it does nothing or so long it crowds out the actual task. Two hundred examples of correspondence your team is happy with will do what three paragraphs of adjectives will not.

A narrow classification task. Routing, triage, tagging, sentiment on your particular kind of text. These are the oldest and least glamorous machine learning problems and they remain the ones tuning is best at. A small tuned model on a well-defined label set will often beat a much larger model being asked nicely, and it will cost a fraction as much per call.

Cost and latency, by replacing something large with something small. In practice that means distillation: collect the outputs of an expensive model on your real traffic, use them to tune a cheap one, and serve the cheap one. OpenAI describes exactly this case, "train a smaller, cheaper, faster model to excel at a particular task where a larger model is not cost-effective", and Microsoft's Foundry documentation gives the same recipe. It only works for a narrow task, and narrow is doing real work in that sentence.

A domain whose language the base model handles badly. This one is easy to confuse with the knowledge case, so here is the tell. The problem is not that the model lacks facts. It is that the model misreads the input: unusual notation, a discipline-specific grammar, a language poorly represented in the training data, an internal schema that looks like something else. The model is failing at comprehension rather than recall. That is a form problem, and form is what tuning fixes.

What you have actually signed up for

This is the part that surprises people, and it is the part nobody puts in the pitch.

The data is the work. OpenAI's minimum is ten examples and its recommendation is to "start with 50 well-crafted demonstrations", but a serious task needs hundreds or thousands, and every one of them is a judgement call by somebody who knows what good looks like. That person is expensive and busy. Curation, not compute, is what your fine-tuning project will actually cost.

You need an evaluation before you start and forever afterwards. Not a vibe check. A held-out set, a score, and a decision rule. Otherwise you cannot answer the only question that matters: is this better than what you had before?

You inherit the safety properties. Qi and colleagues at Princeton, Virginia Tech and IBM showed in 2023 that fine-tuning on ten adversarial examples, at a cost of under twenty cents, moved GPT-3.5 Turbo's harmfulness rate from 1.8% to 88.8%. The finding that should worry a normal team is the other one. Fine-tuning on Alpaca, an ordinary benign instruction dataset with no harmful content in it, moved the same model from 5.5% to 31.8%. Alignment is not a permanent property of a model. You can train it away by accident.

And the model is now yours to maintain. Microsoft's own documentation is refreshingly blunt about this: "Fine-tuning may need to be repeated whenever the data is updated, or when an updated base model is released. This involves monitoring and updating regularly." That release cadence is not under your control. The next base model will very likely beat your tuned model at your own task, for free, and you will find out by running the evaluation you built, which is the second reason to have built it.

That risk stopped being hypothetical this year. OpenAI's fine-tuning documentation now carries a banner reading: "OpenAI is winding down the fine-tuning platform. The platform is no longer accessible to new users, but existing users of the fine-tuning platform will be able to create training jobs for the coming months." Their deprecations page lists every ft- model snapshot shutting down on 23 October 2026, announced on 22 April. The stated policy is that "Inference on fine-tuned models will continue to be available until the base models are deprecated", which is a sentence worth reading twice before you build on one.

I would not read that as fine-tuning being over. Open-weight tuning is thriving, other platforms still offer it, and the technique is sound. Read it instead as the most direct evidence available that hosted fine-tuning is a smaller market than the discourse around it suggests, and that a tuned model on somebody else's platform is a dependency with a clock on it.

None of this is an argument against training models. It is an argument for arriving at that decision with the alternatives genuinely exhausted, which is a different thing from having tried a prompt twice and got bored. The teams I have seen do this well all looked the same at the start: they had a measurement, they had run out of road on prompting and retrieval, and they could say in one sentence what behaviour they were trying to install.

See it for yourself

Take the thing your organisation is considering tuning a model for and answer four questions in writing, before anybody prices a training run.

Is the gap knowledge or behaviour? Take one failing example and say whether the model got it wrong because it did not know something, or because it mishandled what it did know. Knowledge sends you to retrieval and the exercise is over.

Can you show it? Write ten examples of correct output by hand. If you cannot manage ten, you do not yet know what you are asking for, and neither will the model. If it takes an afternoon, multiply by the hundreds you will need.

How would you know it worked? Name the score and the held-out set. If the answer is that it will feel better, stop.

And what happens when the base model is replaced in nine months? Name who re-runs the job, who re-checks the evaluation, and whose budget pays.

Most teams stop at the first of those, and the ones who reach the last are usually the ones who should proceed. The exercise is not built to talk you out of it. It makes the commitment visible while declining is still cheap.

Sources

Linked so you can check them, and labelled where the publisher has a commercial interest in the answer.

• Zhou, Liu, Xu et al., LIMA: Less Is More for Alignment, 2023. Author affiliations are not stated on the abstract page I checked, so I have not attributed it to an organisation. The Superficial Alignment Hypothesis quoted above; a 65B model tuned on 1,000 curated examples, rated equivalent or better than GPT-4 in 43% of human comparisons. The authors note LIMA "is not as robust as product-grade models".
• Hu, Shen, Wallis et al., LoRA: Low-Rank Adaptation of Large Language Models, Microsoft, 2021. 10,000x fewer trainable parameters and 3x less GPU memory against GPT-3 175B tuned with Adam, with no additional inference latency. Author affiliation is a vendor, though the method is now universal and independently reimplemented many times over.
• Dettmers, Pagnoni, Holtzman and Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, 2023. 65B parameters finetuned on a single 48GB GPU; their best model trained in 24 hours.
• Biderman, Portes, Gonzalez Ortiz et al., LoRA Learns Less and Forgets Less, TMLR 2024, Featured Certification. LoRA "substantially underperforms full finetuning" on programming and mathematics in standard low-rank settings, while better preserving out-of-domain performance; full finetuning learns perturbations of 10 to 100 times greater rank.
• Ovadia, Brief, Mishaeli and Elisha, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, Microsoft, 2023. The current-events task: 910 questions, Mistral 7B scoring 0.481 base, 0.504 fine-tuned, 0.588 fine-tuned with paraphrase augmentation, 0.875 with retrieval. Their stated limitations include hyperparameter sensitivity, three models only, and Wikipedia as the sole source. This is my strongest single number and it comes from 7B models in 2023, so it is the claim in this post I would most want replicated at frontier scale.
• Soudani, Kanoulas and Hasibi, Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge, 2024. Twelve language models; "RAG surpasses FT by a large margin particularly for least popular factual knowledge".
• Qi, Zeng, Xie et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!, Princeton, Virginia Tech and IBM Research, 2023. GPT-3.5 Turbo harmfulness 1.8% to 88.8% after ten adversarial examples costing under $0.20; 5.5% to 31.8% after tuning on the benign Alpaca dataset.
• OpenAI, Fine-tuning and Model optimization guides, accessed September 2026. Vendor documentation. Source of "The prompt engineering process may be all you need", "Only invest in fine-tuning after setting up evals", the minimum of 10 and recommended 50 examples, the smaller-cheaper-faster model case, and the wind-down banner quoted above.
• OpenAI, Deprecations, accessed September 2026. Vendor documentation. The ft- model snapshots shut down on 23 October 2026, announced 22 April 2026, and the policy that inference on fine-tuned models continues only until their base models are deprecated.
• Microsoft, Fine-tuning considerations, Microsoft Foundry documentation, page dated March 2026. Vendor documentation, and the page carries Microsoft's own ai-assisted authoring label, so I have used it only for the use-case and limitations lists, which match my own experience. Source of the repeat-on-base-model-release warning and the distillation case.
• I could not verify Google's equivalent Vertex AI guidance at primary: the relevant pages returned navigation shells or 404s during writing, so Google is absent from this post rather than paraphrased second-hand.