What training a model actually looks like

People ask me about this half of my work in a particular tone of voice. Slightly lowered. Roughly the tone you use when asking what it is like inside a building you have only ever seen from the road.

I would like to get rid of that tone, because it is doing damage in both directions. It talks people out of work they would be good at, and it talks other people into work they would hate, on the strength of how the words sound rather than what the days contain.

So this is a description, not a course. Nobody learns to train a model from a blog post and I am not going to pretend otherwise. What I can do is say what the work consists of, which is unusual but not mysterious: the feedback loop is measured in days, most of the effort goes into text rather than mathematics, and the machine you are working on is big enough that parts of it fail while you are using it.

Almost everything I am about to describe is documented publicly, by the way. Labs publish technical reports, and a few of them have published the raw logs. Those documents admit considerably more than the coverage of them ever does.

Most of it is not modelling

The picture people arrive with is somebody at a whiteboard inventing an architecture. That does happen. In practice it is a small share of the calendar, and the organisations doing it at scale are unusually blunt about that.

Meta's report on Llama 3 puts it in a single sentence I would not have dared to write myself. Llama 3 "does not deviate significantly from Llama and Llama 2 in terms of model architecture; our performance gains are primarily driven by improvements in data quality and diversity as well as by increased training scale." The architecture was close to settled, and the results still moved, because the results were coming from somewhere else.

Where they were coming from is the corpus. Corpus work is not glamorous and it is not abstract. It is deciding what text goes in and what does not, at a scale where every decision has to be a rule rather than a judgement, and then finding out whether the rule was any good.

Their filtering pipeline is described in ordinary engineering terms: remove domains heavy with personal data, deduplicate at the level of URLs, again across documents with MinHash, again line by line. Count dirty words to drop adult sites. Measure the token distribution of a document against the corpus and throw away the outliers. Train a small classifier to guess whether Wikipedia would ever cite this page, and use its guess as a filter.

Read enough of these reports and the grain of the work becomes very clear. AI2's OLMo 2 team drop any document containing a run of thirty-two or more repeated n-grams, and any document where the single most frequent word makes up more than thirty per cent of it. That is the unit of progress at this layer: a rule about text, argued over, tested, and usually less obviously correct than it sounds.

What testing a rule costs is the part nobody expects. When Hugging Face built FineWeb, fifteen trillion tokens drawn from ninety-six Common Crawl snapshots, they trained more than seventy models to make the decisions, at roughly eighty thousand H100 GPU hours. Not one of those models was an experiment about architecture. They existed to answer questions about which text to keep.

I find this is the single most useful thing to tell somebody weighing up this work. If you imagine yourself deriving things, you have imagined the wrong job. A great deal of it is data engineering with a research question attached, and the people who are good at it are the ones who can stand to read the documents.

What a training run actually is

Everything in ordinary software gives you an answer quickly. A build fails in seconds and tells you which line. A test suite is red or green before you have finished reading your own diff. We stop noticing that tightness is even there.

A training run is the opposite of that, in every respect at once. It runs for weeks. It costs money continuously while it runs. Almost nothing about whether it is going well is visible from the outside, and the failures do not announce themselves at the moment they begin.

Start with the hardware, because that is the least discussed and the most relentless. Meta published the numbers for a fifty-four day stretch of Llama 3 405B pre-training: 466 job interruptions, of which 47 were planned maintenance and 419 were not. Around 78 per cent of the unexpected ones traced to confirmed or suspected hardware faults, and GPU problems alone accounted for 58.7 per cent. They still kept effective training time above 90 per cent, which is a considerable engineering achievement and also tells you what the baseline looks like. At that scale, something is always broken.

For texture, read the OPT-175B logbook, which Meta AI published alongside the model. This is a working document, written by the people on call, and it opens with the goal stated as "get a 175B dense model up and running by any means necessary." The run identifiers walk from 11.0 up through 12.57. Fifty-odd relaunches of one experiment. There are entries about nodes drained for uncorrectable memory errors, an auto-recovery script that crashed while recovering, and a job that hung on Christmas morning and had to be traced back through a network error message to a single machine. One line reads, in full: "Unable to determine the device handle for GPU 0001:00:00.0: GPU is lost. Reboot the system to recover this GPU."

Then there is the other kind of failure, the one that is genuinely about the model. Google's PaLM report describes roughly twenty loss spikes during the 540B run, at irregular intervals, sometimes very late. They could not afford to find out why. What they did instead was restart from a checkpoint about a hundred steps before the spike and skip the next two to five hundred batches of data, and the spike did not come back.

The reason it did not come back is the interesting part, and it is the sentence I would put in front of anyone who thinks this resembles debugging. They ran the same data through an earlier checkpoint and nothing went wrong. So it was not bad data. It was, in their words, "the combination of specific data batches with a particular model parameter state." The bug existed only in the meeting of the two, and it was not reproducible from either one alone.

That is not a failure mode software engineering has an equivalent for. You cannot bisect it. You cannot write a regression test for it. You back up, go round, and carry on.

The loop is days long

The instruments are thin. There is a loss curve, a gradient norm, throughput, and whatever evaluations you have arranged to run on intermediate checkpoints. That is close to all of it, for a process with hundreds of billions of parameters in flight.

So you learn to read the little you have. OLMo 2's team describe watching the gradient norm creep upward across a run, and note that sharper spikes in it often arrived before spikes in the loss. Most of the skill on a running job is reading a leading indicator and deciding whether to spend a week finding out.

Deciding is the right word, because the experiment that would settle it is often one you cannot afford. Elsewhere in the same report is a line I think is the most honest sentence in the whole genre: "Due to cost concerns we did not explore the full range of learning rates." That is not carelessness. It is a research group saying out loud that the question stayed open because answering it cost too much, which is a normal condition here and almost never a normal condition at Layer 2.

Working inside that rhythm is genuinely strange. The run is going. You are not going to know anything conclusive today. So you spend the day on small models, running the ablation that will inform the next decision, writing up what the last run did, reading documents that a filter is about to throw away. Then you look at the curves again.

The compensation for a slow loop is that you have to think before you act, and that turns out to be a habit rather than a virtue. When the cost of trying something is nothing, you try things. When it is four days of a cluster, you write down what you expect to see, and why, and what would falsify it, before you start. Most people arriving from software find that the hardest adjustment, and the ones who take to this work are usually the ones who were quietly doing it already.

What it costs, and who is in the room

Now the part that decides whether any of this is available to you.

Nobody publishes what their model cost. What exists are estimates, and the most careful public ones come from Epoch AI, who reconstructed amortised hardware and energy costs for the final training runs of forty-five frontier models. Their finding is that this figure has grown about 2.4 times per year since 2016, with a 95 per cent confidence interval of 2.0 to 3.1, and they project the largest runs passing a billion dollars by 2027. In the models where they broke development down further, hardware came to 47 to 67 per cent of the total, research staff 29 to 49 per cent including equity, and energy only 2 to 6 per cent. Treat those as reconstructions rather than invoices, which is what their authors call them.

The one widely quoted figure that came from a lab is DeepSeek-V3, reported as 2.788 million H800 GPU hours across pre-training, context extension and post-training, priced at an assumed two dollars per GPU hour to give 5.576 million dollars. Headlines carried it a long way as the cost of a frontier model. Read the next sentence in the report and it says the costs "include only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data." That excluded remainder is most of what the previous sections were about.

So be clear-eyed about scale. Pre-training a frontier model happens at a small number of organisations with resources almost nobody reading this will have, and no amount of enthusiasm changes that.

Post-training is the accessible end. Taking a released open-weights model and continuing to train it on your own data is a different activity from building one from scratch, and the equipment has come down to something a person can own. QLoRA, out of the University of Washington, got fine-tuning a 65 billion parameter model onto a single 48GB GPU, down from the 780GB the standard approach needed, and trained their largest model in about a day on that one card. That is the door, and it is genuinely open.

Why this is a different profession

I say this to engineers constantly, and it lands badly about half the time, so let me be exact about it.

When I tell somebody they do not need to understand training in order to build with a model, I am not being kind to them. I do both jobs. The maths that matters enormously in here matters hardly at all out there. This is not a harder version of software engineering, it is a different one, and being excellent at either tells you very little about the other.

The differences are all in what the day asks of you. At Layer 2 the loop is tight and the discipline is handling failure well. Here the loop is long, the failures are late and partly unexplainable, and the discipline is deciding what is worth finding out with a budget that will not answer every question. The vocabulary is different too, and unlike the vocabulary at Layer 2, you do genuinely need the mathematics behind it.

Which makes it worth entering deliberately rather than by drifting towards it because it sounded like the serious option. The tell that this is your work is not that you find the models impressive, it is that you find the corpus interesting. If reading a filtering rule and wondering what it quietly deletes is a pleasant thought, you may be in the right building. If the appeal was mostly the idea of standing in the building, the other two layers are better places to spend a career, and there is no shame in the map having three rooms.

See it for yourself

You cannot train a model this afternoon, so do the next best thing and read one being trained.

Open the OPT-175B logbook, which Meta AI published in full. Do not read it for the results. Read four or five dated entries in a row and count how many describe something breaking, and how many describe a decision made without knowing whether it was right. Notice the run identifiers climbing.

Then read one paragraph of a technical report: the training instability section of the PaLM paper. Short enough to read standing up. It says the spikes happened about twenty times, that they could not afford a principled fix, and that restarting a hundred steps back and skipping a few hundred batches made them go away.

Two things usually land at once. The first is that this is much less polished than the finished announcements suggest. The second is why I recommend this to people at every layer: the people who built the thing you use every day did not know what was happening either, and they wrote it down anyway. Reading one of these properly does more for your sense of the field than any amount of reading about it.

Sources

Linked so you can check them, and flagged where the number is somebody's reconstruction rather than a measurement.

• Meta, The Llama 3 Herd of Models, 2024. The architecture sentence in Section 3.2; the data pipeline in Section 3.1; and Section 3.3.4 for the reliability figures: 466 job interruptions in a 54-day pre-training snapshot, 419 of them unexpected, approximately 78% attributed to confirmed or suspected hardware issues, GPU issues 58.7% of unexpected interruptions, effective training time above 90%. Reported by the developer about its own run, so treat it as a self-report, but it is a self-report against the developer's interest.
• Meta AI, OPT-175B Logbook, 2022. The working log, November 2021 to January 2022. Run identifiers from 11.0 to 12.57, the stated goal, the Christmas Day hang, the lost GPU. Primary document, published unedited.
• Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, Google, 2022. Section 5.1: roughly 20 loss spikes in the 540B run, mitigation by restarting about 100 steps earlier and skipping 200 to 500 batches, and the conclusion that spikes came from "the combination of specific data batches with a particular model parameter state".
• OLMo team, OLMo 2 Furious, Allen Institute for AI, 2025. Gradient norm growth and spikes preceding loss spikes, the 32-repeated-n-gram and 30% most-frequent-word filters, and the learning rate sentence quoted above.
• Penedo et al., The FineWeb Datasets, Hugging Face, 2024. 15 trillion tokens from 96 Common Crawl snapshots; more than 70 ablation models trained at an estimated 80,000 H100 GPU hours to make the filtering and deduplication decisions.
• DeepSeek-AI, DeepSeek-V3 Technical Report, 2024. 2.788M H800 GPU hours in total, costed at an assumed $2 per GPU hour to give $5.576M, with the report's own statement that this excludes prior research and ablation experiments. The dollar figure applies a rental rate the authors did not necessarily pay, and it is the weakest number in this post.
• Cottier, Rahman, Fattorini, Maslej and Owen, How Much Does It Cost to Train Frontier AI Models?, Epoch AI, 2024. 45 models; amortised hardware and energy cost of the final training run growing 2.4x per year since 2016 (95% CI 2.0x to 3.1x); hardware 47 to 67% of development cost, R&D staff 29 to 49%, energy 2 to 6%. These are third-party estimates built from disclosures and reporting, not audited figures, and the authors say so themselves.
• Dettmers, Pagnoni, Holtzman and Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, University of Washington, 2023. Finetuning a 65B parameter model on a single 48GB GPU, against more than 780GB for the standard 16-bit approach. Their claim that this matches full 16-bit finetuning performance is measured on their own benchmark selection.