How Much Math Operations Does It Take to Train an AI Model?

A FLOP is a single floating-point operation. In the words of numerical analyst Nick Higham, it is "one of the elementary arithmetic operations +, -, *, / carried out on floating-point numbers." Training a model means performing an enormous number of them, and the dataset counts the total. One billion petaFLOP is 10⁹ times 10¹⁵, or 10²⁴ operations, so GPT-6 Astra's 940 is 9.4 × 10²⁶ FLOP.

Almost all of that work is matrix multiplication, which is long runs of multiply-then-add steps. Ivanov and colleagues at ETH Zurich measured one encoder layer of BERT-large in 2020 and found that these matrix-type operations made up 99.80% of the floating-point operations, though only 61.0% of the runtime. It is one layer of one older model, not a frontier run, but it shows where the arithmetic piles up.

How operations are counted matters too. Epoch AI treats one fused multiply-add as two FLOP, matching how chip makers quote peak speed, and it applies no adjustment for number format, so an operation at 16-bit precision counts the same as one at 32-bit. Parameters describe how large a model is; compute describes how much arithmetic it took to train, and the two are separate figures.

Our World in Data's 2025 analysis of AI scaling used 50 billion petaFLOP as its illustration of training scale. Six runs in this set now reach it: Gemini 1.0 Ultra at exactly 50 in December 2023, and five strictly above, Llama 4 Behemoth (preview) at 51.84, Grok 3 at 350, GPT-4.5 at 380, Grok 4 at 500 and GPT-6 Astra at 940. All five of the strictly-above runs were published in 2025 or later.

The floor rises year by year. Every 2022 row sits between 2.53 and 2.74, the 2023 rows between 3.76 and 50, the 2024 rows between 18 and 38, and the 2025 rows between 39.11 and 500. Because Epoch's frontier list is the top five by training compute at release, a threshold that rises as larger runs appear, that climbing floor is partly built into how the list is drawn.

Nothing in these counts ranks one model above another as a tool. Epoch's frontier label is a compute threshold, not a verdict on quality, and the figures record how much arithmetic was done to produce each model. Set each number against your own sense of how capable that model is. If you rate a smaller run above a larger one for your own work, the table does not disagree: it measures effort, not outcome.

Numbers this large need a yardstick, so take one card: NVIDIA's GeForce RTX 3090, rated in the company's Ampere GA102 whitepaper at 35.6 trillion operations per second (35.6 TFLOPS) at its FP32 peak, the theoretical top speed of 32-bit arithmetic. Over a year that is about 1.12 million petaFLOP, or 0.0011 billion petaFLOP. A card-year here means one RTX 3090 doing that arithmetic, flat out, for 365.25 days.

On that basis GPT-6 Astra's 940 billion petaFLOP is about 837,000 card-years. Grok 4's 500 is about 445,000, GPT-4's 21 is about 18,700, and PaLM's 2.53 is about 2,250. Our World in Data's rounded 1.1 million petaFLOP per card-year gives 45,455 for its 50 billion illustration, against about 44,500 on NVIDIA's own rate. We use NVIDIA's rate throughout and do not mix the two.

The 837,000 is not a minimum. It is the answer at one stated basis, FP32 at peak, and changing the basis moves it both ways. Using the card's tensor cores at their best dense rate, the same arithmetic would take about 210,000 card-years. At the 30 to 50% utilisation Epoch cites as typical for large distributed runs, FP32 work on a 3090 would take 1.7 to 2.8 million. The statement that survives every basis is that Astra's arithmetic is on the order of a million card-years.

It is not a build, either. NVIDIA's whitepaper supports at most two RTX 3090s linked over NVLink, at 112.5 GB/s in total, while reports for Astra put more than 100,000 GPUs on a single run. Finishing 837,000 card-years in 90 days, the duration Epoch assumes, would take about 3.4 million cards at peak. Card-years is a unit of arithmetic, not a plan for assembling one.

Peak speed is a ceiling no run reaches. In the BERT-large layer above, the matrix work took 61.0% of the time, and the rest went on steps that involve almost no arithmetic. Google reported 46.2% model FLOPs utilisation for PaLM, Meta reported 38 to 43% for Llama 3.1 405B, and Epoch's Astra estimate assumes 25% at FP8, a lower-precision number format.

The 940 billion petaFLOP is Epoch AI's estimate, not an OpenAI disclosure. What OpenAI's vice president of research, Aidan Clark, told Fortune on 3 September 2026 was that Astra is the company's largest training run "by far" and "the first time we've pretrained on more than 100,000 GPUs at our Stargate site in Texas." Epoch built its figure on that reported count.

Epoch's note sets out the method: about 100,000 NVIDIA GB200 chips, running for 90 days, at 25% of their FP8 peak. The chip count is reported and treated as a lower bound, while the 90 days and the 25% are Epoch's assumptions. Change any of the three and the answer moves in proportion.

Epoch labels the figure Likely, its term for a 90% interval of a factor of 10 either way, which spans 94 to 9,400 billion petaFLOP. Its own note also cites a narrower range from a separate estimate, roughly 500 to 2,000 billion petaFLOP. These are two statements about the same estimate, and the 940 should be read as one point inside a range, not a measurement.

Epoch grades each record by how well its compute can be pinned down, stated as a 90% confidence interval. Of the 24 rows, nine are Confident (within a factor of 3 either way), nine are Likely (a factor of 10) and six are Speculative (a factor of 31). The six Speculative rows are GPT-3.5 (davinci-002), SenseChat, Claude 2, Gemini 1.0 Ultra, Claude 3.5 Sonnet and Grok 4.

Applied to the bars, Gemini 1.0 Ultra's 50 could in principle lie anywhere from about 1.6 to 1,550, while PaLM's Confident 2.53 stays between about 0.8 and 7.6. The grade covers the least certain of compute, parameters and dataset size on a row, so the compute figure may be firmer than its label suggests, but the label is the only grade Epoch publishes.

The methods behind the figures differ as well. Epoch records how each was reached: reported by the builders, counted from the architecture, inferred from hardware, or drawn from a benchmark or a cost. The series is consistent in its unit but not in its certainty, so a gap of a few percent between two rows, such as PaLM's 2.53 and Flan-PaLM's 2.54, means very little.

GPT-4 appears twice, in March and June 2023, both at 21, so those two rows are not two separate runs. Four rows belong to the PaLM family: PaLM (540B) at 2.53, U-PaLM at 2.53, Flan-PaLM at 2.54 and Minerva at 2.74. Epoch counts total training compute including any pretrained base model used as a component, so each of those four carries PaLM's own work inside it.

Epoch's frontier label also limits who is listed. It means a model in the top five by training compute at the time of its release, so these are records from a moving list, not every large model and not a census of any company's lineup. A model outside the top five on its release day does not appear, however widely used it is.

Hardware is listed for 18 of the 24 rows: six ran on Google TPU v4, six on NVIDIA H100, five on A100 and one, GPT-6 Astra, on NVIDIA GB200. The other six, among them Grok 4 and GPT-4.5, list none. The filter applied here keeps records published after 1 January 2022 with training compute recorded.