Since 2010, the length of training runs has increased by 1.2x per year among notable models, excluding those that are fine-tuned from base models.
| Model | Domain | Training compute (FLOP) | Training time (days) | Organization | Publication date |
|---|---|---|---|---|---|
| iCCCP | Vision | 1.1e+15 | 1.00 | Massachusetts Institute of Technology (MIT) | 2010-06-13 |
| RNN LM | Language | 5.4e+16 | 21.00 | Johns Hopkins University | 2010-09-26 |
| Deep Autoencoders | Vision | 3.7e+16 | 2.00 | University of Toronto | 2011-04-29 |
| High Performance CNN (NORB) | Vision | 2.6e+16 | 0.17 | IDSIA, SUPSI | 2011-07-16 |
| CNN Committee (MNIST) | Vision | 5.2e+16 | 4.10 | IDSIA | 2011-09-18 |
| CNN Committee (NIST) | Vision | 2.6e+16 | 1.80 | IDSIA | 2011-09-18 |
| NLP from scratch | Language | NA | 3.00 | NEC Laboratories, Princeton University | 2011-11-08 |
| Dropout (CIFAR) | Vision | 4.3e+15 | 0.06 | University of Toronto | 2012-06-03 |
| Dropout (ImageNet) | Vision | 2.7e+17 | 4.00 | University of Toronto | 2012-06-03 |
| Unsupervised High-level Feature Learner | Vision | 6.0e+17 | 3.00 | 2012-07-12 | |
| AlexNet | Vision | 4.7e+17 | 5.50 | University of Toronto | 2012-09-30 |
| DistBelief Speech | Speech | 3.1e+17 | 5.00 | 2012-12-03 | |
| DNN EM segmentation | Vision | 4.8e+17 | 0.71 | IDSIA, SUPSI | 2012-12-03 |
| DistBelief NNLM | Language | 2.6e+18 | 14.00 | 2013-01-16 | |
| ReLU-Speech | Speech | 1.3e+17 | 7.00 | 2013-05-26 | |
| Multilingual DNN | Speech | NA | 28.00 | 2013-05-26 | |
| RCTM | Language | 9.3e+15 | 0.62 | University of Oxford | 2013-10-01 |
| RNTN | Language | 1.4e+16 | 0.21 | Stanford University | 2013-10-01 |
| Word2Vec (large) | Language | 3.9e+16 | 1.00 | 2013-10-16 | |
| RNN for 1B words | Language | NA | 10.00 | 2013-12-11 | |
| SPPNet | Vision | 3.4e+18 | 28.00 | 2014-06-18 | |
| SmooCT | Games | 6.9e+16 | 2.00 | University College London (UCL) | 2014-07-01 |
| VGG16 | Vision | 1.2e+19 | 21.00 | University of Oxford | 2014-09-04 |
| Seq2Seq LSTM | Language | 5.6e+19 | 10.00 | 2014-09-10 | |
| SPN-4+KN5 | Language | 4.4e+16 | 1.70 | 2014-09-14 | |
| TA-CNN | Vision | 1.1e+16 | 0.12 | Chinese University of Hong Kong (CUHK) | 2014-11-29 |
| Fractional Max-Pooling | Vision | 1.0e+17 | 0.75 | University of Warwick | 2014-12-18 |
| MSRA (C, PReLU) | Vision | 2.4e+19 | 24.00 | Microsoft Research | 2015-02-06 |
| TRPO | Games | NA | 1.20 | University of California (UC) Berkeley | 2015-02-19 |
| genCNN + dyn eval | Language | 3.4e+16 | 2.00 | 2015-03-17 | |
| TC-DNN-BLSTM-DNN | Speech | 1.9e+17 | 2.10 | Carnegie Mellon University (CMU) | 2015-04-06 |
| U-Net | Vision | 5.1e+16 | 0.42 | University of Freiburg | 2015-05-18 |
| DCNN | Vision | 4.8e+17 | 9.00 | University of Maryland, Rutgers University | 2015-08-07 |
| AlphaGo Lee | Games | 1.9e+21 | 29.00 | DeepMind | 2016-01-27 |
| Named Entity Recognition model | Language | 9.7e+16 | 0.33 | Carnegie Mellon University (CMU) | 2016-03-04 |
| PixelCNN | Vision | NA | 2.50 | Google DeepMind | 2016-06-16 |
| ResNet-200 | Vision | 3.0e+19 | 21.00 | Microsoft Research Asia | 2016-09-17 |
| Xception | Vision | 4.4e+20 | 30.00 | 2016-10-07 | |
| BIDAF | Language | 3.5e+18 | 2.50 | 2016-11-05 | |
| EnhanceNet | Vision | 1.3e+17 | 1.00 | Max Planck Institute for Intelligent Systems | 2016-12-23 |
| DeepStack | Games | 1.4e+19 | 9.10 | 2017-01-06 | |
| MoE-Multi | Language | 9.4e+19 | 12.00 | Jagiellonian University, Google Brain | 2017-01-23 |
| Transformer | Language | 7.4e+18 | 3.50 | Google Research, Google Brain | 2017-06-12 |
| DeepLoc | Biology | 5.8e+17 | 3.30 | 2017-07-07 | |
| JFT | Vision | 8.4e+20 | 60.00 | 2017-07-10 | |
| RetinaNet-R101 | Vision | 2.1e+18 | 1.50 | Facebook AI Research | 2017-08-07 |
| Rainbow DQN | Games | NA | 10.00 | DeepMind | 2017-10-06 |
| AlphaGo Zero | Games | 6.5e+20 | 20.00 | DeepMind | 2017-10-18 |
| AlphaGo Master | Games | 3.4e+20 | 3.00 | DeepMind | 2017-10-19 |
| AlphaZero | Games | 1.1e+20 | 1.00 | DeepMind | 2017-12-05 |
| QRNN | Language | 6.9e+17 | 0.50 | Salesforce Research | 2018-02-01 |
| IMPALA | Games | 1.7e+20 | 4.20 | DeepMind | 2018-02-05 |
| 4 layer QRNN (h=2500) | Language | 5.9e+17 | 0.50 | Salesforce Research | 2018-03-22 |
| ResNeXt-101 32x48d | Vision | 8.7e+21 | 21.00 | 2018-05-02 | |
| GPT-1 | Language | 1.8e+19 | 30.00 | OpenAI | 2018-06-01 |
| Big Transformer for Back-Translation | Language | 4.8e+20 | 1.20 | Facebook AI Research, Google Brain | 2018-08-28 |
| Transformer (Adaptive Input Embeddings) WT103 | Language | 4.5e+19 | 2.80 | Facebook AI Research | 2018-09-28 |
| BERT-Large | Language | 2.8e+20 | 4.00 | 2018-10-11 | |
| Mesh-TensorFlow Transformer 4.9B (language) | Language | 1.6e+20 | 0.54 | Google Brain | 2018-11-05 |
| Language | 6.8e+19 | 0.92 | Google Brain | 2018-11-05 | |
| StyleGAN | Image generation | 3.9e+16 | 7.00 | NVIDIA | 2018-12-12 |
| KataGo | Games | 2.3e+19 | 19.00 | Jane Street | 2019-02-27 |
| SciBERT | Language | 8.9e+19 | 7.00 | Allen Institute for AI | 2019-03-26 |
| WeNet (Penn Treebank) | Language | 7.3e+17 | 1.00 | Amazon | 2019-04-08 |
| AWD-LSTM-DRILL + dynamic evaluation† (WT2) | Language | 4.1e+17 | 1.20 | IDIAP | 2019-05-14 |
| RoBERTa Large | Language | 8.5e+21 | 5.00 | Facebook, University of Washington | 2019-07-01 |
| trRosetta | Biology | 3.8e+19 | 45.00 | 2019-08-22 | |
| Megatron-BERT | Language | 2.2e+22 | 16.00 | NVIDIA | 2019-09-17 |
| Megatron-LM (8.3B) | Language | 9.1e+21 | 14.00 | NVIDIA | 2019-09-17 |
| T5-11B | Language | 3.3e+22 | 20.00 | 2019-10-23 | |
| AlphaStar | Games | 1.1e+23 | 44.00 | DeepMind | 2019-10-30 |
| CamemBERT | Language | 8.3e+20 | 1.00 | Facebook, INRIA, Sorbonne University | 2019-11-10 |
| Noisy Student (L2) | Vision | 2.6e+22 | 6.00 | Carnegie Mellon University (CMU), Google | 2019-11-11 |
| StyleGAN2 | Image generation | NA | 84.00 | NVIDIA, Aalto University | 2019-12-03 |
| OpenAI Five | Games | 6.7e+22 | 296.00 | OpenAI | 2019-12-13 |
| DD-PPO | Robotics | 7.8e+20 | 2.80 | 2019-12-19 | |
| AlphaFold | Biology | 1.0e+20 | 5.00 | DeepMind | 2020-01-15 |
| ContextNet + Noisy Student | Speech | 8.2e+21 | 60.00 | 2020-01-19 | |
| Meena | Language | 1.1e+23 | 30.00 | Google Brain | 2020-01-28 |
| ALBERT-xxlarge | Language | 2.4e+21 | 1.30 | 2020-02-09 | |
| Feedback Transformer | Language | 7.7e+18 | 3.50 | 2020-02-21 | |
| TransformerXL + spectrum control | Language | 2.6e+19 | 9.10 | 2020-03-11 | |
| GPT-3 175B (davinci) | Language | 3.1e+23 | 15.00 | OpenAI | 2020-05-28 |
| GShard (dense) | Language | 4.8e+22 | 42.00 | 2020-06-30 | |
| DeLighT | Language | 3.8e+18 | 1.20 | 2020-08-03 | |
| ProBERTa | Biology | 9.7e+18 | 0.75 | 2020-09-01 | |
| Conformer + Wav2vec 2.0 + Noisy Student | Speech | 7.6e+21 | 7.00 | Google, Google Research, Google Brain | 2020-10-20 |
| German ELECTRA Large | Language | 1.4e+21 | 7.00 | deepset, Bayerische Staatsbibliothek Muenchen | 2020-10-21 |
| AlphaFold 2 | Biology | 3.0e+21 | 11.00 | DeepMind | 2020-11-30 |
| CPM-Large | Language | 2.6e+20 | 14.00 | 2020-12-01 | |
| DensePhrases | Language | 2.1e+18 | 0.83 | Korea University, Princeton University | 2020-12-23 |
| CLIP (ViT L/14@336px) | Multimodal, Vision, Language, Video | 1.0e+22 | 12.00 | OpenAI | 2021-01-05 |
| Switch | Language | 8.2e+22 | 27.00 | 2021-01-11 | |
| DeiT-B | Vision | 7.9e+19 | 2.20 | Meta AI, Sorbonne University | 2021-01-15 |
| DLWP | Earth science | 5.7e+18 | 7.00 | University of Washington, Microsoft Research | 2021-02-09 |
| Meta Pseudo Labels | Vision | 4.8e+22 | 11.00 | Google Brain, Google AI | 2021-03-01 |
| PLUG | Language | 3.6e+22 | 35.00 | Alibaba | 2021-04-19 |
| MedBERT | Medicine | 9.5e+18 | 7.00 | 2021-05-20 | |
| EMDR | Language | 1.9e+21 | 15.00 | 2021-06-09 | |
| DeBERTa | Language | 2.6e+22 | 30.00 | Microsoft | 2021-06-10 |
| ALIGN | Multimodal, Vision, Language | 2.6e+22 | 14.00 | Google Research | 2021-06-11 |
| StyleGAN3-T | Image generation | 1.7e+21 | 66.00 | NVIDIA, Aalto University | 2021-06-21 |
| StyleGAN3-R | Image generation | 2.4e+21 | 94.00 | NVIDIA, Aalto University | 2021-06-21 |
| EfficientNetV2-XL | Vision | 9.6e+19 | 1.90 | Google, Google Brain | 2021-06-23 |
| Adaptive Input Transformer + RD | Language | 8.6e+19 | 3.30 | Microsoft Research Asia, Soochow University | 2021-06-28 |
| SEER | Vision | 1.8e+22 | 8.10 | Facebook AI Research, INRIA | 2021-07-29 |
| DNABERT | Biology | 1.1e+20 | 25.00 | Northeastern University | 2021-08-15 |
| Turing ULRv5 | Language | 2.9e+22 | 14.00 | Microsoft | 2021-09-28 |
| Megatron-Turing NLG 530B | Language | 8.6e+23 | 32.00 | Microsoft, NVIDIA | 2021-10-11 |
| CodeT5-base | Language | 1.6e+21 | 12.00 | Salesforce, Nanyang Technological University | 2021-11-01 |
| Florence | Vision | 4.8e+22 | 10.00 | Microsoft | 2021-11-22 |
| Gopher (280B) | Language | 6.3e+23 | 38.00 | DeepMind | 2021-12-08 |
| GLaM | Language | 3.6e+23 | 57.00 | 2021-12-13 | |
| XGLM-7.5B | Language | 2.2e+22 | 21.00 | Meta AI, Facebook AI Research | 2021-12-20 |
| Detic | Vision | 2.3e+19 | 1.00 | Meta AI, University of Texas at Austin | 2022-01-07 |
| AlphaCode | Language | 2.4e+23 | 6.10 | DeepMind | 2022-02-02 |
| GPT-NeoX-20B | Language | 9.3e+22 | 90.00 | EleutherAI | 2022-02-09 |
| LaMDA | Language | 3.6e+23 | 58.00 | 2022-02-10 | |
| ProteinBERT | Biology | 6.5e+19 | 28.00 | 2022-02-10 | |
| FourCastNet | Earth science | 3.5e+20 | 0.67 | 2022-02-22 | |
| PolyCoder | Language | 1.1e+21 | 42.00 | Carnegie Mellon University (CMU) | 2022-02-26 |
| PaLM (540B) | Language | 2.5e+24 | 64.00 | Google Research | 2022-04-04 |
| Sparse all-MLP | Language | 5.3e+20 | 4.70 | Meta AI | 2022-04-14 |
| OPT-175B | Language | 4.3e+23 | 33.00 | Meta AI | 2022-05-02 |
| UL2 | Language | 1.2e+23 | 31.00 | Google Research, Google Brain | 2022-05-10 |
| Gato | Multimodal, Robotics, Games, Language | 4.0e+21 | 4.00 | DeepMind | 2022-05-12 |
| Imagen | Image generation | 1.5e+22 | 4.00 | Google Brain | 2022-05-23 |
| GPT-2 Medium (FlashAttention) | Language | 8.9e+20 | 6.90 | Stanford University, University at Buffalo | 2022-05-27 |
| Tranception | Biology | 7.2e+21 | 14.00 | 2022-05-27 | |
| CoCa | Vision | 7.3e+22 | 5.00 | Google Research | 2022-06-14 |
| CodeT5-large | Language | 2.7e+21 | 21.00 | Salesforce | 2022-07-05 |
| BLOOM-176B | Language | 3.7e+23 | 117.00 | Hugging Face, BigScience | 2022-07-11 |
| ESM2-15B | Biology | 7.4e+22 | 60.00 | 2022-07-21 | |
| AlexaTM 20B | Language | 2.0e+23 | 120.00 | Amazon | 2022-08-02 |
| GLM-130B | Language | 3.5e+23 | 60.00 | Tsinghua University | 2022-08-04 |
| SauTech | Speech, Audio | NA | 30.00 | 2022-09-14 | |
| PaLI | Language, Vision, Multimodal | 1.7e+23 | 10.00 | 2022-09-14 | |
| DiffDock | Biology | 7.2e+19 | 18.00 | Massachusetts Institute of Technology (MIT) | 2022-10-04 |
| AlphaTensor | Other, Games, Mathematics | 7.1e+20 | 7.00 | DeepMind | 2022-10-05 |
| EVA-01 | Vision | 1.5e+22 | 14.00 | 2022-11-14 | |
| Fusion in Encoder | Language | 1.3e+20 | 2.00 | Samsung | 2022-11-18 |
| Discriminator Guidance | Image generation | 2.2e+20 | 20.00 | 2022-11-28 | |
| Vega v2 | Language | 7.8e+22 | 30.00 | 2022-12-04 | |
| CaLM | Biology | 2.9e+19 | 40.00 | University of Oxford | 2022-12-19 |
| DreamerV3 | Multimodal, Games | 2.2e+20 | 272.00 | DeepMind, University of Toronto | 2023-01-10 |
| Nucleotide Transformer | Biology | 8.1e+21 | 28.00 | 2023-01-15 | |
| DDPM-IP (CelebA) | Image generation | 3.5e+20 | 5.00 | Utrecht University | 2023-01-27 |
| BLIP-2 (Q-Former) | Vision, Language | 1.2e+21 | 8.30 | Salesforce Research | 2023-01-30 |
| ViT-22B | Vision | 1.9e+23 | 14.00 | 2023-02-10 | |
| LLaMA-65B | Language | 5.5e+23 | 21.00 | Meta AI | 2023-02-24 |
| AudioGen | Audio | 9.5e+21 | 7.00 | Meta AI, Hebrew University of Jerusalem | 2023-03-05 |
| Falcon-40B | Language | 2.4e+23 | 60.00 | Technology Innovation Institute | 2023-03-15 |
| GPT-4 | Multimodal, Language, Vision | 2.1e+25 | 95.00 | OpenAI | 2023-03-15 |
| PanGu-Σ | Language | 4.7e+23 | 100.00 | Huawei Noah's Ark Lab | 2023-03-20 |
| SigLIP 400M | Vision | 4.9e+21 | 5.00 | Google DeepMind | 2023-03-27 |
| BloombergGPT | Language | 2.4e+23 | 53.00 | Bloomberg, Johns Hopkins University | 2023-03-30 |
| Incoder-6.7B | Language | 3.0e+21 | 24.00 | 2023-04-09 | |
| Agile Soccer Robot | Robotics | NA | 10.00 | Google DeepMind | 2023-04-26 |
| StarCoder | Language | 8.5e+22 | 26.00 | 2023-05-09 | |
| HyenaDNA | Biology | 1.8e+21 | 28.00 | 2023-06-27 | |
| Pangu-Weather | Earth science | 4.0e+22 | 64.00 | Huawei | 2023-07-05 |
| xTrimoPGLM -100B | Biology | 6.2e+23 | 163.00 | Tsinghua University, BioMap Research | 2023-07-06 |
| Llama 2-70B | Language | 8.1e+23 | 72.00 | Meta AI | 2023-07-18 |
| Jais | Language | 4.9e+22 | 25.00 | 2023-08-29 | |
| Swift | Robotics | 5.3e+16 | 0.03 | Intel Labs | 2023-08-30 |
| Falcon-180B | Language | 3.8e+24 | 180.00 | Technology Innovation Institute | 2023-09-06 |
| Amazon Titan | Language, Image generation | 4.8e+24 | 48.00 | Amazon | 2023-09-28 |
| CODEFUSION (Python) | Language | 7.9e+18 | 0.46 | Microsoft, Microsoft Research | 2023-10-26 |
| Skywork-13B | Language | 2.5e+23 | 39.00 | Kunlun Inc. | 2023-10-30 |
| MultiBand Diffusion | Audio, Speech | 2.6e+19 | 2.00 | 2023-11-08 | |
| Nemotron-3-8B | Language | 1.8e+23 | 19.00 | NVIDIA | 2023-11-15 |
| Gemini 1.0 Ultra | Multimodal, Language, Vision | 5.0e+25 | 100.00 | Google DeepMind | 2023-12-06 |
| MegaScale (Production) | Language | 3.9e+24 | 21.00 | ByteDance, Peking University | 2024-02-23 |
| Mistral Large | Language | 1.1e+25 | 104.00 | Mistral AI | 2024-02-26 |
| AlphaFold 3 | Biology | 4.1e+22 | 20.00 | Google DeepMind, Isomorphic Labs | 2024-05-08 |
| Octo-Base | Robotics | 5.8e+20 | 0.58 | 2024-05-20 | |
| Nemotron-4 340B | Language | 1.8e+25 | 92.00 | NVIDIA | 2024-06-14 |
| Llama 3.1-405B | Language | 3.8e+25 | 89.00 | Meta AI | 2024-07-23 |
| Movie Gen Video | Video, Vision | 1.6e+24 | 14.00 | Meta AI | 2024-10-04 |
| RDT-1B | Robotics | 4.1e+22 | 30.00 | Tsinghua University | 2024-10-10 |
| CHAI-1 | Biology | 7.8e+21 | 30.00 | Chai discovery | 2024-10-15 |
| Yi-Lightning | Language | 1.5e+24 | 30.00 | 01.AI | 2024-10-18 |
| Grok 3 | Language, Vision, Multimodal | 3.5e+26 | 90.00 | xAI | 2025-02-17 |
| FGN | Earth science | 9.6e+21 | 3.00 | Google DeepMind | 2025-06-12 |
| EXAONE 4.0 (32B) | Language | 2.7e+24 | 133.00 | LG AI Research | 2025-07-15 |
A continuation of this trend would ease hardware constraints, by increasing training compute without requiring more chips or power.
However, longer training times face a tradeoff. For very long runs, waiting for future improvements to algorithms and hardware might outweigh the benefits of extended training.
Epoch's work is free to use, distribute, and reproduce provided the source and authors are credited under the Creative Commons BY license.
Download this data
CSV, Updated Nov. 24, 2025
Explore this data
Our comprehensive database of over 3500 models tracks key factors driving machine learning progress.