Across all domains of ML, models are using more and more training data. In language modeling, datasets are growing at a rate of 3.7x per year. The largest models currently use datasets with tens of trillions of words. The largest public datasets are about ten times larger than this, for example Common Crawl contains hundreds of trillions of words before filtering.
| Model | Domain | Training compute (FLOP) | Training dataset size (datapoints) | Organization | Publication date |
|---|---|---|---|---|---|
| RNN LM | Language | 5.4e+16 | 6.4e+6 | Johns Hopkins University | 2010-09-26 |
| Vector Space Model | Language | NA | 7.5e+4 | Stanford University | 2011-06-19 |
| Recursive Neural Network | Vision, Language | NA | 8.3e+5 | Stanford University | 2011-06-28 |
| NLP from scratch | Language | NA | 8.5e+8 | NEC Laboratories, Princeton University | 2011-11-08 |
| LSTM LM | Language | 1.7e+16 | 2.7e+7 | RWTH Aachen University | 2012-09-09 |
| DistBelief NNLM | Language | 2.6e+18 | 6.0e+9 | 2013-01-16 | |
| RCTM | Language | 9.3e+15 | 4.1e+6 | University of Oxford | 2013-10-01 |
| RNTN | Language | 1.4e+16 | 1.6e+5 | Stanford University | 2013-10-01 |
| Word2Vec (large) | Language | 3.9e+16 | 3.3e+10 | 2013-10-16 | |
| Word2Vec (small) | Language | NA | 6.9e+5 | 2013-10-16 | |
| TransE | Language | 1.3e+18 | 1.7e+7 | 2013-12-05 | |
| RNN for 1B words | Language | NA | 1.0e+9 | 2013-12-11 | |
| GloVe (6B) | Language | NA | 6.0e+9 | Stanford University | 2014-01-01 |
| GloVe (32B) | Language | NA | 4.2e+10 | Stanford University | 2014-01-01 |
| Paragraph Vector | Language | NA | 7.5e+4 | 2014-05-14 | |
| AdaRNN | Language | NA | 6.2e+3 | Beihang University | 2014-06-01 |
| RNNsearch-50* | Language | 1.6e+18 | 3.5e+8 | 2014-09-01 | |
| Seq2Seq LSTM | Language | 5.6e+19 | 6.5e+8 | 2014-09-10 | |
| SC-NLM | Multimodal, Vision, Language | NA | 6.0e+5 | University of Toronto | 2014-11-10 |
| SNM-skip | Language | 3.0e+20 | 1.0e+9 | 2014-12-03 | |
| genCNN + dyn eval | Language | 3.4e+16 | 9.3e+5 | 2015-03-17 | |
| BPE | Language | NA | 3.8e+7 | University of Edinburgh | 2015-08-31 |
| Variational (untied weights, MC) LSTM (Large) | Language | 5.9e+15 | 9.3e+5 | University of Cambridge | 2015-12-16 |
| Named Entity Recognition model | Language | 9.7e+16 | 2.0e+5 | Carnegie Mellon University (CMU) | 2016-03-04 |
| GNMT | Language | 6.6e+21 | 3.9e+11 | 2016-09-26 | |
| Pointer Sentinel-LSTM (medium) | Language | 7.5e+15 | 9.3e+5 | MetaMind Inc, Salesforce | 2016-09-26 |
| VD-LSTM+REAL Large | Language | 2.1e+16 | 9.3e+5 | Salesforce Research, Stanford University | 2016-11-04 |
| NAS with base 8 and shared embeddings | Language | 1.0e+16 | 9.3e+5 | Google Brain | 2016-11-05 |
| BIDAF | Language | 3.5e+18 | 4.7e+7 | 2016-11-05 | |
| MoE-Multi | Language | 9.4e+19 | 1.3e+11 | Jagiellonian University, Google Brain | 2017-01-23 |
| Mnemonic Reader | Language | NA | 1.1e+5 | Fudan University, Microsoft Research | 2017-05-08 |
| Transformer | Language | 7.4e+18 | 1.9e+9 | Google Research, Google Brain | 2017-06-12 |
| ConvS2S (ensemble of 8 models) | Language | 5.6e+19 | 4.7e+7 | Meta AI | 2017-07-25 |
| GSM | Language | NA | 4.0e+6 | Peking University, Microsoft Research | 2017-07-30 |
| Language | 3.0e+17 | 2.0e+6 | Salesforce Research | 2017-08-07 | |
| EI-REHN-1000D | Language | 1.1e+16 | 9.3e+5 | 2017-08-14 | |
| Language | 4.6e+17 | 2.0e+6 | Ben-Gurion University of the Negev | 2017-08-29 | |
| ISS | Language | 3.4e+15 | 9.3e+5 | Duke University, Microsoft | 2017-09-15 |
| AWD-LSTM+WT+Cache+IOG (WT2) | Language | 3.2e+15 | 2.0e+6 | NTT Communication Science Laboratories | 2017-09-26 |
| PhraseCond | Language | NA | 9.0e+4 | 2017-10-28 | |
| S-Norm | Language | NA | 2.0e+9 | 2017-10-29 | |
| DCN+ | Language | NA | 1.1e+5 | Salesforce Research | 2017-10-31 |
| Fraternal dropout + AWD-LSTM 3-layer (WT2) | Language | 3.1e+17 | 2.0e+6 | 2017-10-31 | |
| AWD-LSTM-MoS + dynamic evaluation (WT2, 2017) | Language | 3.4e+18 | 2.0e+6 | Carnegie Mellon University (CMU) | 2017-11-10 |
| DL scaling LM | Language | NA | 4.0e+8 | Baidu | 2017-12-01 |
| QRNN | Language | 6.9e+17 | 1.0e+8 | Salesforce Research | 2018-02-01 |
| 4 layer QRNN (h=2500) | Language | 5.9e+17 | 1.0e+8 | Salesforce Research | 2018-03-22 |
| Dropout-LSTM+Noise(Bernoulli) (WT2) | Language | 1.3e+17 | 2.0e+6 | 2018-05-03 | |
| aLSTM(depth-2)+RecurrentPolicy (WT2) | Language | 7.3e+16 | 2.0e+6 | 2018-05-22 | |
| GPT-1 | Language | 1.8e+19 | 1.0e+9 | OpenAI | 2018-06-01 |
| Big Transformer for Back-Translation | Language | 4.8e+20 | 3.4e+9 | Facebook AI Research, Google Brain | 2018-08-28 |
| (ensemble): AWD-LSTM-DOC (fin) × 5 (WT2) | Language | 6.7e+17 | 2.0e+6 | 2018-08-30 | |
| LSTM+NeuralCache | Language | 9.8e+14 | 2.0e+6 | KU Leuven, ESAT - PSI, Apple | 2018-09-24 |
| Transformer (Adaptive Input Embeddings) WT103 | Language | 4.5e+19 | 1.0e+8 | Facebook AI Research | 2018-09-28 |
| BERT-Large | Language | 2.8e+20 | 3.3e+9 | 2018-10-11 | |
| TrellisNet | Language | 2.8e+18 | 1.0e+8 | 2018-10-15 | |
| Language | 6.8e+19 | 1.8e+9 | Google Brain | 2018-11-05 | |
| Mesh-TensorFlow Transformer 4.9B (language) | Language | 1.6e+20 | 6.3e+9 | Google Brain | 2018-11-05 |
| Multi-cell LSTM | Language | 2.0e+15 | 9.3e+5 | University of Hyderabad | 2018-11-15 |
| GPipe (Transformer) | Language | NA | 2.0e+10 | 2018-11-16 | |
| Transformer-XL (257M) | Language | 3.8e+20 | 1.0e+8 | 2019-01-09 | |
| GPT-2 (1.5B) | Language | 1.9e+21 | 1.1e+10 | OpenAI | 2019-02-14 |
| SciBERT | Language | 8.9e+19 | 3.3e+9 | Allen Institute for AI | 2019-03-26 |
| BERT-Large-CAS (PTB+WT2+WT103) | Language | 1.5e+20 | 1.3e+9 | Amazon | 2019-04-20 |
| AWD-LSTM + MoS + Partial Shuffled | Language | 3.2e+17 | 2.0e+6 | University of Texas at Austin | 2019-06-10 |
| RoBERTa Large | Language | 8.5e+21 | 3.2e+10 | Facebook, University of Washington | 2019-07-01 |
| Megatron-BERT | Language | 2.2e+22 | 4.6e+10 | NVIDIA | 2019-09-17 |
| Megatron-LM (1.2B) | Language | 1.1e+22 | 1.6e+11 | NVIDIA | 2019-09-17 |
| Megatron-LM (8.3B) | Language | 9.1e+21 | 3.5e+10 | NVIDIA | 2019-09-17 |
| ALBERT | Language | NA | 3.3e+9 | 2019-09-26 | |
| T5-3B | Language | 9.0e+21 | 2.6e+10 | 2019-10-23 | |
| T5-11B | Language | 3.3e+22 | 2.0e+11 | 2019-10-23 | |
| Base LM + kNN LM + Continuous Cache | Language | 3.1e+19 | 1.0e+8 | Stanford University, Facebook AI Research | 2019-11-01 |
| XLM-RoBERTa | Language | 2.1e+22 | 1.3e+11 | Facebook AI | 2019-11-05 |
| Sandwich Transformer | Language | 2.4e+19 | 7.0e+8 | Allen Institute for AI, Facebook AI Research | 2019-11-10 |
| CamemBERT | Language | 8.3e+20 | 3.2e+10 | Facebook, INRIA, Sorbonne University | 2019-11-10 |
| Transformer-XL DeFINE (141M) | Language | 1.7e+18 | 1.0e+8 | 2019-11-27 | |
| MMLSTM (WT-2) | Language | 1.9e+17 | 2.0e+6 | 2019-12-05 | |
| MMLSTM (PTB) | Language | 5.8e+16 | 9.1e+5 | 2019-12-05 | |
| Meena | Language | 1.1e+23 | 4.0e+10 | Google Brain | 2020-01-28 |
| TaLK Convolution | Language | 2.7e+19 | 1.0e+8 | Carleton University | 2020-02-08 |
| ALBERT-xxlarge | Language | 2.4e+21 | 3.3e+9 | 2020-02-09 | |
| Turing-NLG | Language | 1.6e+22 | 4.6e+10 | Microsoft | 2020-02-13 |
| Feedback Transformer | Language | 7.7e+18 | 1.0e+8 | 2020-02-21 | |
| TransformerXL + spectrum control | Language | 2.6e+19 | 1.0e+8 | 2020-03-11 | |
| Tensor-Transformer(1core)+PN (WT103) | Language | 1.6e+18 | 1.0e+8 | University of California (UC) Berkeley | 2020-03-17 |
| ELECTRA | Language | 3.1e+21 | 2.5e+10 | Stanford University, Google, Google Brain | 2020-03-23 |
| NAS+ESS (23M) | Language | NA | 9.3e+5 | 2020-05-06 | |
| GPT-3 175B (davinci) | Language | 3.1e+23 | 3.7e+11 | OpenAI | 2020-05-28 |
| GShard (dense) | Language | 4.8e+22 | 3.5e+11 | 2020-06-30 | |
| DeLighT | Language | 3.8e+18 | 1.0e+8 | 2020-08-03 | |
| ERNIE-GEN (large) | Language | 2.0e+20 | 8.6e+10 | Baidu | 2020-08-06 |
| mT5-XXL | Language | 8.2e+22 | 1.0e+12 | Google, Google Research | 2020-10-20 |
| German ELECTRA Large | Language | 1.4e+21 | 3.6e+10 | deepset, Bayerische Staatsbibliothek Muenchen | 2020-10-21 |
| CPM-Large | Language | 2.6e+20 | 1.7e+10 | 2020-12-01 | |
| DensePhrases | Language | 2.1e+18 | 5.8e+7 | Korea University, Princeton University | 2020-12-23 |
| CT-MoS (WT2) | Language | 5.4e+17 | 2.0e+6 | Google, National Tsing Hua University | 2020-12-25 |
| CLIP (ResNet-50) | Multimodal, Vision, Language, Video | NA | 4.0e+8 | OpenAI | 2021-01-05 |
| CLIP (ViT L/14@336px) | Multimodal, Vision, Language, Video | 1.0e+22 | 4.0e+8 | OpenAI | 2021-01-05 |
| Switch | Language | 8.2e+22 | 5.8e+11 | 2021-01-11 | |
| SRU++ Large | Language | 2.1e+19 | 1.0e+8 | ASAPP | 2021-02-24 |
| Generative BST | Language | 1.4e+22 | 5.7e+10 | Facebook AI Research | 2021-03-05 |
| M6-T | Multimodal, Language, Vision | 5.5e+21 | 1.9e+12 | Alibaba | 2021-03-05 |
| EMDR | Language | 1.9e+21 | 1.7e+11 | 2021-06-09 | |
| DeBERTa | Language | 2.6e+22 | 1.6e+10 | Microsoft | 2021-06-10 |
| ALIGN | Multimodal, Vision, Language | 2.6e+22 | 1.6e+9 | Google Research | 2021-06-11 |
| ERNIE 3.0 | Language | 2.2e+22 | 6.7e+11 | Baidu | 2021-07-05 |
| Jurassic-1-Jumbo | Language | 3.7e+23 | 2.2e+11 | AI21 Labs | 2021-08-11 |
| Zidong Taichu | Multimodal, Speech, Vision, Language | 8.0e+20 | 4.2e+10 | 2021-08-11 | |
| XLMR-XXL | Language | 3.4e+22 | 1.7e+11 | Facebook AI Research | 2021-08-17 |
| PermuteFormer | Language | 2.8e+18 | 1.0e+8 | Peking University | 2021-09-06 |
| HyperCLOVA 204B | Language | 2.0e+23 | 5.6e+11 | NAVER | 2021-09-10 |
| PLATO-XL | Language | 9.9e+21 | 1.5e+11 | Baidu | 2021-09-20 |
| Megatron-Turing NLG 530B | Language | 8.6e+23 | 2.7e+11 | Microsoft, NVIDIA | 2021-10-11 |
| Yuan 1.0 | Language | 3.5e+23 | 1.0e+12 | Inspur | 2021-10-12 |
| S4 | Language | 7.8e+19 | 1.0e+8 | Stanford University | 2021-10-31 |
| Gopher (280B) | Language | 6.3e+23 | 3.0e+11 | DeepMind | 2021-12-08 |
| GLaM | Language | 3.6e+23 | 6.0e+11 | 2021-12-13 | |
| LongT5 | Language | NA | 2.0e+11 | Google Research | 2021-12-15 |
| XGLM-7.5B | Language | 2.2e+22 | 1.7e+9 | Meta AI, Facebook AI Research | 2021-12-20 |
| ERNIE 3.0 Titan | Language | 1.0e+24 | 6.7e+11 | Baidu, Peng Cheng Laboratory | 2021-12-23 |
| data2vec (language) | Language | NA | 3.3e+9 | Meta AI | 2022-01-20 |
| RETRO-7B | Language | 1.7e+22 | 4.2e+11 | DeepMind | 2022-02-07 |
| GPT-NeoX-20B | Language | 9.3e+22 | 3.4e+11 | EleutherAI | 2022-02-09 |
| LaMDA | Language | 3.6e+23 | 1.6e+12 | 2022-02-10 | |
| ST-MoE | Language | 2.9e+23 | 1.5e+12 | Google, Google Brain, Google Research | 2022-02-17 |
| DeepNet | Language | NA | 1.2e+10 | Microsoft Research | 2022-03-01 |
| Statement Curriculum Learning | Language, Mathematics | NA | 2.8e+11 | OpenAI | 2022-03-02 |
| Segatron-XL large, M=384 + HCP | Language | 2.6e+19 | 1.0e+8 | Microsoft Research, University of Waterloo | 2022-03-21 |
| Chinchilla | Language | 5.8e+23 | 1.4e+12 | DeepMind | 2022-03-29 |
| PaLM (540B) | Language | 2.5e+24 | 5.8e+11 | Google Research | 2022-04-04 |
| Sparse all-MLP | Language | 5.3e+20 | 1.0e+11 | Meta AI | 2022-04-14 |
| OPT-175B | Language | 4.3e+23 | 1.8e+11 | Meta AI | 2022-05-02 |
| UL2 | Language | 1.2e+23 | 1.0e+12 | Google Research, Google Brain | 2022-05-10 |
| Gato | Multimodal, Robotics, Games, Language | 4.0e+21 | 5.2e+11 | DeepMind | 2022-05-12 |
| GPT-2 Medium (FlashAttention) | Language | 8.9e+20 | 1.0e+10 | Stanford University, University at Buffalo | 2022-05-27 |
| DITTO | Language | 3.3e+18 | 1.0e+8 | 2022-06-06 | |
| CodeT5-large | Language | 2.7e+21 | 1.0e+10 | Salesforce | 2022-07-05 |
| NLLB | Language | 1.8e+22 | 3.6e+11 | Meta AI | 2022-07-06 |
| BLOOM-176B | Language | 3.7e+23 | 3.8e+11 | Hugging Face, BigScience | 2022-07-11 |
| AlexaTM 20B | Language | 2.0e+23 | 1.3e+12 | Amazon | 2022-08-02 |
| GLM-130B | Language | 3.5e+23 | 4.0e+11 | Tsinghua University | 2022-08-04 |
| PaLI | Language, Vision, Multimodal | 1.7e+23 | 1.6e+9 | 2022-09-14 | |
| Galactica | Language, Biology | 3.2e+23 | 1.1e+11 | Meta AI | 2022-11-16 |
| Hybrid H3-2.7B | Language | 6.5e+21 | 4.0e+11 | Stanford University, University at Buffalo | 2022-12-28 |
| LLaMA-65B | Language | 5.5e+23 | 1.3e+12 | Meta AI | 2023-02-24 |
| Falcon-40B | Language | 2.4e+23 | 1.0e+12 | Technology Innovation Institute | 2023-03-15 |
| GPT-4 | Multimodal, Language, Vision | 2.1e+25 | 4.9e+12 | OpenAI | 2023-03-15 |
| PanGu-Σ | Language | 4.7e+23 | 2.5e+11 | Huawei Noah's Ark Lab | 2023-03-20 |
| BloombergGPT | Language | 2.4e+23 | 5.3e+11 | Bloomberg, Johns Hopkins University | 2023-03-30 |
| StarCoder | Language | 8.5e+22 | 1.0e+12 | 2023-05-09 | |
| PaLM 2 | Language | 7.3e+24 | 2.7e+12 | 2023-05-10 | |
| ONE-PEACE | Multimodal, Vision, Speech, Language | 1.8e+20 | 1.6e+9 | 2023-05-18 | |
| InternLM | Language | 1.0e+24 | 1.6e+12 | Shanghai AI Lab, SenseTime | 2023-07-06 |
| Llama 2-7B | Language | 8.4e+22 | 1.5e+12 | Meta AI | 2023-07-18 |
| Llama 2-70B | Language | 8.1e+23 | 2.0e+12 | Meta AI | 2023-07-18 |
| Jais | Language | 4.9e+22 | 3.0e+11 | 2023-08-29 | |
| Falcon-180B | Language | 3.8e+24 | 2.6e+12 | Technology Innovation Institute | 2023-09-06 |
| Amazon Titan | Language, Image generation | 4.8e+24 | 4.0e+12 | Amazon | 2023-09-28 |
| CODEFUSION (Python) | Language | 7.9e+18 | 4.4e+6 | Microsoft, Microsoft Research | 2023-10-26 |
| ChatGLM3-6B | Multimodal, Language, Vision | 5.0e+22 | 1.4e+12 | Zhipu AI | 2023-10-27 |
| Skywork-13B | Language | 2.5e+23 | 3.2e+12 | Kunlun Inc. | 2023-10-30 |
| Yi-34B | Language | 6.1e+23 | 3.1e+12 | 01.AI | 2023-11-02 |
| Grok-1 | Language | 2.9e+24 | 6.2e+12 | xAI | 2023-11-04 |
| Nemotron-3-8B | Language | 1.8e+23 | 3.8e+12 | NVIDIA | 2023-11-15 |
| Qwen-72B | Language | 1.3e+24 | 3.0e+12 | Alibaba | 2023-11-30 |
| Qwen1.5-72B | Language | 1.3e+24 | 3.0e+12 | Alibaba | 2024-02-04 |
| Aramco Metabrain AI | Language | 1.0e+25 | 7.0e+12 | Saudi Aramco | 2024-03-04 |
| MM1-30B | Multimodal, Language, Vision | 4.9e+23 | 1.5e+12 | Apple | 2024-03-14 |
| DBRX | Language | 2.6e+24 | 9.0e+12 | Databricks | 2024-03-27 |
| Llama 3-70B | Language | 7.9e+24 | 1.5e+13 | Meta AI | 2024-04-18 |
| Yi-Large | Language | 1.8e+24 | 3.0e+12 | 01.AI | 2024-05-13 |
| Qwen2-72B | Language | 3.0e+24 | 7.0e+12 | Alibaba | 2024-06-07 |
| Nemotron-4 340B | Language | 1.8e+25 | 6.8e+12 | NVIDIA | 2024-06-14 |
| Llama 3.1-405B | Language | 3.8e+25 | 1.6e+13 | Meta AI | 2024-07-23 |
| AFM-on-device | Language | 4.5e+23 | 7.6e+12 | Apple | 2024-07-29 |
| AFM-server | Language | 4.3e+24 | 7.4e+12 | Apple | 2024-07-29 |
| Qwen2.5-32B | Language | 3.5e+24 | 1.8e+13 | Alibaba | 2024-09-17 |
| Qwen2.5-72B | Language | 7.8e+24 | 1.8e+13 | Alibaba | 2024-09-19 |
| Telechat2-115B | Language | 6.9e+24 | 1.0e+13 | China Telecom | 2024-09-20 |
| Doubao-pro | Language | 2.5e+25 | 8.4e+12 | ByteDance | 2024-10-28 |
| Hunyuan-Large | Language | 3.5e+24 | 7.0e+12 | Tencent | 2024-11-06 |
| Llama 3.3 70B | Language | 6.9e+24 | 1.5e+13 | Meta AI | 2024-12-06 |
| EXAONE 3.5 32B | Language | 1.3e+24 | 6.5e+12 | LG AI Research | 2024-12-09 |
| DeepSeek-V3 | Language | 3.4e+24 | 1.5e+13 | DeepSeek | 2024-12-24 |
| Doubao-1.5-pro | Language | NA | 9.0e+12 | ByteDance | 2025-01-22 |
| Hunyuan-TurboS | Language | NA | 1.6e+13 | Tencent | 2025-03-11 |
| Llama 4 Behemoth (preview) | Multimodal, Language, Vision | 5.2e+25 | 3.0e+13 | Meta AI | 2025-04-05 |
| Llama 4 Maverick | Multimodal, Language, Vision | 2.2e+24 | 3.0e+13 | Meta AI | 2025-04-05 |
| Llama 4 Scout | Multimodal, Language, Vision | 4.1e+24 | 3.0e+13 | Meta AI | 2025-04-05 |
| Pangu Ultra | Language | 1.1e+25 | 1.3e+13 | Huawei | 2025-04-10 |
| Qwen3-235B-A22B | Language | 4.8e+24 | 3.6e+13 | Alibaba | 2025-04-29 |
| Seed1.5-VL | Vision, Language, Multimodal, Video | 1.4e+24 | 3.0e+12 | ByteDance | 2025-05-11 |
| Kimi K2 | Language | 3.0e+24 | 1.6e+13 | Moonshot | 2025-07-11 |
| EXAONE 4.0 (32B) | Language | 2.7e+24 | 1.4e+13 | LG AI Research | 2025-07-15 |
| Qwen3-Coder-480B-A35B | Language | 1.6e+24 | 7.5e+12 | Alibaba | 2025-07-22 |
| GLM 4.5 | Language | 4.4e+24 | 2.3e+13 | Zhipu AI, Tsinghua University | 2025-08-05 |
| gpt-oss-20b | Language | 5.5e+23 | 2.3e+13 | OpenAI | 2025-08-05 |
| gpt-oss-120b | Language | 4.9e+24 | 1.5e+14 | OpenAI | 2025-08-05 |
Epoch's work is free to use, distribute, and reproduce provided the source and authors are credited under the Creative Commons BY license.
Download this data
CSV, Updated Nov. 24, 2025
Explore this data
Our comprehensive database of over 3500 models tracks key factors driving machine learning progress.