---
title: Synthetic continued pretraining
url: https://www.ml-quant.com/papers/arxiv/2409.07431/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-09-26
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2409.07431
source_url: https://arxiv.org/abs/2409.07431
featured: 2024-09-18
citations: 61
topic: LLMs & Text
---


# Synthetic continued pretraining

Researchers suggest using EntiGraph, a synthetic data augmentation algorithm, for synthetic continued pretraining to help language models answer questions and follow instructions related to source documents more effectively.

- Source: https://arxiv.org/abs/2409.07431
- Identifier: arXiv:2409.07431
- Released: 2024-09-11
- First featured: Quant Letter No. 66 (2024-09-18): https://www.ml-quant.com/issues/2024-09-18/
- Citations (Semantic Scholar): 61
- Published in: International Conference on Learning Representations
- Topic: LLMs & Text

## Related

- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://www.ml-quant.com/papers/arxiv/2406.20094/): A new method for creating synthetic data uses 1 billion diverse personas, potentially transforming large language model research and development.
- [Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources](https://www.ml-quant.com/papers/arxiv/2409.08239/): Synthetic Data for LLMs: Source2Synth, a new method for teaching Large Language Models new skills without human annotations, has improved multi-hop and tabular question answering by 22.57% and 25.51% respectively.
- [Better Alignment with Instruction Back-and-Forth Translation](https://www.ml-quant.com/papers/arxiv/2408.04614/): A new method, instruction back-and-forth translation, is introduced for creating high-quality synthetic data to improve large language models, outperforming other datasets on AlpacaEval.
- [Understanding Synthetic Context Extension via Retrieval Heads](https://www.ml-quant.com/papers/arxiv/2410.22316/): The paper finds that fine-tuning long-context language models with synthetic data improves performance in retrieval and reasoning tasks, with attention heads predicting performance.
- [Language Model Training on Edit Sequences Enhances Code Synthesis](https://www.ml-quant.com/papers/web/9b37a775db/): The paper presents LintSeq, a synthetic data generation algorithm that refactors code into a sequence of edits, resulting in more diverse programs and improved code synthesis performance.
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://www.ml-quant.com/papers/arxiv/2402.03300/): Advancing Math Reasoning in Language Models: DeepSeekMath7B is a new language model that uses web data and Group Relative Policy Optimization for advanced mathematical reasoning, scoring high on the MATH benchmark.