What Do LLMs Feed On? — Building and Acquiring Datasets

How LLMs Are Trained — 1. Datasets · 2. Modeling · 3. Sovereign AI

When people talk about LLMs, model size, GPU count, and new Attention architectures tend to attract attention first. They are easy to compare numerically in papers and convenient to feature in product materials. Yet once you actually train a model or adapt one to a particular domain, a large part of what determines its performance turns out to happen in the data. The same architecture can become a very different model depending on which documents it saw, how often similar sentences were repeated, whether the responses labeled as correct were genuinely good, and whether evaluation data leaked into training.

It is therefore misleading to think of dataset construction as simply collecting a lot of text. It is closer to securing data that you have the right to use, designing the distribution the model should learn, and tracking that design so it survives through training and evaluation. Volume still matters, but recent open research has repeatedly shown that quantity and quality cannot be considered separately. RefinedWeb, FineWeb, and DataComp-LM demonstrate rather transparently how much filtering, deduplication, and data mixing can change the value of the same raw sources.

There Is No Single Dataset: Each Training Stage Needs Its Own Data

It is easy to picture LLM data as one enormous corpus, but actual training uses several collections with different purposes in sequence. Pre-training data teaches a broad distribution of language and knowledge. Continued pre-training data adjusts the emphasis toward a domain such as law, medicine, or finance, or toward a language such as Korean. SFT data teaches the model the format and behavior of producing a useful answer to a question, while preference data tells it which of several answers is better. Reinforcement learning requires responses generated by the model itself and reward signals that evaluate those responses. Finally, evaluation data must remain uncontaminated by every one of these training sets.

This distinction matters because quality means something different at every stage. Broad coverage and natural context are important in pre-training, whereas a single incorrect SFT answer can directly teach the model the wrong behavior. Preference data often does not require an objectively correct answer, but its comparison criteria must remain consistent. Ranking all data with one filter and one score may look simple, but it rarely produces a good design.

Where Does the Data Come From?

The web remains the broadest source. Web data is large, diverse in subject matter, and continuously updated, but it also brings questions of usage rights and privacy, spam, automatically generated content, and language-identification errors. Public datasets and public-sector platforms tend to have clearer provenance and formats. In Korea, AI Hub provides many constructed datasets spanning speech, images, and text. But publicly accessible does not mean freely redistributable without conditions. Before using a dataset, you need to check its usage policy and rights terms.

Purchasing licenses or contracting with content providers costs money, but it lets you agree on the scope of rights and the update cycle. Data produced while operating a service—internal documents, support records, search logs, and user-corrected outputs—can be exceptionally well matched to the domain. At the same time, it is likely to contain personal information and trade secrets, and its original collection purpose may differ from model training. Consent, retention, and deletion policies therefore have to be designed along with the dataset. Human-written or human-reviewed data is especially useful for SFT and preference training, but it is expensive; if the task instructions are vague, each annotator ends up imagining a different model.

Then there is synthetic data. A strong teacher model can generate questions, answers, critiques, and revisions, making it possible to strengthen a scarce language or domain quickly. Synthetic data, however, does not create missing knowledge for free. It can reproduce the teacher's mistakes, writing style, and safety policy, and if its samples are too similar to one another, their information content will be much smaller than their apparent count suggests. It is safer to treat generation → rule-based or execution-based verification → deduplication → selective human review as a single pipeline.

Much More Work Remains After Collection

Once raw data arrives, the first task is to reconstruct document structure. If menus and ads in HTML, headers and footers in PDFs, OCR errors, code blocks, and tables are not separated from the main text, the model will see the debris of websites more often than sentences meant for people to read. Next come language and domain classification, followed by document-level quality filters. Rules can remove many documents that are too short, have abnormal character ratios, or repeat keywords as SEO spam. The stronger those rules become, however, the more legitimate material they also erase, including colloquial language, dialects, and short creative works. A common recent approach is to combine rule-based filters with a small classifier that recognizes high-quality documents, while checking that only a narrow set of writing styles does not survive.

Deduplication is more than a way to save memory. If exact copies, documents with copied paragraphs, template-variant articles, and translated repetitions all remain, the model may mistake frequently repeated claims for important facts and is more likely to memorize particular sentences. Exact hashes find identical documents quickly, while MinHash and locality-sensitive hashing help identify near duplicates. Areas such as code and mathematics, where recurring forms may carry meaning of their own, need separate deduplication thresholds.

PII and harmful-content removal is also necessary. Patterns such as email addresses and phone numbers can be found with rules, but personal information embedded in context and indirect identifiers are far harder. Rather than assuming filters eliminate every risk, the pipeline should retain source-specific preservation policies, access controls, and data lineage capable of responding to deletion requests. Copyright is similar: being visible on the internet does not automatically permit training and redistribution. The Korea Copyright Commission's Guide to Fair Use of Copyrighted Works for Generative AI Training explains that fair-use judgments depend on the particular mode of use, its market effect, and other circumstances. For dataset design, what matters is not a single checkbox pretending to replace legal advice, but a system that continuously tracks provenance, licenses, the scope of consent, and transformation history.

Mixing Data Means Designing the Model's Curriculum

Even cleaned data cannot simply be poured together. A model's strengths change depending on whether you preserve a raw distribution that is 90% web documents and 2% code, or deliberately raise the share of code and mathematics. Too much data-rich English may improve aggregate benchmarks while weakening Korean expression and cultural context. Repeating only Korean, on the other hand, can reduce knowledge coverage and multilingual transfer. Because Korean has extensive particle and ending variation, the tokenizer's vocabulary and fertility must also be considered. Splitting the same sentence into too many tokens raises both training and inference cost and reduces the information that fits within a long context.

Curricula that begin with broad, varied data and increase the share of high-quality or domain-specific data toward the end are also common. OLMo 2 makes its data mixture, late-stage curriculum, and even checkpoints public, showing clearly that this is part of the training recipe rather than mere housekeeping. What matters is not the sentence “we increased the proportion of high-quality data,” but how quality was scored and whether ablations reveal the effect on general and specialist capabilities separately.

Evaluation-contamination checks need to finish before mixing. It is not enough to search for benchmark questions and answers verbatim; paraphrases, explanations, and translations can also enter the training corpus. This blurred boundary is one reason recent benchmark scores can diverge from how models feel in actual use. Public benchmarks should be supplemented with holdouts kept private until just before release, time-split recent data, and failure cases gathered from real work. Together, they make it somewhat clearer what the model memorized and what it generalized.

What SFT and Preference Data Define

The central concern in SFT data is the response standard, not the number of prompts. Accuracy, relevance, brevity, supporting evidence, refusal behavior, and tool-use rules can conflict within one answer, so annotation guidelines need explicit priorities. If “admit when you do not know” and “always be helpful” are emphasized at the same time without resolving the tension, raters will select different answers. A good SFT set does more than collect successful examples: it shows how common failures should be corrected and varies length and format so that the longer answer does not always win.

Preference data has the same issue. Pairwise comparison between two answers is more stable than assigning absolute scores, but label noise rises when the question is ambiguous or the answers differ only slightly. It helps to measure inter-rater agreement, record rationales, and distinguish easy comparisons from hard ones. After deployment, follow-up questions, copied responses, user edits, and explicit feedback can all become learning signals, but a single click should not immediately be interpreted as quality. User behavior depends on the interface and the work setting, so validating what the signal actually means is necessary alongside consent and anonymization.

A Practical Strategy for Acquiring Data

For most teams, building a giant general-purpose corpus from scratch is inefficient. A better starting point is to narrow the target to a few capabilities, create evaluation sets that measure them, and inventory which areas are covered by public, licensed, and internal data. Human-authored or synthetic data can fill the gaps, followed by small experiments comparing the efficiency of each source. Without an ablation that fixes model size and changes only the data, it is difficult to tell whether an improvement came from the dataset, the learning rate, or the token count.

Once a service launches, its failures become the most valuable data. Collect the questions the model gets wrong, classify the cause as missing knowledge, instruction-following failure, retrieval failure, tool error, or safety-policy error, and route each case back to the appropriate training stage. Adding every failure to SFT may hide symptoms without addressing causes. RAG may be the better answer to current-knowledge problems; SFT to output-format problems; preference training to selection criteria; and RL with verifiable rewards to mathematical and coding problems that require exploration.

In the end, a good dataset is not a collection of files but an operating system connecting rights, distribution, quality, and evaluation. More important than how much you collected is whether you can explain why each dataset was included, what was removed, and which capability those decisions produced. If data is the model's textbook, data lineage and evaluation sets are the ledger that records the book's sources and the scope of its exam. Only with that ledger can we judge what pre-training, distillation, and reinforcement learning—the subjects of the next article—actually improved.

Next: How Are LLMs Trained? — From Pre-training to Distillation and RL