Real-world text, speech, and video sequences have different lengths, while GPUs work best on dense batches with one shape. Training a Transformer therefore requires deciding how to combine those sequences. Padding is simple, but wide length variation can spend more memory and compute on padding than on data. How can we reduce or remove it without changing the learning problem?

Why Is Padding Inefficient?

For lengths $L_1,\ldots,L_B$ and maximum length $L_{\max}$, a dense batch has shape

$X\in\mathbb{R}^{B\times L_{\max}\times d}$

with

$N_{\mathrm{real}}=\sum_{i=1}^{B}L_i,\qquad N_{\mathrm{pad}}=BL_{\max}-\sum_{i=1}^{B}L_i.$

Waste in token-wise FFNs and LayerNorm scales roughly with padded tokens. Dense Attention costs about $BL_{\max}^2$, versus $\sum_iL_i^2$ if each length is handled separately. One long outlier among short samples is especially expensive.

Properly masked padding is not necessarily training noise. If attention_mask excludes it and language-modeling labels use an ignore index such as -100, padding contributes no direct loss. The main problem is wasted compute and activation memory. Incorrect masks, including careless EOS/PAD reuse, can create a real training error.

1. Global Static Padding

Every sample is padded to a dataset-wide max_length. Shapes remain fixed, debugging and graph compilation are easy, but waste is largest when lengths vary, while a smaller limit truncates long samples. Without a strict static-shape requirement, this is rarely the best standalone choice.

2. Dynamic Padding

Dynamic padding extends samples only to the longest sequence in the current batch. A batch of [31, 28, 35, 30] ends at 35, while [180, 210, 192, 205] ends at 210. Hugging Face Data Collators support this directly.

It requires no model change, works across Transformer families, and retains compatibility with ordinary masks, distributed training, and evaluation. One outlier still pads the whole batch; changing shapes can expand compilation caches; and different maximums on DDP ranks can create stragglers. It is the general-purpose baseline, strengthened by length bucketing.

3. Length Bucketing and Sortish Sampling

Length bucketing groups similar samples. Fully sorting the dataset minimizes padding but destroys ordering randomness, so Sortish Sampling sorts within broad windows and shuffles buckets and samples again. Hugging Face Trainer's group_by_length=True combines this with dynamic padding.

Only the sampler changes, but strict sorting can bias batches when length correlates with label or domain. Shuffling each epoch remains important.

4. Token-Budget Batching Instead of Fixed Example Counts

batch_size=8 fixes examples, not work. Token-budget batching adds samples while

$\sum_{i\in\mathcal{B}}L_i\le T_{\mathrm{budget}}.$

It puts more short samples and fewer long ones in a batch, stabilizing token throughput and memory. Equal token counts do not mean equal Attention cost:

$1024^2\ne4\times256^2.$

Because example counts vary, teams must define whether loss is averaged per sample or valid token. Gradient accumulation is also more consistent when measured in tokens.

5. Sequence Packing

Packing fills otherwise empty space with other samples. For a maximum of 16 and lengths 7, 5, and 4:

[ A A A A A A A | B B B B B | C C C C ]

Efficient Sequence Packing without Cross-contamination reported that padding can occupy 50%, and in some settings 89%, of NLP tokens. It frames packing as bin packing and shows that blocking cross-sample Attention preserves equivalence to individual training.

Concatenate-then-Split in Pre-training

Decoder-only pre-training often joins documents with EOS and slices a token stream into fixed blocks. This nearly eliminates padding. Without block-diagonal masks, however, later documents can attend to earlier ones. That may be an intentional pre-training design, but is usually inappropriate for independent SFT or classification samples.

Preventing Cross-contamination

For independent samples $a$ and $b$, use a block-diagonal mask:

$$M_{ij}=\begin{cases}0&\text{if }i,j\text{ belong to the same sample}\newline-\infty&\text{otherwise.}\end{cases}$$

Position IDs may restart for each sample or continue across the pack depending on positional encoding. In SFT, prompt labels, assistant labels, EOS, and the first token of each sample also need explicit handling. Removing padding while breaking Attention or loss boundaries changes the task itself.

Which Packing Algorithm?

• Greedy / First Fit: fast and streamable, but leaves more gaps.
• Best-Fit Decreasing (BFD): places long samples first into the tightest remaining space; efficient but reorders data.
• Wrapped: concatenates everything and cuts fixed blocks; maximizes occupancy but can split samples and mix contexts.
• Semantic Packing: groups by topic as well as length. Methods such as Threshold Filtering Packing consider SFT quality but add similarity cost and possible bias.

TRL SFTTrainer supports packing=True, defaulting to BFD, and distinguishes bfd, bfd_split, and wrapped according to how long samples are split.

6. Padding-Free and Variable-Length Attention Kernels

Packing decides how to fill batches; a padding-free kernel ensures the GPU computes only real tokens. Samples can be flattened into

$X_{\mathrm{flat}}\in\mathbb{R}^{(\sum_iL_i)\times d},$

with cu_seqlens = [0, L_1, L_1+L_2, ...] marking boundaries. The kernel uses them to prevent Attention across samples.

ByteTransformer removes padding computation across BERT-like Transformers. FlashAttention-2 improves IO-aware exact Attention without materializing the full matrix in HBM. Enabling FlashAttention alone does not remove padding: dense [B,L_max,d] input remains dense. A variable-length path with flattened input is required.

Hugging Face's padding-free training documentation recommends DataCollatorWithFlattening; TRL can connect BFD packing to FlashAttention 2 or 3. Compatibility with custom biases, positional encodings, and multimodal layouts must still be tested.

7. Ragged and Nested Tensors

PyTorch Nested Tensor stores different lengths in a jagged layout and can connect to SDPA or FlexAttention. It avoids materialized padding, but unsupported operators or conversion back to padded tensors reintroduce copies and waste. It is best suited to custom models with a known supported operator set.

What Are 2025–2026 Studies Examining?

Packing Analysis finds that benefits vary with model and dataset scale: preprocessing complexity may dominate small experiments, while utilization matters increasingly from 8B to 70B models and 69K to 1.2M samples.

Equal token counts can still leave unequal Attention work. Libra notes that a pack costs roughly $\sum_jL_j^2$, leaving stragglers across data-parallel ranks and pipeline stages. The focus is shifting from removing padding, to balancing tokens, to balancing Attention FLOPs and communication.

For diffusion Transformers with highly variable image and video token counts, KnapFormer, Dynamic Context Parallelism, and ChunkFlow dynamically adjust sequence parallelism and sample placement. These address large long-context or multimodal clusters, not the first step of ordinary fine-tuning.

Comparison

Most Common Choices Today

For general encoders, classification, translation, and modest fine-tuning, use dynamic padding + Attention masks + length bucketing. For decoder-only pre-training, join documents with EOS into fixed token blocks while controlling document Attention and keeping global tokens per step stable. For SFT, start with dynamic padding and bucketing; when padding remains high, use BFD packing + correct sample boundaries + a FlashAttention variable-length path.

Implementation

from transformers import DataCollatorWithPadding, TrainingArguments

collator = DataCollatorWithPadding(tokenizer=tokenizer, pad_to_multiple_of=8)
args = TrainingArguments(
    output_dir="outputs",
    per_device_train_batch_size=8,
    group_by_length=True,
    length_column_name="length",
)

Padding to a multiple of eight can improve Tensor Core utilization even though it adds a few pad tokens. Minimum padding and minimum step time are not always identical.

from trl import SFTConfig

args = SFTConfig(
    output_dir="outputs",
    max_length=4096,
    packing=True,
    packing_strategy="bfd",
    eval_packing=False,
)

If long samples must be preserved, consider bfd_split or a separate chunking policy and verify that the model actually uses FlashAttention's padding-free route.

Production Checklist

1. Measure tokenized P50, P90, P95, and P99 lengths.
2. Measure truncated tokens and samples before fixing max_length.
3. Start with dynamic padding and length bucketing.
4. Record both pad_ratio and tokens/sec.
5. Normalize loss and accumulation by valid tokens where appropriate.
6. Unit-test cross-sample Attention, position IDs, EOS, and label masks after packing.
7. Profile whether a variable-length kernel is actually called.
8. In distributed training, compare $\sum_iL_i^2$ and step time across ranks alongside token counts.
9. Disable packing for evaluation when per-sample metrics and generations need easy separation.

Summary

Correctly masked padding is primarily computational waste, not learning noise. The safest improvement is dynamic padding with similar lengths grouped together. Higher efficiency comes from token budgets and packing, provided block-diagonal or variable-length Attention preserves sample boundaries.

The practical conclusion is simple: dynamic padding + length bucketing is the general baseline; BFD packing + padding-free FlashAttention is the high-efficiency LLM setup. At hundreds of thousands of context tokens and many GPUs, the next bottleneck is no longer padding but Attention work and load imbalance between distributed workers.