Sequence Packing: Reclaiming GPU Hours in Long-Context SFT

AI This article was created with the help of AI.

The Padding Tax in Long-Context SFT

When fine-tuning large language models on long-context tasks, batching is typically configured around a fixed maximum sequence length. In supervised fine-tuning (SFT), datasets rarely follow a uniform length distribution: NVIDIA's NeMo documentation notes that many fine-tuning datasets have a skewed distribution of sequence lengths, with many short sequences and a few long ones, following Zipf's Law. Because transformer models require fixed-length inputs, shorter sequences must be padded, and the computation performed on those pad tokens is eventually masked out, resulting in wasted GPU computation. Without optimization, long-context ablations and full parameter runs across large clusters can therefore spend a large share of their GPU hours on tokens that carry no gradient signal.

You can quantify this inefficiency directly from your tokenized dataset before changing a single line of training code. The core metric is token occupancy, defined as the ratio between real information-carrying tokens and the total tensor volume allocated to the batch:

• Token Occupancy = (Sum of non-pad tokens in batch) / (Batch Size * max_len)
• Padding Fraction = 1.0 - Token Occupancy
• FLOP Waste = 1.0 - Token Occupancy (linear lower bound for feedforward blocks)

Plotting your dataset length histogram against the target context window immediately reveals the extent of the collapse. When the tail of long documents forces max_len outward, token occupancy can fall to a small fraction of the allocated tensor volume. It is critical to distinguish token occupancy from hardware metrics like GPU utilization or Model FLOPs Utilization (MFU). Monitoring tools such as nvidia-smi measure whether tensor cores and memory buses are active, not whether the arithmetic is useful. A cluster reporting near-full GPU utilization can still spend much of its cycles computing self-attention over masked pad tokens, creating severe idle cost waste that remains invisible on infrastructure dashboards.

Attention Complexity vs. Mask Elision

The computational penalty of padding is substantially worse than the raw token fraction suggests. While multilayer perceptron (MLP) layers scale linearly with sequence length (O(N)), standard self-attention scales quadratically (O(N^2)). When an input sequence is inflated with pad tokens to fill a rectangular tensor, the attention layer computes pairwise dot products across the entire padded grid. Even if attention logits corresponding to pad tokens are subsequently masked with negative infinity before softmax, the forward and backward GEMM operations still execute across the full dimension unless specialized kernels intervene.

Modern fused kernels mitigate some of this overhead, but their mechanisms are often misunderstood. The primary contribution of FlashAttention is IO-awareness: tiling queries, keys, and values to compute attention within fast on-chip SRAM rather than materializing the intermediate N x N score matrix in high-bandwidth memory (HBM). FlashAttention does not change the fundamental O(N^2) complexity of the mathematical operation; it reduces memory traffic bottlenecks. Work elision on masked regions occurs only when specific variable-length (varlen) API paths or block-sparse layouts are invoked.

Hardware execution characteristics also vary dramatically across kernel generations. FlashAttention-2 was tuned for NVIDIA Ampere architectures, reaching 50-73% of the theoretical maximum FLOPs/s on A100 and getting close to the efficiency of GEMM operations. On the H100 it achieves only 35% utilization, because it does not exploit newer Hopper capabilities. FlashAttention-3 redesigns the kernel for Hopper GPUs (such as H100 and H200), exploiting asynchrony of the Tensor Cores and the Tensor Memory Accelerator (TMA) to overlap computation and data movement via warp specialization, to interleave block-wise matmul and softmax operations, and to use block quantization for FP8 low-precision support. FlashAttention-3 does not run on Ampere architectures.

Because attention is quadratic while feedforward layers are linear, you cannot assume your GPU-hour savings will directly equal your dataset padding fraction. To evaluate actual improvements, benchmark tokens per second before and after enabling optimization. Ensure you hold model architecture, numeric precision, GPU count, distributed topology, per-device batch size, and input-output distributions strictly constant. Always report aggregate system throughput rather than per-user interactive latency, as mixing the two creates misleading performance numbers.

Step One: Length-Grouped Batching

The simplest step to reduce padding overhead without modifying model forward definitions is length-grouped batching, often exposed as group_by_length in high-level training frameworks. Instead of drawing training examples uniformly at random from the shuffled dataset, the dataloader sorts or buckets samples by their tokenized sequence lengths before assembling micro-batches.

Under length-grouped batching, short sequences are batched with other short sequences, while the handful of very long documents form batches of their own. The collator pads each micro-batch only to the maximum length present within that specific batch, rather than the global max_len of the entire dataset. For a batch whose longest example is a few hundred tokens short of the bucket ceiling, the allocated context length tracks that batch instead of the dataset-wide maximum, instantly recovering memory and compute.

• Implementation: Set group_by_length=True in training configuration files (such as Hugging Face TRL or Transformers).
• Memory profile: VRAM allocation becomes dynamic across steps, requiring sufficient headroom to accommodate the largest sequence clusters without triggering OOM errors.
• Throughput gain: Recovers a meaningful share of wasted padding FLOPs in typical right-skewed conversational datasets with minimal engineering friction, though the exact share depends on your own length distribution and must be measured.

However, length-grouped batching introduces a statistical trade-off: batch composition is no longer independent and identically distributed (IID). Grouping by length correlates mini-batch gradient updates with document size. Short prompt-response pairs dominate certain optimizer steps, while complex multi-step reasoning traces dominate others. In practice, this correlation introduces gradient variance that can alter convergence dynamics on smaller datasets, making it an intermediate compromise rather than a definitive solution for long-context workloads.

Step Two: Padding-Free Training (Varlen)

Padding-free training, commonly referred to as variable-length or varlen training, eliminates pad tokens entirely. Instead of constructing a 2D or 3D tensor of shape [batch_size, seq_len] with trailing pad IDs, the dataloader flattens all tokens across the micro-batch into a continuous 1D array of shape [1, total_real_tokens].

To prevent tokens from different documents from attending to each other, the training pipeline passes an auxiliary metadata tensor called cu_seqlens (cumulative sequence lengths). This 1D tensor of integers defines the boundary offsets of each packed document within the flattened stream. For example, if a micro-batch contains three sequences of lengths 128, 256, and 512, cu_seqlens is defined as [0, 128, 384, 896]. The conventional alternative is a custom block-triangular attention mask; NVIDIA's NeMo instead routes packed sequences through the variable-length attention kernels in FlashAttention and TransformerEngine, passing sequence-boundary information in the cumulative sequence length variable so that attention values between sequences are never calculated.

1. Token Concatenation: Extract raw input IDs from variable-length sequences and concatenate them sequentially into a 1D tensor.
2. Offset Generation: Compute torch.cumsum of the individual sequence lengths, prepending a zero index to establish cu_seqlens.
3. Kernel Dispatch: Pass the flattened tensor and cu_seqlens directly into a varlen kernel API (such as flash_attn_varlen_func or TransformerEngine).

Varlen execution carries no pad tokens by definition: no compute cycles or memory bytes are spent on padding. Because boundary separation is enforced directly inside the kernel loop via cu_seqlens, cross-document attention contamination is prevented by construction without allocating memory for padding masks. This approach significantly reduces activation memory overhead, allowing larger effective batch sizes when managing extensive context length requirements.

Step Three: True Sequence Packing

While variable-length training provides high efficiency, its dynamic tensor shapes present operational challenges for optimized production runtimes. Modern deep learning compilers, such as torch.compile and CUDA graph capture, require static tensor shapes to eliminate kernel launch overhead and optimize memory layout. Variable-length tensors change dimensions at every iteration, forcing graph recompilations or falling back to slower dynamic kernel launches.

True sequence packing resolves this by packing variable-length sequences into fixed-length vectors of length pack_len using offline or streaming bin-packing algorithms. Rather than simple chunking, bin-packing heuristics like First-Fit-Decreasing (FFD) or Best-Fit sort sequences by length and assign them into fixed-size bins to maximize fill rates, pushing occupancy close to the buffer ceiling while preserving static tensor shapes.

The trade-off of true sequence packing is that you must handle sequence boundaries explicitly in your model definition. When using compiled backends, you can enforce attention isolation using modern masking primitives like PyTorch FlexAttention. The FlexAttention API lets you express variants such as document masking and sample packing in a few lines of idiomatic PyTorch, then lowers that function into a single fused FlashAttention kernel through torch.compile, generating a kernel that does not materialize any extra memory and that can take advantage of sparsity in the attention mask.

In FlexAttention, document separation is achieved by generating a block mask from a document_id tensor that maps each token in the pack to its parent sequence index:

• Define document index array: document_id = [0, 0, 0, 1, 1, 2, 2, 2...] for all tokens in the pack.
• Construct mask function: def doc_mask(b, h, q_idx, kv_idx): return document_id[q_idx] == document_id[kv_idx]
• Generate sparse mask: block_mask = create_block_mask(doc_mask, B=None, H=None, Q_LEN=pack_len, KV_LEN=pack_len).
• Execute attention: pass the block mask into flex_attention(query, key, value, block_mask=block_mask); the user-defined function is lowered into a single fused kernel by torch.compile, which also takes advantage of the mask's sparsity.

The Three Silent Correctness Failures

Implementing sequence packing or varlen training provides immediate throughput gains, but naive implementations frequently introduce three silent bugs that degrade final model quality without raising runtime exceptions.

The first failure is cross-contamination. In standard causal language modeling, attention masks are lower triangular. If multiple documents are concatenated into a single pack without boundary masking, tokens in the second document attend to tokens in the first document. In instruction fine-tuning, this causes the model to condition its answers on unrelated prompts from preceding sequences, polluting associative memory and degrading task performance.

The second failure involves Rotary Position Embeddings (RoPE) and position IDs. If position IDs are not reset at sequence boundaries, the position index increments monotonically across the entire pack length. A short query packed at the end of a buffer then receives position IDs drawn from the far end of the pack rather than starting at zero. This forces the model to evaluate rotary embeddings in extrapolation regimes it may not support, severely damaging retrieval accuracy and short-context understanding.

The third failure is loss normalisation. Standard training loops average token cross-entropy loss over the total sequence length. When multiple documents share a pack, unweighted averaging reweights each document's contribution proportionally to its token length, causing long documents to dominate gradient updates. Label masking must also survive the packing transform: trainers exclude tokens from the loss by setting their labels to an ignore index (-100 by default), which is how padding and, for prompt-completion datasets, the prompt itself are kept out of the objective. Loss must then be normalized explicitly by the global count of active completion tokens across the distributed batch.

• Boundary isolation: Use block-diagonal masks, cu_seqlens, or BlockMask to guarantee zero cross-sequence attention.
• Position ID resets: Reset position indices to zero at the first token of every concatenated document before computing rotary embeddings.
• Active token normalisation: Compute loss by dividing the sum of unmasked token cross-entropy values by the total number of non-ignored label tokens (labels!= -100).
• Document atomicity: Use bin-packing rather than naive chunking to avoid splitting examples across buffer boundaries mid-sentence.

Running Packed SFT on Managed GPU Infrastructure

Once your training script correctly implements sequence packing, varlen dataloaders, and position resets, running high-throughput jobs requires compute infrastructure that eliminates provisioning friction and idle hardware waste. Distributed SFT workloads on European data infrastructure benefit from direct execution models where infrastructure matches the scale of the dataset.

Lyceum Serverless Training is designed for managed execution of containerized training and fine-tuning workloads. You provide your container image from Docker Hub, GitHub Packages, Amazon ECR, or Google Artifact Registry (GAR), and the platform handles cluster orchestration and execution with job start times in under 60 seconds. Storage integrates with S3-compatible endpoints, allowing checkpoints and dataset shards to stream directly into compute instances.

The compute fleet provides dedicated hardware across European facilities in Paris and Finland: NVIDIA L40S (48 GB) and NVIDIA A100 (80 GB) instances located in France and the Nordics, alongside NVIDIA H100 (80 GB), NVIDIA H200 (141 GB), NVIDIA B200 (192 GB), and NVIDIA B300 accelerators deployed across the EU and EEA. Every Serverless Training workload is billed precisely per GPU-hour, ensuring that gains achieved through sequence packing directly reduce your total infrastructure bill. Service level agreements (SLAs) for Serverless Training are agreed individually per business contract. For engineering teams that prefer direct SSH access and custom cluster management, the On-demand GPU VM provides raw compute billed per second with zero egress fees.

• Measure your baseline: Calculate dataset token occupancy before changing architectures to establish true padding overhead.
• Choose the right implementation: Adopt varlen kernels (cu_seqlens) for dynamic pipelines or bin-packed static buffers with FlexAttention BlockMask for compiled graphs.
• Verify correctness: Enforce block-diagonal attention masking, reset position IDs to zero per document, and normalize loss by global active label counts.

To scale your fine-tuning jobs on sovereign European GPU infrastructure with transparent per-GPU-hour billing, explore current capacity and specifications on the pricing page.