Gradient Accumulation Loss Bug: Why Effective Batch Is Wrong

The Gap Between Sequences and Tokens

When configuring distributed fine-tuning, every machine learning engineer recites the canonical formula for effective batch size: per-device batch size multiplied by gradient accumulation steps multiplied by world size (the total number of GPU ranks). On paper, running a batch size of 16 with 1 accumulation step should produce the exact same parameter update as running a batch size of 4 with 4 accumulation steps. Yet across fine-tuning runs with variable-length sequences, these configurations diverge.

The arithmetic that teams calculate counts sequences, but the cross-entropy loss function normalizes over individual tokens. The gradient accumulation bug lives entirely in the gap between these two denominators. When you scale sequence counts, you assume that dividing the accumulated loss by the number of accumulation steps is mathematically equivalent to computing the global token average over the full batch. It is not.

Before analyzing the math, let us separate adjacent failure modes. If your loss spikes after resuming from a checkpoint, the culprit is unpreserved optimizer state or learning rate scheduler drift. If you are burning GPU cycles on zero-padded regions, sequence packing optimizes that throughput, though packing changes per-micro-batch token counts without resolving the underlying normalization error. Similarly, gradient checkpointing is an activation recomputation strategy designed to reduce VRAM footprints, sharing a similar name but solving a completely different problem. The failure discussed here belongs strictly to loss normalization arithmetic during micro-batch accumulation.

• Sequence-level batching assumes every sequence contributes equal gradient magnitude to an optimizer step.
• Token-level loss reduction divides the cross-entropy sum by the number of non-padded, active tokens.
• Gradient accumulation historically applied a sequence-level average over micro-batches that each contained disparate token counts.
• The result is a silent distortion where the optimizer receives skewed gradient directions while reporting plausible loss values.

The Mechanism: Why Mean-of-Means Distorts Weights

PyTorch's CrossEntropyLoss takes reduction='mean' as its default, and the documentation for ignore_index says the loss is averaged over non-ignored targets: with mean reduction the per-token losses are summed and divided by the count of targets that are not the ignore index. Inside a single forward pass, the cross-entropy loss sums the negative log-likelihoods of all active tokens and divides that sum by the number of loss-contributing tokens in its specific micro-batch. When gradient accumulation is enabled, a standard training loop executes several forward and backward passes before invoking the optimizer. To keep the accumulated gradient at the expected scale, the framework divides each micro-batch loss by the number of accumulation steps before calling backward.

This creates a mean-of-means. Instead of computing the total loss across all tokens in the effective batch and dividing once by the total token count, the trainer calculates an arithmetic mean of several independent averages. A mean-of-means equals the true global token mean if and only if every micro-batch contains the exact same number of loss-contributing tokens. With variable-length data, that condition never holds.

Consider a concrete example with 2 accumulation steps where the full effective batch contains 100 loss-contributing tokens. Micro-batch 1 contains a short sequence of 10 active tokens, while micro-batch 2 contains a long sequence of 90 active tokens. The true global loss should be the sum of all 100 token losses divided by 100. In naive gradient accumulation, the trainer computes the mean loss of micro-batch 1 (sum divided by 10), computes the mean loss of micro-batch 2 (sum divided by 90), scales both by 1/2, and adds them together.

Because the trainer assigns a flat 1/2 weight to each micro-batch average, it treats the 10-token micro-batch as if it represented 50 percent of the total batch information. Consequently, each token in the 10-token micro-batch receives an effective gradient weight of 1/(2 * 10) = 1/20, whereas each token in the 90-token micro-batch receives an effective gradient weight of 1/(2 * 90) = 1/180. A single token in the short sequence is weighted 9 times heavier than a token in the long sequence during the exact same optimizer step. The distortion ratio matches the inverse ratio of the token counts.

How Completion-Only Masking Multiplies the Error

In supervised fine-tuning (SFT) and instruction tuning, engineers rarely compute loss over the prompt. Standard practice sets the prompt label tokens to -100 so that PyTorch's cross-entropy loss ignores them, computing loss only over the completion tokens. This completion-only masking magnifies the normalization distortion beyond what raw sequence length variance would suggest.

Many engineering teams attempt to mitigate accumulation errors by bucketing sequences of similar total length into the same batch or by padding sequences to a uniform length. However, when prompt masking is applied, two sequences padded to the exact same length of 2048 tokens can yield drastically different active token counts. Sequence A might consist of an 1800-token prompt followed by a 200-token completion. Sequence B might feature a 100-token prompt followed by a 1900-token completion.

• Equal sequence lengths do not equal identical loss-contributing token counts under completion masking.
• A micro-batch dominated by short completions has its per-token gradients scaled up significantly compared to long-completion micro-batches.
• Instruction datasets with high prompt-to-response variance experience severe gradient skewing across accumulation windows.
• Sequence packing packs multiple examples into fixed windows, changing the per-micro-batch distribution but leaving the mean-of-means bug intact unless global normalization is enforced.

Because the loss denominator is determined solely by the unmasked label count, any variation in completion length between micro-batches skews the gradients. Even with strict sequence-length bucketing, completion masking breaks the assumption of uniform token counts across accumulation steps.

The Silent Failure: Symptoms Without Errors

The most dangerous aspect of the gradient accumulation bug is that it fails silently. No CUDA Out-of-Memory ( OOM errors) occur, no warnings are raised in standard logs, and the loss curve appears normal. The model continues to train and converge, but it optimizes an altered objective where shorter completions dominate the gradient updates.

The observable symptoms appear as secondary discrepancies during scaling. Hyperparameters tuned on a single GPU with 16 accumulation steps fail to transfer when moving to 4 GPUs with 4 accumulation steps, or when scaling out to multi-node clusters. Engineers observe that changing accumulation steps shifts downstream evaluation metrics, such as benchmark accuracy or instruction adherence, despite maintaining an identical theoretical effective batch size.

• Non-transferable learning rates: An optimal learning rate found at accumulation step 8 causes instability or underfitting when switching to accumulation step 1.
• Downstream quality drift: Models trained on smaller per-device batches with high accumulation develop biases toward shorter responses due to prompt-masking gradient distortion.
• Weight divergence: In a writeup dated 15 October 2024, Unsloth trained a LoRA adapter with the full batch (bsz=16, ga=1) and compared the resulting weights against every accumulated combination down to bsz=1, ga=16. It reported an inherent accumulation error due to floating-point arithmetic of 0.0068 L2 norm, and an L2 norm difference that increased from 0.0196 to 0.0286 as the number of gradient accumulation steps increased. Full fine-tuning runs are exposed to the same denominator arithmetic.
• Unexplained multi-GPU divergence: Switching from local single-GPU prototyping to multi-GPU Distributed Data Parallel (DDP) alters the token weighting across ranks, producing different model checkpoints.

When teams observe these discrepancies, they often blame floating-point non-determinism, CUDA kernel variations, or learning rate warm-up schedules. In reality, the mathematical objective function itself changes whenever accumulation steps or batch splits are altered.

The Fix: Normalizing Across the Whole Accumulation Window

The architectural solution requires a single conceptual shift: normalise once, over the total number of active tokens in the entire accumulation window. Instead of computing a mean loss on each micro-batch, the trainer must compute an un-normalised, sum-reduced cross-entropy loss for each micro-batch, accumulate those sums, and divide by the total count of loss-contributing tokens across all accumulated steps.

In practice, implementing this requires handling the denominator before running backward passes. If backward is called on each micro-batch sequentially, the exact total token count of the entire accumulation window must be known in advance to scale the micro-batch loss correctly. Frameworks accomplish this by pre-scanning the micro-batches in the current accumulation window to sum the unmasked tokens, or by accumulating sum-reduced gradients and rescaling the parameters prior to the optimizer step.

This defect was historically present in mainstream training frameworks. In a post published on 16 October 2024, Hugging Face wrote that for gradient accumulation across token-level tasks like causal LM training the correct loss should be computed as the total loss across all batches in a gradient accumulation step divided by the total number of all non-padding tokens in those batches, which is not the same as the average of the per-batch loss values, and showed the patch that switched the causal-LM loss to reduction='sum' divided by num_items. That post named pull request 34191 as the change shipping the fix for the most popular models, and transformers v4.46.0 was tagged on 24 October 2024. Because custom training loops and legacy codebases frequently override compute_loss, engineering teams should check their specific installed framework versions and loss implementations against that fix rather than assuming library defaults handle global token reduction.

In distributed environments (DDP or FSDP), the mean-of-means distortion reappears across GPU ranks. If Rank 0 processes 120 tokens while Rank 1 processes 40 tokens, standard gradient all-reduce operations average the gradients equally across both ranks (multiplying by 1 / world_size). Correct global normalization requires all-reducing the total token count across all ranks and scaling local sums by that global denominator, ensuring the framework's native world-size division is properly balanced.

The Runnable Diagnostic: Proving the Denominator Bug

To determine definitively whether your current training pipeline is affected by the gradient accumulation denominator bug, run this four-step diagnostic test on your cluster.

1. Prepare a test dataset with high sequence-length variance (for example, completions ranging from 32 tokens to 1024 tokens) and set a fixed random seed.
2. Run Configuration A: Set per-device batch size to N and gradient accumulation steps to 1. Disable dropout and ensure deterministic data loading. Execute exactly 1 optimizer step and log both the reported loss and the gradient L2 norm.
3. Run Configuration B: Set per-device batch size to N/k and gradient accumulation steps to k on the exact same dataset batch, using identical random seeds and disabled dropout. Execute 1 optimizer step and log the loss and gradient norm.
4. Run the Control Test: Repeat both Configuration A and Configuration B on a synthetic dataset where every sequence has the exact same number of active, unmasked completion tokens.

Analyze the output between the runs. If Configuration A and Configuration B yield different losses and gradient norms on the variable-length dataset, but produce identical metrics to numerical precision on the uniform-length control dataset, your pipeline suffers from the token denominator bug. If the discrepancy persists on the uniform dataset, investigate other sources of variance, such as optimizer state initialization, learning rate scheduler updates, or non-deterministic data loaders.

Running the A/B on Serverless Training

The two-config A/B is a small submit-and-compare workload rather than a machine anyone needs to hold, which makes it a natural fit for Serverless Training. You submit the training job, it is containerized and executed with job start in under 60 seconds, using S3-compatible storage and pulling custom images directly from ECR, GAR, or Docker Hub.

Training jobs are served from European data centres in Paris and Finland within the EU / EEA, which keeps the run inside European data residency boundaries. For large-scale workloads and European AI infrastructure, Serverless Training provides access to high-performance GPUs with the following published fleet VRAM capacities: NVIDIA L40S (48 GB), NVIDIA H100 (80 GB), A100 (80 GB), H200 (141 GB), and B200 (192 GB). Availability and service level agreements are agreed individually per business contract.

Lyceum executes your containerized workloads without modifying your model definitions or loss functions. Whether you are running single-device tests or scaling across distributed nodes, our sovereign infrastructure delivers transparent per-second compute without egress penalties. For live GPU availability and current infrastructure rates, consult our pricing page (lyceum.technology/pricing).