Solving OOM Errors in 70B Model Fine-Tuning

Before you can fix an OOM error, you must understand where your memory is going. When fine-tuning a 70B model, VRAM is consumed by four primary components: model weights, gradients, optimizer states, and activations. If you are using standard AdamW optimization in 16-bit precision, the numbers are staggering.

• Model Weights

  70 billion parameters at 2 bytes (FP16/BF16) = 140 GB.

• Gradients

  Another 2 bytes per parameter = 140 GB.

• Optimizer States

  AdamW stores two states per parameter in FP32 (4 bytes each) = 560 GB.

• Total Static Memory

  840 GB.

This total does not even include activations, which scale with your batch size and sequence length. On a single 80GB H100, you cannot even load the model, let alone train it. Even on a cluster of eight H100s (640GB total VRAM), you are still 200GB short of just the static requirements. This is why naive fine-tuning scripts fail instantly with a RuntimeError: CUDA out of memory.

To survive the 70B threshold, you must move away from full-parameter fine-tuning in high precision. You are essentially fighting a war against the 8-byte-per-parameter overhead of the AdamW optimizer. Your first line of defense is reducing the precision of these states or sharding them across your entire fabric.

If you cannot shard the model across 16+ GPUs, Parameter-Efficient Fine-Tuning (PEFT) is your only path forward. Specifically, QLoRA (Quantized Low-Rank Adaptation) has become the gold standard for 70B models. QLoRA works by quantizing the base model to 4-bit NormalFloat (NF4) and only training a small set of adapter weights in higher precision.

The memory savings are transformative. By using 4-bit quantization, the 140GB weight requirement for a 70B model drops to roughly 35GB. Because you are only training adapters, the optimizer states and gradients only apply to a fraction of the total parameters (usually less than 1%).

As shown in the table, QLoRA allows a 70B model to fit comfortably on a single 80GB GPU with room to spare for activations. According to the original QLoRA research paper from the University of Washington, this method maintains virtually the same performance as full-precision fine-tuning. For European enterprises concerned with efficiency and cost, QLoRA on sovereign infrastructure provides the best balance of performance and resource utilization.

When you move beyond a single GPU, you need a strategy to shard the model. The two industry standards are PyTorch Fully Sharded Data Parallel (FSDP) and Microsoft DeepSpeed. Both aim to solve the same problem: making the model appear as one giant pool of VRAM across multiple cards.

DeepSpeed ZeRO-3 (Zero Redundancy Optimizer) is the veteran choice. It shards the optimizer states, gradients, and parameters. When a layer needs to be computed, ZeRO-3 fetches the necessary shards from other GPUs, performs the calculation, and then discards them. This allows you to fit models that are theoretically much larger than any single GPU's memory.

However, for 70B models, we often recommend FSDP for its native integration with the PyTorch ecosystem and superior performance on modern NVIDIA architectures. FSDP's FULL_SHARD strategy is particularly effective. It ensures that no single GPU holds a full copy of the model, effectively turning an 8x H100 node into a single 640GB memory unit. According to the 2025 PyTorch Performance Report, FSDP can achieve up to 90% scaling efficiency on high-bandwidth interconnects like NVLink.

At Lyceum, our orchestration layer automates the configuration of these distributed environments. We handle the NCCL backend tuning and the interconnect topology so you can focus on the weights, not the networking. When you deploy a 70B fine-tuning job on Lyceum Cloud, the environment is pre-optimized for FSDP, ensuring that your inter-GPU communication doesn't become the new bottleneck after you solve the memory issue.

Even with sharding and quantization, you might still hit OOM errors during the backward pass. This is usually due to activation memory. Every intermediate calculation in the forward pass is stored to compute gradients later. For long context windows (e.g., 8k or 32k tokens), this can consume hundreds of gigabytes.

1. Gradient Checkpointing

   This is non-negotiable for 70B models. Instead of storing all activations, you only store a few 'checkpoints.' During the backward pass, you re-compute the missing activations. This trades a 30% increase in compute time for a massive 70-80% reduction in VRAM usage.

2. Gradient Accumulation

   If your batch size of 1 is still causing OOM, use a micro-batch size of 1 and accumulate gradients over multiple steps. This simulates a larger batch size without the memory cost.

3. CPU Offloading

   If you are truly desperate for VRAM, DeepSpeed and FSDP allow you to offload optimizer states to system RAM. Be warned: this introduces significant latency as data moves over the PCIe bus, which is much slower than NVLink.

We recommend starting with Gradient Checkpointing enabled by default. In our internal benchmarks, fine-tuning Llama 3 70B with a sequence length of 4096 on 4x A100 (80GB) was only possible once checkpointing was active. Without it, the job crashed before completing the first step.