Keep GPU jobs running: memory, orchestration and recovery

A checkpoint earns your trust when a fresh process can load it and continue training. Make that recovery test part of the workload.

Why does our GPU training job fail, stall or resume with different behaviour?

The short answer

Keeping GPU jobs running starts with separating failure types. An out-of-memory error needs a different investigation from a stalled worker or a bad restart. Record the error, input, code version, environment and last completed step before changing the job.

Lyceum is a sovereign European AI inference provider for open-source models. For teams training their own models, our serverless training runs a plain Docker container on the GPU they choose. Our GPU VMs provide root access in European data centres when you need to manage the environment yourself.

The storage boundary matters in both cases. Our persistent storage bucket survives runs and VMs; a VM’s own disk is wiped on termination. Docker runs mount the bucket at `/mnt/s3` by default. VMs do not mount it automatically, so configure access before relying on it for recovery.

One hypothetical interrupted run will carry the decisions below. Assume the job fails at minute 95, the latest complete checkpoint is from minute 90, and restoration takes five minutes.

The calculation excludes queueing and checkpoint-writing time. It is not a measured recovery time or a promise. Its purpose is to show what you must measure: the work lost and the time needed to restore it.

Step 1: Memory & OOM

Memory failures mean the job cannot obtain the memory it needs at that point. OOM means out of memory. Read this if a run fails as the batch, sequence or model grows.

Begin with the smallest input that reproduces the error. Record memory use around the failing operation. PyTorch Profiler can report memory consumed by tensors, the arrays holding model data, and record operation input shapes.

Change one variable at a time and keep the result. A smaller batch may fit while the model’s learned values or the information used to update them still exceed available memory. Measure the whole step, including the update to the model.

Reading route:

• Where GPU memory goes
• Using the PyTorch memory profiler
• Preventing training memory failures
• Fine-tuning memory errors
• Large-model memory failures
• Repeating calculations to use less memory
• Training a small add-on versus the full model

In the interrupted example, first establish whether memory caused the failure at minute 95. More frequent saves would limit replay, but would not repair a repeatable memory failure.

Step 2: Distributed Training

Distributed training coordinates work across multiple processes or machines. Read this if a small run works but additional GPUs bring stalls or failures.

Keep the small working run as a reference. Check that workers receive the intended configuration, can communicate and agree on the workload. Capture the earliest error across workers before treating every later message as a separate problem.

When a worker fails, joins or leaves, PyTorch’s torchrun launcher stops the surviving workers so the group can restart together. Its documentation tells you to checkpoint progress and choose frequency according to tolerance for lost work.

Reading route:

• Setting up distributed training
• Splitting a model across GPUs
• Dividing model data across GPUs
• Training on long contexts
• Balancing training across a model’s specialist parts
• Synchronising training and serving weights

A restart of the worker group is only one part of recovery. Test that the training programme loads the intended saved state and continues from the correct point.

Step 3: Orchestration

Orchestration is the machinery that starts, places and restarts jobs. Read this if failures involve missing resources, inconsistent environments or jobs restarting without useful progress.

Write down what the scheduler controls and what your training code must restore. Record the container image, command, configuration, input version, secrets and output destination. Docker images are immutable, so using a recorded image gives you a fixed package to compare between runs.

Our training workflow lets you submit that container with the GPU you choose. Verify its output location before a long run. A job that completes without exporting its required files has not delivered the result you need.

Reading route:

• Provisioning GPUs with Terraform
• Automating GPU pipelines
• Moving between job schedulers (Slurm and Kubernetes)
• Setting up Kubernetes GPU nodes
• Running a small-team ML pipeline
• Accessing GPU virtual machines
• Setting up your first GPU cloud
• Allocating GPUs with a job scheduler
• Coordinating multi-node reinforcement learning

For the example, test a restart on a fresh process using the checkpoint from minute 90. That tests the handover between the scheduler, storage and training code. Merely seeing a new worker start does not test the whole chain.

Step 4: Fine-Tuning & RL

Fine-tuning continues training a model on your examples. Reinforcement learning, or RL, trains from a reward signal. Read this if the job runs but quality, resource use or training loss changes unexpectedly. Loss is the training measure of prediction error.

Choose the simplest training path that answers your experiment. Keep the evaluation set and acceptance criteria fixed while you compare changes. Record both the completed result and the resources used to obtain it.

A successful restart needs more than model weights, the learned values inside the model. PyTorch’s checkpoint guide also saves the optimiser state: the training procedure’s information for its next updates. It recommends saving other information needed to resume, such as the training epoch, or completed pass through the training data.

Reading route:

• Combining small batches for one model update
• Combining short examples in one training input
• Budgeting text-to-video training
• Sharing GPUs between model updates and generated examples
• Choosing how to train from ranked responses
• Sizing reinforcement-learning memory
• Budgeting multi-turn training
• Evaluating reduced-precision training

Use those guides for the training method you are actually running. Reducing memory use changes what fits. Changing numerical precision changes how values are represented. Changing the training goal changes what the model is taught to do. Test each against the same acceptance criteria before combining them.

Step 5: Reliability

Reliability requires proving that recovery preserves useful progress. Read this if the job restarts but its training loss changes sharply or it repeats earlier work.

Save and restore every state component your training procedure needs. Alongside the model and optimiser, save the learning-rate schedule: the rule controlling how large each update is. Save your position in the input data, so a restart uses the intended next examples. If the run makes random choices, save the random-number generator’s state so those choices can continue from the same point. Then compare a resumed short run with the uninterrupted reference.

Reading route:

• Diagnosing loss spikes after restart
• Recovering distributed training

Return to the example: the complete save at minute 90 leaves five minutes to replay after the failure at minute 95. The assumed five-minute restore makes ten minutes together. Measure both on your own workload before choosing a checkpoint schedule.

Saving more often should be evaluated alongside the time spent writing and validating each save. Keep a previous valid checkpoint until a replacement is complete and readable. Test recovery from persistent storage on a new machine, not only in the process that wrote the file.

When investigating a possible service incident, consult our service status alongside your job logs. Use the job’s timestamps to connect the two records.

Where to start, by situation

Choose the guide from the first observed failure, then define what a successful repair would show.

Scale after the small run and its recovery both meet your acceptance criteria. To budget the validated workload, compare our current billing options.