Lyceum Magazine cover: the title GPU cloud per-second billing: stop paying for idle compute, beside line art of one short purple-topped block standing at the front of a long, low outlined slab, suggesting a short job inside a much longer paid block.

How short GPU jobs turn into idle cost

Idle cost is GPU time you pay for while no job is computing. It comes from two places: the billing unit rounds each job up, and machines keep running after the job ends.

The question teams bring to us is direct. Should we pay for GPUs by the second, by the hour or by reservation, when we don't need them all the time?

Short jobs are the normal case. Hu and colleagues studied six months of 2020 traces from four GPU clusters at SenseTime, covering more than 3 million jobs. The median GPU job ran for 206 seconds. Roughly three-quarters of GPU jobs lasted under 1,000 seconds, about 17 minutes, mostly model evaluation and debugging. A study of Microsoft's Philly GPU cluster found the same spread: jobs run from minutes to days or even weeks. Large language model work follows the pattern. A 2024 study of an LLM development datacenter at Shanghai AI Laboratory compared its average job length with three earlier GPU cluster traces. Its jobs were 2.7 to 12.8 times shorter, depending on the trace compared. The authors trace this to many short tasks such as evaluation.

We are Lyceum, a sovereign European AI inference provider for open-source models: built and run in Europe, and our GPUs run in European data centres. We also rent out the GPUs themselves. Here we count what each billing rule charges for the same work. We count in GPU-hours, one GPU running for one hour, so the numbers hold at any price.

Per-second, hourly and reserved GPU billing compared

The billing unit is the smallest slice of time a provider charges for. A unit that has started is charged in full. Reserved capacity works differently: you book GPUs for an agreed term and pay for the term.

Billing ruleCharged for one 14-minute jobIdle time charged for that job
Per second840 seconds (14 minutes)None, if the machine stops with the job
Per started hour60 minutes46 minutes: the unused rest of the started hour
Reserved capacityNothing extra: the term is already paidEvery hour of the term that no job uses

Under every rule, a machine that keeps running after its job is also charged for each extra second or hour it runs.

Where our billing differs

Our GPU virtual machines (VMs) are rented machines with root access that you start and stop. They are billed per second, with no subscription. STACKIT lists its GPU servers with hourly billing, "billed per billing unit or part thereof" (STACKIT GPU price page, read 21 October 2026). Under that rule, the 14-minute job above is charged for a full hour.

Worked example: one team's 14-minute jobs, billed three ways

Take a hypothetical team of 10 engineers. Each runs five fine-tuning jobs a day, and each job takes 14 minutes on one GPU. The GPU time the jobs use is the same under every billing rule:

jobs per day    = 10 engineers × 5 jobs           = 50 jobs
GPU time used   = 50 jobs × 840 seconds            = 42,000 seconds
                = 42,000 seconds ÷ 3,600           = 11.67 GPU-hours

The billed time is what changes:

Billing rule (hypothetical team)GPU-hours billed per dayBilled ÷ used
Per second, machine stopped after each job11.671.0×
Per started hour, machine stopped after each job50 jobs × 1 hour = 504.3×
Per second, each engineer's VM running an 8-hour workday10 VMs × 8 hours = 806.9×

Hourly rounding adds 50 − 11.67 = 38.33 GPU-hours a day. Each job pays 60 minutes for 14 minutes of work, and 60 ÷ 14 = 4.3. Leaving the VMs up all day costs more: 80 GPU-hours billed for 11.67 used, 6.9 times. Per-second billing cannot save that idle VM, because a running VM is billed for every second.

Calculated example for a hypothetical team of 10 engineers who each run five 14-minute GPU jobs a day, 11.67 GPU-hours of work. Billed per second with the machine stopped after each job: 11.67 GPU-hours. Billed per started hour: 50 GPU-hours. Billed per second but with each engineer's VM left running for an 8-hour workday: 80 GPU-hours, highlighted. A dashed line marks the 11.67 GPU-hours the jobs used.
In our calculated example, the team's VMs left running for an 8-hour day bill 80 GPU-hours, more than the 50 that hourly rounding charges for the same 11.67 hours of jobs.

Shorter jobs make the rounding worse. The 206-second median job from the SenseTime traces would pay for 3,600 seconds under hourly billing: 3,600 ÷ 206 = 17.5 times its runtime.

The example favours per-second billing in one way: it assumes each job starts on a machine that is already set up, with its packages installed. Billing starts once a new VM is running, so the 1 to 3 minutes of provisioning are free, but installing packages afterwards is billed. Suppose each job needed 3 billed minutes of setup. The per-second row would then grow to 14.17 GPU-hours, still well under the 50 GPU-hours of hourly billing:

billed per day = 50 jobs × (14 + 3) minutes = 850 minutes = 14.17 GPU-hours

How to stop paying for idle GPUs

Our VMs persist until you terminate them, and they bill for their entire uptime, whether anything runs on them or not. Terminating a VM ends both the VM and its billing. It also wipes the VM's local disk, so copy results to your storage bucket first. The bucket is the S3-compatible storage of your account, and it persists across VMs.

lyceum vm list                    # find the VM id
lyceum vm terminate <vm_id> -f    # end the VM and its billing

A forgotten VM is the expensive case. The example script below runs on your own computer. Every minute it runs nvidia-smi, NVIDIA's command-line tool for reading GPU status, on the VM over SSH. It reads GPU utilisation: the percent of the last sample period in which a kernel was running on the GPU. A kernel is a function that runs on the GPU. On a VM with several GPUs, nvidia-smi prints one value per GPU. The sort and tail commands at the end of that line keep the highest. The VM therefore counts as idle only when every GPU is idle. After 30 idle minutes in a row, the script terminates the VM.

#!/usr/bin/env bash
# Example watchdog: run it on your own computer, not on the VM.
VM_ID="<vm_id>"
VM_IP="<ip from lyceum vm status>"
idle=0
while [ "$idle" -lt 30 ]; do
  util=$(ssh lyceum@"$VM_IP" nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits | sort -n | tail -1)
  if [ "$util" -eq 0 ]; then idle=$((idle + 1)); else idle=0; fi
  sleep 60
done
lyceum vm terminate "$VM_ID" -f   # the disk is wiped: save results first

Three of our products stop billing without a script:

  • Serverless training: a run executes your container on the GPU you choose. It is billed per second of wall-clock time, and the machine is released when the code finishes.
  • Spot VMs: the same GPUs at about 50% less, which we can reclaim at any time. Save checkpoints, so a reclaimed job can resume. A checkpoint is a saved copy of the model and of the optimizer state, the values the optimizer updates as the model trains. Our guide to spot GPUs for training covers the setup.
  • Dedicated endpoints: your model on GPUs reserved for it. Each running copy of the model is called a replica, and you are billed for the time the replicas run. A deployment with no requests for 1 hour is paused, which stops billing until the next request arrives. Scaleway's FAQ says its dedicated deployments are billed while running and "can therefore not be paused" (read 21 October 2026).
What ends the billing after a job, for three of our products. GPU virtual machine: billing goes on after the job ends and stops only when you terminate the VM, highlighted. Serverless training run: the machine is released when the code finishes, and billing stops. Dedicated endpoint: after 1 hour with no requests the deployment is paused, and billing stops until the next request.
Of our three products, only a GPU virtual machine keeps billing after its job ends, until you terminate it.

To submit a training job as a run instead of on a VM:

lyceum docker run your-team/train -m gpu.h100 -c "python train.py"

Our article on scale-to-zero GPU inference compares idle costs for serving.

How to see what each GPU job cost

Every billable action creates an entry in your organisation's ledger, the running record of charges. Charges are pulled against your balance in real time, so you can watch spend during a long job and stop it. The CLI lists the entries:

lyceum org transactions list

For runs, the metrics endpoint returns GPU utilisation for each execution. The metrics come from DCGM, NVIDIA's monitoring library for data-centre GPUs. The endpoint reports a value every 15 seconds by default, and the step parameter sets another interval, 5 seconds in this example:

curl "https://api.lyceum.technology/api/v2/external/execution/$EXEC/metrics?step=5s" \
  -H "Authorization: Bearer $LYCEUM_API_KEY" | jq

A run with low gpuUtilizationPercent during its main phase paid for a waiting GPU. The usual cause is data loading or CPU pre-processing. With PyTorch's default setting of num_workers=0, data loading runs in the main process, and training waits for it. Setting num_workers above 0 loads data in parallel worker processes. Our guide to low GPU utilization covers the fixes.

When reserved GPU capacity pays off

Reserved GPUs and clusters are capacity held for you, priced per contract. Our compute page names long training runs, steady inference and multi-node jobs as their use. A multi-node job spans several machines. A reservation bills for the term whether jobs fill it or not.

In the example, the team uses 11.67 GPU-hours a day. One reserved GPU offers 24 GPU-hours a day, so the jobs would fill 11.67 ÷ 24 = 49% of it. Half of that GPU's term would be idle.

We recommend per-second VMs and serverless training for experiments and CI tests, and a reservation only once your ledger shows weeks of steady use. Our comparison of reserved and on-demand GPUs covers the capacity side. To start, launch a VM or submit a run from our compute page.

Sources

  1. arxiv.org
  2. arxiv.org
  3. arxiv.org
  4. docs.nvidia.com
  5. docs.nvidia.com
  6. docs.pytorch.org
  7. docs.nvidia.com
  8. docs.pytorch.org