The bill grows when the GPU sits idle while the CPU still decodes the next batch. We want to run DINOv3, SAM or CLIP over millions of images, and nobody needs the results within seconds. What will a million images cost, and which GPU do we need?
Batch vision is a pipeline problem, not a GPU problem
Batch inference means running a model over a fixed set of inputs as fast as possible, with no user waiting for each answer. What you pay for is throughput (the number of images the job finishes per second). DINOv3 versions range from 21 million to 0.8 billion parameters, so their weights take a few gigabytes at most. The slow part is often the host, the CPU side of the machine, which reads, decodes and resizes every image first.
The three model families load the pipeline differently. In model names such as ViT-L/14, ViT stands for vision transformer, L for large and 14 for the patch size in pixels:
| Model | What it returns | Input size from its paper | What it stresses |
|---|---|---|---|
| CLIP ViT-L/14 | One embedding (a vector of numbers) per image | 224 or 336 pixels square | Decoding and loading many small images |
| DINOv3 ViT-L/16 | Features for every 16-pixel patch, or one vector | Trained at 256, works up to 768 and beyond; the example uses 256 | Compute and memory, both growing with resolution |
| SAM (huge vision transformer) | Masks, one per prompt (a point or box on the image) | 1,024 pixels square | Memory, plus mask work per prompt |
SAM has two parts. A large network turns each image into features, numbers that describe each small area, once. A light one then draws a mask, the outline of one object, for each prompt.
The running example is hypothetical: 10 million catalogue images through DINOv3 ViT-L/16 at 256 by 256 pixels on one H100 GPU. Its rates are assumed to show the method. They favour a well-tuned GPU and do not show your hardware.
Measuring whether you are GPU-bound at all
A job is GPU-bound when the GPU never waits for input. It is input-bound when the GPU finishes a batch and then waits for the host. Watch the GPU while the real job runs:
nvidia-smi dmon -s ut
The u group prints utilisation. The t group prints how many megabytes per second move between host and GPU over PCIe, the link between them. Utilisation is the share of time the GPU was running any work at all. So 100% means it never sat idle, not that it was fully loaded. A value that keeps dropping towards zero between batches means the GPU is waiting.
On our serverless runs, the metrics endpoint returns the same signals per run, every 15 seconds by default. Consistently low GPU utilisation during the heavy phase points at data loading or CPU pre-processing.
The decisive test takes two runs. First, feed the model random tensors (arrays of numbers shaped like images) that already sit on the GPU. Then run the real pipeline with decoding. In the example, the first run reaches 2,400 images per second and the real one 900. The GPU therefore works at 900 / 2,400 = 37.5% of its own speed. Our guide to fixing low GPU utilisation covers more causes.
Sizing batch against memory at your resolution
The batch size is the number of images the GPU processes in one pass. Larger batches use the GPU better, until throughput stops rising or memory runs out. Find that point at the resolution you will actually use, because a batch that fits at 256 pixels may not fit at 512. This sweep doubles the batch, measures images per second on random input and records peak memory:
import time
import torch
def sweep(model, res, batches=(16, 32, 64, 128, 256, 512)):
model = model.cuda().eval()
for bs in batches:
x = torch.randn(bs, 3, res, res, device="cuda")
torch.cuda.reset_peak_memory_stats()
try:
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
for _ in range(3):
model(x) # warm-up
torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(20):
model(x)
torch.cuda.synchronize()
except torch.cuda.OutOfMemoryError:
print(f"batch {bs}: out of memory")
break
per_s = 20 * bs / (time.perf_counter() - start)
peak_gb = torch.cuda.max_memory_allocated() / 1e9
print(f"batch {bs}: {per_s:,.0f} images/s, peak {peak_gb:.1f} GB")
Pass your loaded DINOv3, CLIP or SAM image encoder as model. When doubling the batch adds almost no images per second, keep the smaller of the two. In the example, that is batch 256 at 2,400 images per second.
SAM needs a separate check. Its masks grow with the number of prompts. Automatic mask generation in its paper prompts a 32 by 32 grid of points: 1,024 prompts per image. An H100 has 80 GB of GPU memory. Our right-sizing guide compares GPUs by job.
Feeding the device fast enough to keep it busy
The host has to decode, resize and copy images as fast as the GPU consumes them. Our H100 machine profile lists 20 vCPUs, virtual CPU cores. Keeping the example GPU at 2,400 images per second then means 2,400 / 20 = 120 images per virtual CPU core each second. The real pipeline managed 900 / 20 = 45, well short. Measure what one core decodes on your own files. Past 20 workers there are no more cores, so more workers cannot close the gap.
PyTorch's DataLoader exposes the main controls. Setting num_workers above zero runs loading in separate processes. The pin_memory=True setting keeps batches in memory the GPU can copy from directly. That makes transfers faster. The prefetch_factor setting, 2 by default, sets how many batches each worker prepares in advance:
from torch.utils.data import DataLoader
loader = DataLoader(
dataset,
batch_size=256,
num_workers=18, # leave a core or two for the main process
pin_memory=True,
prefetch_factor=4, # 4 x 18 = 72 batches prepared ahead
)
When the cores fall short, move decoding to the GPU. NVIDIA's DALI library takes on the CPU bottleneck by running data preprocessing on the GPU, including image decoding. The H100 also has seven hardware JPEG decoders of its own. Or resize the whole dataset once to the model's input size, so every later run skips that step.
Storage is the last link. Our storage bucket keeps your files after a VM is gone, while VM disks are wiped on termination. In a Docker run it appears as the folder /mnt/s3, but each file you read there is fetched over the network. With one file per image, that is one fetch per image. Pack the images into shards instead: large archive files that each hold thousands of images.
What input resolution costs you
Resolution sets the compute per image, so a resize step in preprocessing decides the cost of the whole job. A vision transformer cuts the image into square patches and treats each one as a token, an input unit like a word in a sentence. CLIP ViT-L/14 at 224 pixels gives (224 / 14)² = 256 patches, and at 336 pixels (336 / 14)² = 576. That is 2.25 times the tokens. Attention, the step that compares every token with every other, then grows faster still.
DINOv3's paper prints the cost directly. Its ViT-L model needs 163 GFLOPs (billion floating-point operations) per image at 256 pixels and 721 at 512. Four times the pixels cost 721 / 163 = 4.42 times the compute. In the example, the GPU-bound rate falls from 2,400 to 2,400 / 4.42 ≈ 543 images per second.
Higher resolution can be worth it. CLIP's 336-pixel model scored 85.4% instead of 83.9% on an ImageNet classification test. DINOv3 reports better per-pixel results, such as segmentation (labelling which object each pixel belongs to), at larger image sizes. Test your task at two resolutions before you pay for the higher one. SAM always runs at 1,024 pixels, so its lever is the number of prompts. For image-and-text models, see running a vision-language model.
Where lower precision needs validating
Lower precision stores each number in fewer bits, which saves memory and time. PyTorch's autocast runs suitable operations in float16 or bfloat16, two 16-bit formats, and keeps others in float32. Its documentation notes that the matrix operations at the heart of these models run much faster in 16 bits.
For CLIP or DINOv3 embeddings, embed a sample of images you did not tune on, once in 32-bit and once in 16-bit. Then compare the most similar images each version finds for the same query. For SAM, compare the 16-bit masks with the 32-bit masks on your images, especially small objects and thin edges. Keep 16 bits only where your task's own quality measure holds.
Cost per million images with assumptions stated
Cost per million images follows from one measured number: the images per second on the real pipeline.
GPU-hours per million images = 1,000,000 / images per second / 3,600
cost per million images = GPU-hours per million × your hourly GPU rate
| Example run (hypothetical) | Images per second | GPU-hours per million | GPU-hours for 10 million |
|---|---|---|---|
| Input-bound, CPU decoding | 900 | 0.309 | 3.09 |
| GPU-bound at 256 pixels | 2,400 | 0.116 | 1.16 |
| GPU-bound at 512 pixels | 543 | 0.512 | 5.12 |
The input-bound run costs 2,400 / 900 = 2.67 times the GPU-bound one on the same GPU. We recommend fixing the host before you rent a bigger GPU: in the example, it takes the job from 3.09 to 1.16 GPU-hours. The 512-pixel run costs 4.4 times the 256-pixel run, before any accuracy gain counts. These are calculated examples with assumed rates, leaving out storage and failed shards. Note the resolution, batch size and precision next to every rate you measure, because rates do not transfer.
We are Lyceum, a sovereign European AI inference provider for open-source models. Our GPUs run in European data centres. On our inference API, prompts and outputs are processed, not stored, and never used for training. Our GPU virtual machines come with root access and are billed per second, with no subscription. Spot machines cost about 50% less but can be reclaimed at any time. That suits a batch job that writes its results shard by shard and can restart. The compute page lists the current hourly rate for each GPU, so multiply it by your GPU-hours.
Frequently asked questions
Can I run a batch vision job without managing a VM?
Yes. A serverless run starts your Docker container once on the GPU you choose, with no machine to manage, then stops. We list them under serverless training, but batch jobs are one of their uses. Your storage bucket is mounted inside the container.