PyTorch 'torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate ...' during training or inference: how to reduce GPU memory

Cause: The GPU does not have enough free memory for the next allocation. PyTorch uses a caching allocator, so nvidia-smi shows cached-but-unused memory as used. The real causes are usually a batch or sequence length too large for the model, activations kept for backward, tensors kept alive across iterations (for example appending loss tensors that still hold the graph), evaluation run with gradients on, other processes on the same GPU, or fragmentation from varying sizes. Fix (best first): 1. Lower the batch size or input size; if you need a large effective batch, use gradient accumulation. 2. Run evaluation and inference under torch.no_grad() or torch.inference_mode() so no graph is stored. 3. Do not keep graph-attached tensors: store loss.item() or tensor.detach() for logging, not the tensor itself. 4. Use lower precision (mixed precision with autocast, or half/bfloat16 weights where suitable) and gradient checkpointing to trade compute for activation memory. 5. Diagnose with torch.cuda.memory_allocated(), torch.cuda.max_memory_allocated(), torch.cuda.memory_reserved() and torch.cuda.memory_summary(). If much memory is reserved but not allocated, you are probably fragmented. 6. For fragmentation, set the allocator environment variable PYTORCH_ALLOC_CONF (alias PYTORCH_CUDA_ALLOC_CONF), for example expandable_segments:True (experimental, helps when sizes vary between iterations) or max_split_size_mb as a last resort. 7. torch.cuda.empty_cache() only returns unused cached blocks to the driver, mostly useful so other processes can use the GPU; it does not free memory held by live tensors. Delete references (del tensor) first. 8. Check nvidia-smi for other processes using the card, and consider a smaller model, offloading, or multiple GPUs. Verify: a full training step or inference pass completes, and max_memory_allocated() stays below the card's capacity with headroom for the largest batch. Sources: https://docs.pytorch.org/docs/2.15/notes/cuda.html https://docs.pytorch.org/docs/2.15/torch.html