PyTorch / Hugging Face 'NotImplementedError: Cannot copy out of meta tensor; no data!' when calling model.to('cuda'), after from_pretrained(device_map='auto'), or with init_empty_weights / low_cpu_mem_usage. How do I fix it?

What it is: a 'meta' tensor stores only shape and dtype, no actual values. Libraries use the meta device to build huge models without allocating memory: Accelerate's init_empty_weights(), transformers' low-memory loading, and device_map='auto' (which can leave some weights offloaded to CPU or disk and attach hooks that load them on demand). If anything then tries to copy a meta tensor to a real device (model.to('cuda'), .cuda(), .cpu(), or a forward pass touching a parameter that was never filled in), PyTorch raises 'Cannot copy out of meta tensor; no data!' because there is nothing to copy. Am I affected / common causes: - You loaded with device_map='auto' (or another device_map) and then called model.to(...) yourself. Accelerate already placed the layers; offloaded weights live on the meta device until its hooks load them. - You built the model under init_empty_weights() (or with torch.device('meta')) but never loaded a checkpoint into it. - The checkpoint doesn't contain some parameters (a mismatched architecture, a custom head, or tied weights), so they stay on meta. transformers usually warns that some weights were not initialized or not used. - Check: [n for n, p in model.named_parameters() if p.is_meta] lists parameters that still have no data; print(model.hf_device_map) shows where Accelerate put each module. Fix: 1. With device_map, don't move the model afterwards. Pass the placement you want when loading instead (device_map='cuda:0' to put everything on one GPU, or device_map='auto' with max_memory). Send only the inputs to the model's device (inputs.to(model.device)). 2. If you need the whole model on one device and it fits, load it without device_map (or with device_map set to that device) and then use it normally. 3. When building an empty model yourself, load the real weights before use: Accelerate's load_checkpoint_and_dispatch(model, checkpoint, device_map=...), or module.to_empty(device=...) followed by explicitly initializing or loading every parameter, as PyTorch's meta device docs describe. 4. If specific parameters stay on meta, fix the mismatch: use the model class that matches the checkpoint, load the missing weights, or initialize the new layers explicitly before moving the model. 5. For training, don't rely on CPU/disk offload from device_map='auto'; Accelerate's big-model inference is meant for inference, not training. Verify: the list of is_meta parameters is empty (or only offloaded ones managed by Accelerate when using device_map), and a forward pass with inputs on model.device runs. Sources: https://docs.pytorch.org/docs/stable/meta.html https://huggingface.co/docs/accelerate/concept_guides/big_model_inference https://huggingface.co/docs/accelerate/usage_guides/big_modeling