Hugging Face transformers: 'ValueError: Asking to pad but the tokenizer does not have a padding token' (Llama, Mistral, GPT-2 and other decoder-only models), and the follow-up warning 'A decoder-only architecture is being used, but right-padding was detected!'. How do I fix batching correctly?

What it is: many decoder-only language models (GPT-2, Llama, Mistral and others) ship tokenizers without a padding token, because they were trained on unpadded text. As soon as you batch several prompts with padding=True (in tokenizer calls, data collators, Trainer or pipelines), transformers needs a pad token and raises 'Asking to pad but the tokenizer does not have a padding token'. After adding one, generate() may warn that right-padding was detected: these models continue from the last position, so pad tokens at the end of a prompt hurt generation quality. Am I affected: - tokenizer.pad_token is None (print it), and you pass several inputs at once or use padding=True / padding='max_length'. - For generation, check tokenizer.padding_side; 'right' triggers the warning. Fix: 1. Simplest and most common: reuse the end-of-sequence token as the pad token: tokenizer.pad_token = tokenizer.eos_token, and pass pad_token_id=tokenizer.pad_token_id to generate() (or set model.generation_config.pad_token_id) so generation knows it too. 2. Or add a dedicated pad token: tokenizer.add_special_tokens({'pad_token': '[PAD]'}), then model.resize_token_embeddings(len(tokenizer)) so the model has an embedding for it. Use this when you fine-tune and don't want padding confused with end-of-text. 3. For batched generation, pad on the left: AutoTokenizer.from_pretrained(name, padding_side='left') (or set tokenizer.padding_side = 'left'), as the transformers LLM tutorial recommends, because the model isn't trained to continue from padding. 4. Always pass the attention_mask the tokenizer returns (model.generate(**inputs, ...)) so pad positions are ignored. 5. For training with labels, make sure padded label positions are ignored in the loss (most transformers data collators set them to -100); if you reuse EOS as pad, check that real EOS tokens you want the model to learn aren't masked out too. Verify: tokenizer.pad_token is set, batched tokenization works, generate() runs without the right-padding warning, and batched outputs match running each prompt alone. Sources: https://huggingface.co/docs/transformers/pad_truncation https://huggingface.co/docs/transformers/llm_tutorial https://huggingface.co/docs/transformers/main_classes/tokenizer