pandas 'ValueError: cannot reindex on an axis with duplicate labels' (also from concat, assign, or setting a column): how do I find and fix duplicate index labels?

Cause: pandas needs a unique index to realign data to a new set of labels. If the row index (or column index, for axis=1) has repeated labels, operations that align on labels, such as reindex(), assigning a Series to a column, or concat with axis=1 and join, cannot decide which duplicate should match and raise this ValueError. Duplicates usually come from concatenating frames without ignore_index, reading data with a non-unique key column set as index, or merging on a non-unique key. Fix (best first): 1. Check: df.index.is_unique and df.columns.is_unique, then df.index[df.index.duplicated()].unique() to see the offending labels. 2. If the index is just row numbering, discard it: df = df.reset_index(drop=True), or build frames with pd.concat([...], ignore_index=True). 3. If duplicates are accidental repeats of the same record, keep one: df = df.loc[~df.index.duplicated(keep="first"), :] (keep="last" or keep=False are the other choices in the docs). 4. If the repeats are real data, combine them rather than drop them: df.groupby(level=0).mean() (or sum, first, etc.). 5. For duplicated column names, rename or deduplicate them before the operation, using the same duplicated() approach on df.columns. 6. To catch it earlier, call df = df.set_flags(allows_duplicate_labels=False); pandas then raises DuplicateLabelError at the step that introduces duplicates. Note that not every method propagates this flag. 7. If you are aligning to another frame, check the other frame's index too; the error can come from either side. Verify: df.index.is_unique returns True and the reindex, assignment or concat completes without error, with the row count you expect. Sources: https://pandas.pydata.org/docs/user_guide/duplicates.html https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reindex.html