Python UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d (Windows) or 'utf-8' codec can't decode byte 0xff in position 0. How do I read the file correctly?

Cause: open() in text mode turns bytes into str using an encoding. If you don't pass one, Python uses the locale's preferred encoding: often cp1252 ('charmap') on Windows, UTF-8 on most Linux and macOS systems. The error means the file's actual bytes don't fit the encoding used. Typical cases: a UTF-8 file read on Windows without encoding=; a file saved by Excel or an old Windows tool in cp1252/latin-1 read as UTF-8; a UTF-16 file (often starting with bytes 0xff 0xfe); or a binary file (image, zip, pickle) opened in text mode. Fix (best first): 1. Always pass the encoding explicitly: open(path, encoding='utf-8'). The same applies to pandas.read_csv(..., encoding=...), Path.read_text(encoding='utf-8') and subprocess text output. 2. If the file has a UTF-8 BOM (common from Excel's 'CSV UTF-8'), use encoding='utf-8-sig'. 3. If it isn't UTF-8, use its real encoding (for example 'cp1252', 'latin-1' or 'utf-16'). If you don't know it, check where the file came from, or detect it with a library such as charset-normalizer, then confirm the text looks right. 4. If it's not a text file, open it in binary mode ('rb') and handle bytes. 5. Only if losing a few characters is acceptable: errors='replace' (or 'backslashreplace') avoids the crash but changes the data, so don't use it for data you write back. 6. For whole programs on Windows, enable UTF-8 mode (PYTHONUTF8=1 or python -X utf8). PEP 686 makes UTF-8 mode the default in Python 3.15. Verify: the read succeeds without errors= tricks and non-ASCII characters (accents, quotes, emoji) look correct; write files with an explicit encoding too, so the next reader doesn't hit the same error. Sources: https://docs.python.org/3/howto/unicode.html https://docs.python.org/3/library/functions.html https://peps.python.org/pep-0686/