pandas MemoryError or the process gets killed when calling read_csv / loading a large dataset ('Unable to allocate X GiB for an array'). How do I load and process data that doesn't fit in memory?

Cause: pandas loads the whole dataset into memory as NumPy arrays, and memory use is often several times the file size: text columns stored as Python objects, wide default types (int64, float64), and intermediate copies during operations. When the machine or container runs out, you get MemoryError, 'Unable to allocate', or the OS kills the process (exit code 137 in containers). Fix (pandas' own scaling guide recommends these, roughly in order): 1. Load only what you need: usecols=[...] in read_csv, and filter rows as early as possible. 2. Use smaller, efficient types: pass dtype= (e.g. int32, float32), convert repetitive text columns to 'category', and use pd.to_numeric(..., downcast=...). Check usage with df.memory_usage(deep=True). 3. Process in chunks: read_csv(..., chunksize=100_000) returns an iterator; aggregate or write each chunk instead of concatenating everything. 4. Store data in efficient formats: convert large CSVs once to Parquet (df.to_parquet) and read back only needed columns; Parquet is typically much smaller and faster than CSV. 5. For data far bigger than memory, use a library built for out-of-core or parallel work (the guide mentions Dask; others like DuckDB or Polars can also query Parquet/CSV without loading it all). 6. In containers, check the memory limit: the process may have far less memory than the host shows. Verify: df.memory_usage(deep=True).sum() is comfortably below available memory, or the chunked/out-of-core job completes without being killed. Sources: https://pandas.pydata.org/docs/user_guide/scale.html https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html