Python 'OSError: [Errno 24] Too many open files' (EMFILE) in a server, crawler or test run on Linux/macOS: how do I fix it?

Cause: each process may hold only a limited number of open file descriptors (files, sockets, pipes), set by the RLIMIT_NOFILE resource limit. When the soft limit is reached, open(), socket(), pipe() and similar calls fail with EMFILE, which Python reports as Errno 24. Typical triggers are files, sockets or HTTP responses that are opened but never closed (leaks), very high concurrency (thousands of connections or subprocess pipes), or a low default limit such as 1024 on some systems. Fix (best first): 1. Find the leak first. Use with open(...) as f and with contextlib.closing / context managers for sockets, sessions and responses; close subprocess pipes; reuse one HTTP client/session instead of creating one per request. Check the live count: ls /proc/<pid>/fd | wc -l on Linux, or lsof -p <pid> on Linux/macOS, and look at what the descriptors point to. 2. Bound concurrency with a semaphore, thread/process pool size or connection pool limit so you never hold more descriptors than you need. 3. If the workload legitimately needs more, raise the soft limit up to the hard limit. In the shell: ulimit -n shows the soft limit, ulimit -Hn the hard limit; ulimit -n 4096 raises it for that shell session. In Python: import resource; soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE); resource.setrlimit(resource.RLIMIT_NOFILE, (min(hard, 4096), hard)). 4. For services, set it in the service manager rather than a login shell, for example LimitNOFILE= in a systemd unit, or the container runtime's nofile ulimit setting. System-wide persistent limits are set by your OS configuration (for example limits.conf on many Linux distributions). 5. An unprivileged process cannot raise the hard limit; raising it needs privilege and is capped by the kernel (/proc/sys/fs/nr_open on Linux). Do not run the service as root just to bypass this. Verify: the descriptor count of the process stays flat over time under load (a leak shows a steady climb), and the error no longer appears at the previous peak. Sources: https://docs.python.org/3/library/resource.html https://man7.org/linux/man-pages/man2/getrlimit.2.html