Increase ulimit number of open files
Why increase ulimit -n
maximum number of open files?
Making Matplotlib/Seaborn plots in parallel executing threads in Python is one area where you may need to increase ulimit -n setting
.
That is, increase the limit for the maximum number of open files.
The default maximum number of open file limits are a bit low
Linux (Ubuntu) | Mac OS X (Yosemite/El Capitan) |
---|---|
1024 | 256 |
This can lead to errors from Python with Matplotlib like:
OSError: [Errno 24] Too many open files: ‘/usr/share/fonts/truetype/msttcorefonts/arial.ttf’
Fix
Platform independent ulimit -n increase “ulimit_nofile.py”
In general I avoid one-off or sudo-requiring system parameter setting techniques, and would defer to using platform-independent Python since virtually all of my programs at least incidentally use Python.
This does not use sudo
.
It instead makes temporary setting changes where feasible or at most user-profile specific system changes.
This change will be only for this instance of Python; once Python terminates even the same terminal session will have the original ulimit
again.
#!/usr/bin/env python
try:
import resource as res
except ImportError: #Windows
res = None
def raise_nofile(nofile_atleast=4096):
"""
sets nofile soft limit to at least 4096, useful for running matlplotlib/seaborn on
parallel executing plot generators vs. Ubuntu default ulimit -n 1024 or OS X El Captian 256
temporary setting extinguishing with Python session.
"""
if res is None:
return (None,)*2
# %% (0) what is current ulimit -n setting?
soft,ohard = res.getrlimit(res.RLIMIT_NOFILE)
hard = ohard
# %% (1) increase limit (soft and even hard) if needed
if soft<nofile_atleast:
soft = nofile_atleast
if hard<soft:
hard = soft
print('setting soft & hard ulimit -n {} {}'.format(soft,hard))
try:
res.setrlimit(res.RLIMIT_NOFILE,(soft,hard))
except (ValueError,res.error):
try:
hard = soft
print('trouble with max limit, retrying with soft,hard {},{}'.format(soft,hard))
res.setrlimit(res.RLIMIT_NOFILE,(soft,hard))
except Exception:
print('failed to set ulimit, giving up')
soft,hard = res.getrlimit(res.RLIMIT_NOFILE)
return soft,hard
if __name__ == '__main__':
from argparse import ArgumentParser
p = ArgumentParser()
p.add_argument('-n','--nofile',help='max number of open files',type=int,default=4096)
p = p.parse_args()
soft,hard = raise_nofile(p.nofile)
print('ulimit -n soft,hard: {},{}'.format(soft,hard))