Increase ulimit number of open files

21 May, 2016

Why increase “ulimit -n” maximum number of open files?

Making Matplotlib/Seaborn plots in parallel executing threads in Python is one area where you may need to increase ulimit for files. That is, increase the limit for the maximum number of open files. The default maximum number of open file limits are like 1024 for Linux and 2560 for macOS.

This can lead to problems with Matplotlib like:

Too many open files: ‘/usr/share/fonts/truetype/msttcorefonts/arial.ttf’

Fix

Platform independent ulimit -n increase “ulimit_nofile.py”

In general I avoid one-off or sudo-requiring system parameter setting techniques, and would defer to using platform-independent Python since virtually all of my programs at least incidentally use Python. This does not use sudo. It instead makes temporary setting changes where feasible or at most user-profile specific system changes. This change will be only for this instance of Python; once Python terminates even the same terminal session will have the original ulimit again.

#!/usr/bin/env python
try:
    import resource as res
except ImportError: #Windows
    res = None

def raise_nofile(nofile_atleast=4096):
    """
    sets nofile soft limit to at least 4096, useful for running matlplotlib/seaborn on
    parallel executing plot generators vs. Ubuntu default ulimit -n 1024 or OS X El Captian 256
    temporary setting extinguishing with Python session.
    """
    if res is None:
        return (None,)*2
# %% (0) what is current ulimit -n setting?
    soft,ohard = res.getrlimit(res.RLIMIT_NOFILE)
    hard = ohard
# %% (1) increase limit (soft and even hard) if needed
    if soft<nofile_atleast:
        soft = nofile_atleast

        if hard<soft:
            hard = soft

        print('setting soft & hard ulimit -n {} {}'.format(soft,hard))
        try:
            res.setrlimit(res.RLIMIT_NOFILE,(soft,hard))
        except (ValueError,res.error):
            try:
               hard = soft
               print('trouble with max limit, retrying with soft,hard {},{}'.format(soft,hard))
               res.setrlimit(res.RLIMIT_NOFILE,(soft,hard))
            except Exception:
               print('failed to set ulimit, giving up')
               soft,hard = res.getrlimit(res.RLIMIT_NOFILE)

    return soft,hard


if __name__ == '__main__':
    from argparse import ArgumentParser
    p = ArgumentParser()
    p.add_argument('-n','--nofile',help='max number of open files',type=int,default=4096)
    p = p.parse_args()

    soft,hard = raise_nofile(p.nofile)
    print('ulimit -n soft,hard: {},{}'.format(soft,hard))