Python: Forcibly redirect stdout and stderr (even from F2PY extension modules)

Powerful tools like F2PY and JCC make interfacing with legacy code a breeze. By generating clean Python wrappers around native-language structures, they allow you to seamlessly script and extend them in a much friendlier context:

>>> from legacymodule import SomeFortranAlgorithm as alg

>>> print "Result of running alg on odd numbers 1 to 99:"
>>> for i in xrange(1, 100, 2):
...     print "alg(%d)=%s" % (i, alg(i))
...
alg(1)=2.71828182846
alg(3)=20.0855369232
alg(5)=148.413159103
alg(7)=1096.63315843
...

It’s like there’s no Fortran at all! As you can imagine, this has the potential to be extremely useful for a myriad of scientific applications.

However, one problem with bringing legacy code into the future is dealing with its existing interface. If SomeFortranAlgorithm prints a bunch of über-technical gibberish to stdout with each call, then you could see how that might be a bit of a pain. Unfortunately, the standard Python tricks…

>>> import sys, os
>>> from legacymodule import NoisyAlgorithm as loudmouth
>>> sys.stdout = open(os.devnull, "w")
>>> print "This is going to fall on deaf ears"
>>> answer = loudmouth(36)
COMPUTING RESULT FOR I=36
THE RESULT IS 4.31123154712E+15
>>> # Rarr!
...

…won’t always work. This is because the extension module, native as it is, operates on lower level system APIs. The trick, as revealed in this answer on Stack Overflow, is to overwrite the actual file descriptors (#1 for stdout, #2 for stderr) using os.dup2:

>>> import sys, os
>>> from legacymodule import NoisyAlgorithm as loudmouth
>>> null_fd = os.open(os.devnull, os.O_RDWR)
>>> ops.dup2(null_fd, 1)
>>> print "This is going to fall on deaf ears"
>>> answer = loudmouth(36)
>>> # Yay! Peace and quiet!
...

After liberal application of syntactic sugar, I came up with a handy, reusable context manager Silence which will redirect stdout (and stderr, too) with fewer lines than the box office at a Milli Vanilli reunion concert:

>>> with Silence():
...     answer = loudmouth(36)
...
>>> # ... crickets ...
...

Made possible by the following context manager. You’ll notice I’ve built in support for redirecting to file (sorry, StringIO instances don’t have file descriptors). It’s also available as a recipe on ActiveState under the MIT license.

## {{{ http://code.activestate.com/recipes/577564/ (r2)
import os

class Silence:
    """Context manager which uses low-level file descriptors to suppress
    output to stdout/stderr, optionally redirecting to the named file(s).

    >>> import sys, numpy.f2py
    >>> # build a test fortran extension module with F2PY
    ...
    >>> with open('hellofortran.f', 'w') as f:
    ...     f.write('''\
    ...       integer function foo (n)
    ...           integer n
    ...           print *, "Hello from Fortran!"
    ...           print *, "n = ", n
    ...           foo = n
    ...       end
    ...       ''')
    ...
    >>> sys.argv = ['f2py', '-c', '-m', 'hellofortran', 'hellofortran.f']
    >>> with Silence():
    ...     # assuming this succeeds, since output is suppressed
    ...     numpy.f2py.main()
    ...
    >>> import hellofortran
    >>> foo = hellofortran.foo(1)
     Hello from Fortran!
     n =  1
    >>> print "Before silence"
    Before silence
    >>> with Silence(stdout='output.txt', mode='w'):
    ...     print "Hello from Python!"
    ...     bar = hellofortran.foo(2)
    ...     with Silence():
    ...         print "This will fall on deaf ears"
    ...         baz = hellofortran.foo(3)
    ...     print "Goodbye from Python!"
    ...
    ...
    >>> print "After silence"
    After silence
    >>> # ... do some other stuff ...
    ...
    >>> with Silence(stderr='output.txt', mode='a'):
    ...     # appending to existing file
    ...     print >> sys.stderr, "Hello from stderr"
    ...     print "Stdout redirected to os.devnull"
    ...
    ...
    >>> # check the redirected output
    ...
    >>> with open('output.txt', 'r') as f:
    ...     print "=== contents of 'output.txt' ==="
    ...     print f.read()
    ...     print "================================"
    ...
    === contents of 'output.txt' ===
    Hello from Python!
     Hello from Fortran!
     n =  2
    Goodbye from Python!
    Hello from stderr

    ================================
    >>> foo, bar, baz
    (1, 2, 3)
    >>>

    """
    def __init__(self, stdout=os.devnull, stderr=os.devnull, mode='w'):
        self.outfiles = stdout, stderr
        self.combine = (stdout == stderr)
        self.mode = mode

    def __enter__(self):
        import sys
        self.sys = sys
        # save previous stdout/stderr
        self.saved_streams = saved_streams = sys.__stdout__, sys.__stderr__
        self.fds = fds = [s.fileno() for s in saved_streams]
        self.saved_fds = map(os.dup, fds)
        # flush any pending output
        for s in saved_streams: s.flush()

        # open surrogate files
        if self.combine:
            null_streams = [open(self.outfiles[0], self.mode, 0)] * 2
            if self.outfiles[0] != os.devnull:
                # disable buffering so output is merged immediately
                sys.stdout, sys.stderr = map(os.fdopen, fds, ['w']*2, [0]*2)
        else: null_streams = [open(f, self.mode, 0) for f in self.outfiles]
        self.null_fds = null_fds = [s.fileno() for s in null_streams]
        self.null_streams = null_streams

        # overwrite file objects and low-level file descriptors
        map(os.dup2, null_fds, fds)

    def __exit__(self, *args):
        sys = self.sys
        # flush any pending output
        for s in self.saved_streams: s.flush()
        # restore original streams and file descriptors
        map(os.dup2, self.saved_fds, self.fds)
        sys.stdout, sys.stderr = self.saved_streams
        # clean up
        for s in self.null_streams: s.close()
        return False
## end of http://code.activestate.com/recipes/577564/ }}}

Other Posts