Latest posts.

Simple tweeting from the command line

A while back, we needed a way to monitor the status of some long-running batch processes, in this case some environmental simulations we wanted to run over the weekend while we were away. We can access the machines via remote desktop, but that isn’t always timely or convenient, so I came up with a simple Python program which our batch scripts called to update the run status via Twitter: cmdline_tweet.

From a dedicated Twitter account, we could check on things from our smartphones (or however we like) and know immediately if some error had occurred. Since these runs can take 8 hours or more to complete (and, as always, we’re working with deadlines), there’s a big difference between knowing “now” and knowing “whenever I get around to checking.”

The client has a one-time authentication stage where it gets associated with your account via OAuth, and after that tweeting is as simple as:

~$ tweet "Hello, world!"
~$

The source includes a Makefile which helps compile tweet.py into a binary for dead-simple deployment on machines without Python installed. And copying or renaming the file will let you connect with more than one Twitter account (we used separate accounts for each simulation running in parallel).

It might even be a useful tool for tweeting your mind without leaving the shell:

~$ rm -rf .
~$ # Oh crap, didn't mean to do that.
~$ tweet 'Note to self: be MUCH more careful with "rm -rf"'

More details and source code available at the repository on Github.

Python: Forcibly redirect stdout and stderr (even from F2PY extension modules)

Powerful tools like F2PY and JCC make interfacing with legacy code a breeze. By generating clean Python wrappers around native-language structures, they allow you to seamlessly script and extend them in a much friendlier context:

>>> from legacymodule import SomeFortranAlgorithm as alg
 
>>> print "Result of running alg on odd numbers 1 to 99:"
>>> for i in xrange(1, 100, 2):
...     print "alg(%d)=%s" % (i, alg(i))
...
alg(1)=2.71828182846
alg(3)=20.0855369232
alg(5)=148.413159103
alg(7)=1096.63315843
...

It’s like there’s no Fortran at all! As you can imagine, this has the potential to be extremely useful for a myriad of scientific applications.

However, one problem with bringing legacy code into the future is dealing with its existing interface. If SomeFortranAlgorithm prints a bunch of ├╝ber-technical jibberish to stdout with each call, then you could see how that might be a bit of a pain. Unfortunately, the standard Python tricks…

>>> import sys, os
>>> from legacymodule import NoisyAlgorithm as loudmouth
>>> sys.stdout = open(os.devnull, "w")
>>> print "This is going to fall on deaf ears"
>>> answer = loudmouth(36)
COMPUTING RESULT FOR I=36
THE RESULT IS 4.31123154712E+15
>>> # Rarr!
...

…won’t always work. This is because the extension module, native as it is, operates on lower level system APIs. The trick, as revealed in this answer on Stack Overflow, is to overwrite the actual file descriptors (#1 for stdout, #2 for stderr) using os.dup2:

>>> import sys, os
>>> from legacymodule import NoisyAlgorithm as loudmouth
>>> null_fd = os.open(os.devnull, os.O_RDWR)
>>> ops.dup2(null_fd, 1)
>>> print "This is going to fall on deaf ears"
>>> answer = loudmouth(36)
>>> # Yay! Peace and quiet!
...

After liberal application of syntactic sugar, I came up with a handy, reusable context manager Silence which will redirect stdout (and stderr, too) with fewer lines than the box office at a Milli Vanilli reuinon concert:

>>> with Silence():
...     answer = loudmouth(36)
...
>>> # ... crickets ...
...

Made possible by the following context manager. You’ll notice I’ve built in support for redirecting to file (sorry, StringIO instances don’t have file descriptors). It’s also available as a recipe on ActiveState under the MIT license.

## {{{ http://code.activestate.com/recipes/577564/ (r2)
import os
 
class Silence:
    """Context manager which uses low-level file descriptors to suppress
    output to stdout/stderr, optionally redirecting to the named file(s).
 
    >>> import sys, numpy.f2py
    >>> # build a test fortran extension module with F2PY
    ...
    >>> with open('hellofortran.f', 'w') as f:
    ...     f.write('''\
    ...       integer function foo (n)
    ...           integer n
    ...           print *, "Hello from Fortran!"
    ...           print *, "n = ", n
    ...           foo = n
    ...       end
    ...       ''')
    ...
    >>> sys.argv = ['f2py', '-c', '-m', 'hellofortran', 'hellofortran.f']
    >>> with Silence():
    ...     # assuming this succeeds, since output is suppressed
    ...     numpy.f2py.main()
    ...
    >>> import hellofortran
    >>> foo = hellofortran.foo(1)
     Hello from Fortran!
     n =  1
    >>> print "Before silence"
    Before silence
    >>> with Silence(stdout='output.txt', mode='w'):
    ...     print "Hello from Python!"
    ...     bar = hellofortran.foo(2)
    ...     with Silence():
    ...         print "This will fall on deaf ears"
    ...         baz = hellofortran.foo(3)
    ...     print "Goodbye from Python!"
    ...
    ...
    >>> print "After silence"
    After silence
    >>> # ... do some other stuff ...
    ...
    >>> with Silence(stderr='output.txt', mode='a'):
    ...     # appending to existing file
    ...     print >> sys.stderr, "Hello from stderr"
    ...     print "Stdout redirected to os.devnull"
    ...
    ...
    >>> # check the redirected output
    ...
    >>> with open('output.txt', 'r') as f:
    ...     print "=== contents of 'output.txt' ==="
    ...     print f.read()
    ...     print "================================"
    ...
    === contents of 'output.txt' ===
    Hello from Python!
     Hello from Fortran!
     n =  2
    Goodbye from Python!
    Hello from stderr
 
    ================================
    >>> foo, bar, baz
    (1, 2, 3)
    >>>
 
    """
    def __init__(self, stdout=os.devnull, stderr=os.devnull, mode='w'):
        self.outfiles = stdout, stderr
        self.combine = (stdout == stderr)
        self.mode = mode
 
    def __enter__(self):
        import sys
        self.sys = sys
        # save previous stdout/stderr
        self.saved_streams = saved_streams = sys.__stdout__, sys.__stderr__
        self.fds = fds = [s.fileno() for s in saved_streams]
        self.saved_fds = map(os.dup, fds)
        # flush any pending output
        for s in saved_streams: s.flush()
 
        # open surrogate files
        if self.combine: 
            null_streams = [open(self.outfiles[0], self.mode, 0)] * 2
            if self.outfiles[0] != os.devnull:
                # disable buffering so output is merged immediately
                sys.stdout, sys.stderr = map(os.fdopen, fds, ['w']*2, [0]*2)
        else: null_streams = [open(f, self.mode, 0) for f in self.outfiles]
        self.null_fds = null_fds = [s.fileno() for s in null_streams]
        self.null_streams = null_streams
 
        # overwrite file objects and low-level file descriptors
        map(os.dup2, null_fds, fds)
 
    def __exit__(self, *args):
        sys = self.sys
        # flush any pending output
        for s in self.saved_streams: s.flush()
        # restore original streams and file descriptors
        map(os.dup2, self.saved_fds, self.fds)
        sys.stdout, sys.stderr = self.saved_streams
        # clean up
        for s in self.null_streams: s.close()
        return False
## end of http://code.activestate.com/recipes/577564/ }}}