Py/Invoke

One of great things about Python is that it's relatively easy to extend it. Either by digging into Python internals manually or using Swig to do the dirty work for you, you can create brand new native modules or bindings to libraries in a few hours.

But what if you just happen to need this one function, once, for a quick hack, and there's no binding avaliable? Would you want to go through the (however small) trouble of defining, testing and compiling the extension module? I wouldn't. So, what's there to do? I really like .NET solution for this problem (Platform Invoke) - to invoke the native function you just add enough metadata to describe how to marshall types from .NET to the native function, specify the library and function name, and you're set.

For Python, there's this standard module (for Unix platforms) called dl. It is an interface to dynamic link loader, and allows loading a library, selecting a function by its name and calling it. But, there's a catch - only integers and constant strings may be supplied to the function, and the function must return an integer value. Not very practical, considering that much data-moving in C is done using pointers; which makes good ol' dl almost worthless.

So, inspired by P/Invoke, I decided to extend dl to support passing arbitrary data and mutable strings. I've added a new method call_mutable, which is largely copy-pasted call with a few tweaks to allow mutable (call by reference) parameters. Here it is.

Extension to the dl module

The call_mutable method is an extended version of standard call method available in the dl module. The method allows integers, strings and None as arguments to the native function. Integers are passed by value, and None is passed as NULL pointer. For each string argument, a separate data buffer is allocated, initialized with string data, and passed to the function.

The method returns a tuple containing return value of the native function, and strings holding the data found in string buffers upon native function exit.

Example:

>>> import dl
>>> m = dl.open('libc.so.6')
>>> m.call_mutable('time')
(1123796886,)
>>> m.call_mutable('read', 0, '\0' * 15, 15)
Hello World
(12, 'Hello World\n\x00\x00\x00')

Convenience wrapper: native

Since dealing with raw data buffers understood by call_mutable is cumbersome, I've created the convenience wrapper which combines the functionality of struct and dl modules. The module provides just one function, native, which loads the shared library, performs marshalling and unmarshalling of arguments and calls the native C function.

The data marshalling is done according to format string similar to one used by struct module. Its format is:

        '<type>:<type>:..:<type>'

where 'type' is one of:

• '' - specifies an integer to be passed by value
• 's' - specifies data buffer with size identical to the correspondenting string argument + one byte for the NUL-terminator
• any other - used exactly as in struct module

Upon return from the external C function, the native function unmarshalls the arguments and returns the tuple containing return value (integer) as the first element of the tuple, and unmarshalled values for mutable arguments (that is, all arguments except integers passed by value).

Note that this module caches the dl objects used, so the external library won't be reopened several times on multiple function invocation. To close all open libraries, you can use close_all function provided by the module.

Example:

>>> import native
>>> native.native('libc.so.6', 'time', 'i', (1,))
(1123797259, 1123797259)
>>> native.native('libc.so.6', 'read', ':s:', (0, '\0' * 15, 15))
Hello World
(12, 'Hello World\n\x00\x00\x00')

The code

I've packed my version od dl module along with setup.py script and wrapper native module into a tarball which you can get from my software repository. The tarball also contains a patch against dlmodule.c from Python CVS.
So, get it, play with it and feel free to flame me about it ;-)