alt.gil.die.die.die

Anyone who knows me even moderately well knows that I adore Python. However, once again it seems that nothing’s perfect. Python uses a global interpreter lock, or GIL, to ensure that multiple threads don’t mess up interpreter state too badly. This means that only the thread that holds the GIL can run the interpreter at any specific moment.

In investigating possibilities for the next generation of DeVIDE, I was considering threadifying the whole deal in order to enable the user to steer processing pipelines whilst they’re processing and in order to detach the GUI completely from the processing backend. However, after constructing this example, that seems to be impossible with the GIL currently used by CPython. Running the example, you’ll note that the VTK pipelines execute sequentially instead of in parallel. Unless I’ve misunderstood something somewhere, I have the GIL to blame for this.

ARGH!

Now I’ll have to look at far more hare-brained schemes to detach everything from everything. I was hoping that the threading thing would work out, as I need to chuck hundreds of megabytes of data around at interactive speeds.

Oh yeah, a pre-emptive little something to the asynchronous-programming-it’s-all-a-state-machine-fanboys: I was implementing state-machines on hardware when you were still dirtying your nappies. Now bite me.

4 thoughts on “alt.gil.die.die.die”

  1. Thanks for the suggestion Rudolph. Stackless looks really cool, but it still doesn’t seem to solve my problem. Microthreads are useful for a different type of problem, but they also seem to switch at bytecode (or even python code) level. In my case, a single Python call is invoking long-running wrapped code. This really needs to be in a separate OS thread without a global lock preventing the rest of Python from continuing with its business.

  2. Just another note: if wrapped code calls the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS around blocking code that doesn’t make any calls into the Python API, the GIL is released.

    I either have to implement this for VTK, ITK and all other wrapped code that I want to use, or I have to figure out some other non-threading solution.

Leave a Reply

Your email address will not be published. Required fields are marked *