Parallel Python

Somehow I missed this when searching for something like it the previous time, but Parallel Python (now found via Bruce Eckel’s blog) is exactly what I’ve been looking for.  A simple process pool that can run on multiple cores or on a cluster of machines!

DeVIDE has recently acquired the ability to run in black-box (gui-less) mode, so that networks can also be executed via some other coordination framework, such as Nimrod (see our paper on this, mail me if you want the fulltext).  However, seeing that I’ve neatly abstracted out the interface part of DeVIDE (you can choose between wxPython, simple xml-rpc, command-line with Python driver script, even PyRo), integrating ParallelPython would be a cinch.

I can already see it before me: millions of computers running DeVIDE, all crashing in parallel!

4 thoughts on “Parallel Python”

  1. Hi there Stéfan, thanks for the pointers! I knew about the IPython thingy (a colleague was playing with it), but not pypi. Have you actually done stuff(TM) with these toolkits?

  2. Hi Charl,

    Again, seems like we’ve struck a parallel. I’ve been using PP for distributing large lighting simulations across machines in our network. Its frightingly easy & efficient so far. Than again, I haven’t moved lots of data back & forth, so I’m really curious how that will do. I guess moving large VTK object back & forth to different machines might introduce some overhead, but thats just unavoidable. I’m curious how you plan to move these objects back & forth as well: not all VTK objects can be serialized, right?

  3. Hi there Jelle,

    Cool! I have not started implementing this yet, I’m spending all my DeVIDE time on getting it ready for its open source release. :) I have no idea yet what to do concerning data transfers.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.