Parallel Python

Somehow I missed this when searching for something like it the previous time, but Parallel Python (now found via Bruce Eckel’s blog) is exactly what I’ve been looking for.  A simple process pool that can run on multiple cores or on a cluster of machines!

DeVIDE has recently acquired the ability to run in black-box (gui-less) mode, so that networks can also be executed via some other coordination framework, such as Nimrod (see our paper on this, mail me if you want the fulltext).  However, seeing that I’ve neatly abstracted out the interface part of DeVIDE (you can choose between wxPython, simple xml-rpc, command-line with Python driver script, even PyRo), integrating ParallelPython would be a cinch.

I can already see it before me: millions of computers running DeVIDE, all crashing in parallel!

comments powered by Disqus