Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Multicore Python: A tough, worthy, and reachable goal

Serdar Yegulalp | June 30, 2016
Python has been held back by its inability to natively use multiple CPU cores. Now Pythonistas are aiming to find a solution

Finally, the GIL itself was reworked somewhat in Python 3, with a better thread-switching handler. But all of its underlying assumptions -- and limitations -- remain. There's still a GIL, and it still holds up proceedings.

No GIL? No problem

Despite all this, the quest for a GIL-less Python, compatible with existing applications, continues. Other implementations of Python have done away with the GIL entirely, but at a cost. Jython, for instance, runs on top of the JVM and uses the JVM's object-tracking system instead of the GIL. IronPython takes the same approach via Microsoft's CLR. But both suffer from inconsistent performance, and they sometimes run much slower than CPython. They also can't interface readily with external C code, so many existing Python applications won't work.

PyParallel, a project created by Trent Nelson of Continuum Analytics, is an "experimental, proof-of-concept fork of Python 3 designed to optimally exploit multiple CPU cores." It doesn't remove the GIL, but ameliorates its impact by replacing the async module, so apps that use async for parallelism (such as multithreaded I/O like a web server) benefit most. The project has been dormant for several months, but its documentation states that its developers are comfortable taking their time to get it right, so it can eventually be included in CPython: "There's nothing wrong with slow and steady as long as you're heading in the right direction."

One long-running project by PyPy's creators has been a version of Python that uses a technique called "software transactional memory" (PyPy-STM). The advantage, according to PyPy's creators, is "you can do minor tweaks to your existing, nonmultithreaded programs and get them to use multiple cores."

PyPy-STM sounds like magic, but it has two drawbacks. First, it's a work in progress that currently only supports Python 2.x, and second, it still takes a performance hit for applications running on a single core. Since one of the stipulations cited byPython creator Guido van Rossum for any attempts to remove the GIL from CPython is that its replacement shouldn't degrade performance for single-core, single-threaded applications, a fix like this won't land in CPython in its current state.

Hurry up and wait

Larry Hastings, a core Python developer, shared some of his views at PyCon 2016 about how the GIL could be removed. Hastings documented his attempts to remove the GIL and in doing so ended up with a version of Python that had no GIL, but ran agonizingly slowly because of constant cache misses.

You can lose the GIL, Hastings summed up, but you need to have some way to guarantee that only one thread at a time is modifying global objects -- for instance, by having a dedicated thread in the interpreter handle such state changes.


Previous Page  1  2  3  Next Page 

Sign up for CIO Asia eNewsletters.