Chosen aspects of concurrency in Python 3

1 minutes read


This is my first post, written during my cooperation with Idego Group - (source here)

Although Python fully supports thread programming, parts of the C implementation of the interpreter are not entirely thread safe to a level of allowing fully concurrent execution. GIL that only allows one Python thread to execute at any given time. The most noticeable effect of the GIL is that multithreaded Python programs are not able to fully take advantage of multiple CPU cores (e.g., a computationally intensive application using more than one thread only runs on a single CPU). Threads in Python are great for I/O-bound tasks, but they’re a poor choice for CPU-bound problems. If you are going to use a process pool as a workaround, be aware that doing so involves data serialization and communication with a different Python interpreter. For this to work, the operation to be performed needs to be contained within a Python function defined by the def statement (i.e., no lambdas, closures, callable instances, etc.), and the function arguments and return value must be compatible with pickle. Also, the amount of work to be performed must be sufficiently large to make up for the extra communication overhead. There are many Python modules that support concurrency by means threads and processes. Using them carefully can really speed up your programs.