Multiprocessing and Arcpy -1

Multiprocessing module for python., Wow...

it sounds really crazy to me at least, since I have been struggling with this matter of learning how to do that for long time.

As you may have experienced before, whenever I used to work with large feature or raster files, I had to spend lots of time just staring at my monitor and waiting for my python code to execute. But have you ever checked your Task Manager / Performance and see how many of your cores are involved with the processing job? Whenever I checked them, they were like half of them doing something and the rest of them were idle. So what to do?

Ok, here and in this post I'm going to introduce a python module which allows the programmer to fully leverage multiple processors on a given machine and It runs on both Unix and Windows.

Before getting to the introduction, sample code and description I would like to mention just one or two thing s about the differences between threading and multiprocessing. Maybe the most important thing is the fact that in threading approach, different threads share a same memory so whenever one of them dies other threads are affected and die as well. While in multiprocessing since they have their own separate memory if something goes wrong with one of them, nothings gonna happen to the rest.

In order to get this job done, you need to use multiprocessing module. This module in python is part of standard library module and you do not need to install anything while it allows for parallel code execution.

Here is the sample code for using this package: (Actually, in this first post I just go with simple example while in my next post I will explain an integration of this module with Arcpy)

from multiprocessing import Process

def f(name):

print 'hello', name

if __name__ == '__main__':

p = Process(target=f, args=('bob',))

p.start()

p.join()