TaskDL is a task farming library for IDL that allows you to farm out tasks using multiple cores of a single computer or even multiple computers. It is available on Linux, OS X, and Windows. Task farming is suitable for tasks which do not need to communicate with each other, i.e. “naturally” or “embarrassingly” parallel tasks, such as processing many files independently. For more complicated programs which required interprocess communication, mpiDL provides an interface to MPI (Message Passing Interface).

As an example of using TaskDL, I will present a program to compute some areas of the Mandelbrot set and create output files representing them. If you are interesting in evaluating TaskDL or require more information about, please contact me.

The first program needed when using TaskDL is the compute task program that will be called, in our case this will be called mandelbrot_compute.pro. It is a normal IDL program that typically does not need to know about TaskDL and normally just places output in files. Our mandelbrot_compute.pro example has the following interface:

pro mandelbrot_compute, x_range, y_range, nx, ny, $
                        max_iterations=max_iterations, bound=bound, $
                        color_table=color_table, image_file=image_file, $
                        data_file=data_file, uniform_color=uniform_color

The driver of this program, mandelbrot.pro, is in charge of setting up the task farm, creating the tasks, and sending them off to the workers. To begin, creating a TaskDL object and open a new session on a particular host and port:

oFarm = obj_new('TaskDL', _extra=e)
oFarm->open_session, host=server_host, port=server_port

To run locally, server_host would simply be localhost. Port is typically just any unused port.

TaskDL has optimizations for running locally, use the ::spawn_local_worker and ::spawn_worker methods as needed to create as many workers as required, typically matching the number of processing units (cores or nodes) as available:

for w = 0L, n_workers - 1L do begin
  if (keyword_set(local)) then begin
  endif else begin
    ofarm->spawn_worker, host=_worker_host[w mod n_hosts]

The commands, as strings, to be sent to the workers much be constructed. This construction and the ::add_task call would typically be done in a loop, in our example, over the number of zoom levels desired:

cmd_format = '(%"mandelbrot_compute, %s, %s, %s, %s, max_iterations=%s,' $
               + 'image_file=''%s'', data_file=''%s''")'
cmd = string(x_range_str, $
             y_range_str, $
             nx_str, $
             ny_str, $
             max_iterations_str, $
             image_file, $
             data_file, $

ofarm->add_task, cmd, queueid=0, stage=1

Multiple queues can be created associated with specific workers, but in our simple example we use the default queue. Stages provide the ability to require work to progress in stages, i.e., all stage 1 tasks must complete before stage 2 tasks start, etc. Again, that is not needed for our example.

When done, close the TaskDL session:


Output is placed in mandelbrot-[zoom_level].png and mandelbrot-[zoom_level].nc files.

Full disclosure: I work for Tech-X and I am the product manager for the FastDL suite which includes TaskDL.

UPDATE 3/24/2014: Here is the TaskDL Users Guide.