Since this spring when I sold the last print copies, Modern IDL has been available as PDF only. I have not wanted to make another print run because of the large up front investment required, but I have finally explored the on demand printing services that have been around for awhile. I am happy to announce that a print version of Modern IDL is now available on Lulu.com.

The new version is perfect bound, not spiral bound like the old print version. Also, new features of IDL 8.5 have been included in this new print version and the PDF (still available on the Modern IDL website).

If you purchased the Modern IDL PDF in the past and have not received a link to the new version, please let me know.

The IDL 8.5 feature that I am most excited about is the Jupyter notebook kernel for IDL. For a certain type of analysis, the notebook is a great tool for both recording an interactive session as well as presenting it to others. From the Jupyter documentation:

The Jupyter Notebook is a web application for interactive data science and scientific computing. It allows users to author documents that combine live-code with narrative text, equations, images, video and visualizations. These documents encode a complete and reproducible record of a computation that can be shared with others on GitHub, Dropbox and the Jupyter Notebook Viewer.

The IDL kernel provided in IDL 8.5 works well. The only issue I had with the notebook was that only the first direct graphics plot was displayed. The workaround was simple, though a bit tedious – I just needed a WINDOW command before each plot. I had no issue with function graphics plots, widgets, or non-graphical commands in my first explorations.

I’m still uncertain of the exact use cases for which the notebook will be valuable, but I intend to spend some time trying to find them. As a simple example, here is a notebook (and HTML representation) of the first section of my book, Modern IDL.

This is a great article about managing code in your personal library – code that might not be cared for as much as code in a project seen by others.

Over time you have probably developed a set of python scripts that you use on a frequent basis to make your daily work more effective. However, as you start to collect a bunch of python files, the time you take take to manage them can increase greatly. Your once simple development environment can become an unmanageable mess; especially if you do not try to have some consistency and common patterns for your development process.

Except for the section on pandas, almost everything is just as applicable to IDL as Python. I know I changed my attitude about my own library at some point, converting it into an open source project (even if no one uses it). Making the code usable for others makes it readable for yourself in a few months (weeks, days?). Even if your code isn’t actually available to others, acting like it could be is valuable.

One of the major features of IDL 8.5 is the two-way bridge between IDL and Python. This allows Python functionality to be accessed from IDL (Python has a lot of libraries for things that fall outside of the standard scientific routines found in IDL) as well as accessing IDL functionality from Python (call legacy IDL code).

The IDL-Python bridge works with either Python 2 or 3 (whew, I’m still on Python 2!).

It is possible to use any of the vast array of libraries available for Python from IDL. For example, we can use one of the interpolation methods in SciPy just by importing it:

IDL> interpol = python.import('scipy.interpolate')

Now, create some data in IDL to interpolate:

IDL> lat = randomu(seed, 20) + 40.0
IDL> lon = randomu(seed, 20) - 100.0
IDL> values = randomu(seed, 20)

Now, create the interpolation function with those values:

IDL> rbfi = interpol->Rbf(lat, lon, values)

Finally, call the interpolation routine on with the locations we need values for:

IDL> print, rbfi([40.5], [-99.5])
0.37419477

To make this side of the bridge even easier from the IDL command line, there is a special Python mode that can be entered simply by entering >>>:

IDL> >>>
>>> import numpy
>>>
IDL>

Enter a blank line to get back to IDL.

From Python, use the idlpy package access IDL:

>>> from idlpy import IDL
>>> x = IDL.findgen(10)
>>> print x
[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]

The Python bridge docs are online, check them out for more details.

One of the most exciting aspects of the IDL-Python bridge is the IDL Jupyter notebook kernel, which I will discuss next week.

Another of the side effects of the IDL-Python bridge is to add the ability to define dynamic methods (function pointers are the other). Dynamic methods are a way to define arbitrary methods for a class at runtime.

To implement dynamics for a class, inherit from IDL_Object and define the following method:

function my_class::_overloadmethod, method_name, a, b, c

As always, method names, which will be passed as method_name to this function, must be a valid IDL method name.

While the other Python bridge related feature, function pointers, should be a useful addition that I intend to use, I would recommend staying away from dynamic methods. The only reason I could see for using them is to dynamically make bindings for a library in another language such as is done with the Python bridge.

See the IDL Data Point article for more information.

An important side effect of the IDL-Python bridge is to add some Python features to IDL in order to be able to effectively expose a Python API in IDL. One of these is the introduction of “function pointers”, a way of treating an object as a function. This is useful because objects have a lot of nice properties just from being a variable – they can be stored in a variable, passed to other functions, saved in a .sav file, etc. With the introduction of function pointers, functions have these same properties.

To define an object to be used as a function pointer, a class needs to inherit from IDL_Object and define an _overloadFunction method:

function my_class::_overloadFunction, arg1, arg2, arg3, ...

Then, an object instantiated from this class can be called like a function and this method will be called to produce the result:

IDL> compile_opt strictarr
IDL> obj = my_class()
IDL> print, obj(1, 2, 3)

The “strictarr” or “idl2” compile_opt is needed or IDL will use the _overloadBracketsLeftSide method.

For example, let’s define a simple class that represents the histogram equalization image processing operation in mg_hist_equal__define.pro:

;= operators

function mg_hist_equal::_overloadFunction, im
  compile_opt strictarr

  return, hist_equal(im, percent=self.percent, top=self.top)
end

;= property access</p>
<p>pro mg_hist_equal::setProperty, percent=percent, top=top<br />compile_opt strictarr</p>
<p>if (n_elements(percent) gt 0L) then self.percent = percent<br />if (n_elements(top) gt 0L) then self.top = top<br />end</p>
<p>;= lifecycle methods</p>
<p>function mg_hist_equal::init, _extra=e<br />compile_opt strictarr</p>
<p>if (~self-&gt;IDL_Object::init()) then return, 0</p>
<p>self-&gt;setProperty, _extra=e</p>
<p>return, 1<br />end</p>

pro mg_hist_equal__define
  compile_opt strictarr

  !null = { mg_hist_equal, inherits IDL_Object, $<br />percent: 0.0, $
top: 0 }
end

The class defines a few properties that will be passed to the HIST_EQUAL function when called, but to be a function pointer, all that is needed is to inherit from IDL_Object and define the _overloadFunction method.

As an example of using this class, let’s read in an image:

IDL> dims = [248, 248]
IDL> file = filepath('convec.dat', subdir=['examples', 'data'])
IDL> mantle = read_binary(file, data_dims=dims)

Then define our function pointer:

IDL> he = mg_hist_equal(percent=10.0, top=255)

We could just call our function pointer like this:

IDL> equ_mantle = he(mantle)

But instead let’s define a function that applies any function pointer to a variable:

function mg_function_pointer_demo, im, op
  compile_opt strictarr

  return, op(im)
end

Then we can pass our function pointer to this function:

IDL> equ_mantle = mg_function_pointer_demo(mantle, he)
IDL> window, xsize=dims[0] * 2, ysize=dims[1]
IDL> tv, mantle, 0
IDL> tv, equ_mantle, 1

This code is in mg_function_pointer_demo.pro and can be called with:

IDL> .run mg_function_pointer_demo

Function pointers have many applications in such areas where an algorithm has a callback function such as fitting and optimization algorithms. The syntax of function pointers would be cleaner than passing function names as strings and also allows the various parameters of the function represented by the function pointer to be set before passing it.

See the IDL Data Point article for more information.

IDL 8.5 is available for download from the Exelis VIS site, though the official release will be in September. Chris Torrence, IDL developer:

Just to clear up any speculation or confusion, there were some contractual reasons why we needed to release IDL 8.5 and ENVI 5.3 now. But the “official” release will actually be in September, and that’s when we’ll make an announcement and send out emails to everyone.

So just think of this as a perk for those of you who are reading our blogs and on the newsgroup. It’s like being in on the kickstarter before anyone else.

There are some exciting new features in IDL 8.5, including:

1. function pointers and dynamic methods
2. wget function
3. function graphics updates
4. IDL-Python bridge
5. Project Jupyter notebok

I will have more details about each of these once I get IDL 8.5 licensed and explore a bit.

For some time, I have had a need for an easy way to make use of all of the cores of a single machine through multiple threads or processes, i.e., not a SIMD/vectorized paradigm. The IDL_IDLBridge is capable of doing this, but setup and usage is fairly painful. To make it easier, I have created a simple multicore library for IDL.

The simplest example is to create a process to execute an IDL command and then meet up with the main thread, like the mg_process_join_demo in the examples:

p = mg_process(name='subprocess', output='')
p->execute, 'mg_process_join_demo', /nowait
; main process free to do other stuff at this point
p->join ; main process and subprocess will meet here

Another example uses a pool class to create a pool of processes that work together to perform their part of a larger task. In this example, we will compute x^2 + y^3 for 100 values^[1] using a pool with the number of processes corresponding to the number of cores available. We need a function to apply to each element of our x and y arrays:

function mg_pool_map_demo, x, y
  compile_opt strictarr

  return, x^2 + y^3
end

Now, we create a pool with the default number of processes being the number of cores available:

pool = mg_pool()

Create our datasets:

n = 100L
x = findgen(n)
y = 0.5 * findgen(n)

Then just pass our data to MG_Pool::map and indicate which function to apply to the data:

x_squared = pool->map('mg_pool_map_demo', x, y)

This will block, i.e., it returns control to the main process when finished processing. At this point, the result is computed:

print, x_squared, format='(8(F10.2))'

I have used this library to get fairly good performance on an 8 core laptop doing some image processing operations on a few hundred image files, as shown in the graph below.

mgunit 1.5 has been released! New features include:

• Passing keywords to MGUNIT down to MGutTestCase subclasses.
• Reporting coverage of tested routines.
• Adding Cobertura output option.
• Allowing up to 8 arguments for substituting into ASSERT error message.

I am most excited with the reporting of code coverage in this version, making use of an IDL 8.4 feature. I gave an example of doing this in this post.

You can download a distribution with a .sav file and documentation, or just access the repo as needed.

If you use subloggers in MG_LOG (in the dist_tools directory of mglib), I have made a big change in how the priority of a sublogger message is handled.

First, some background: MG_LOG is a routine that lets you configure log output in an application. For more detailed information, see previous articles I have published about it. But the simplest example is:

IDL> mg_log, 'starting application', /info
2015-04-29 19:45:55 INFO: $MAIN$: starting application

The INFO keyword indicates the level of the message: critical, error, warning, informational, or debug. The logger’s level (along with many other configuration details such as setting a filename for the output log, formatting the log message, etc.) can also be set. For loggers, there is also a “not set” level indicating that the logger’s level should not be used (this will make sense in a second). Only messages with a priority as high or higher than the logger’s will be printed. For example:

IDL> mg_log, logger=logger
IDL> logger->setProperty, level=3 ; warning
IDL> mg_log, 'this message should be printed', /warning
2015-04-29 19:53:33 WARN: $MAIN$: this message should be printed
IDL> mg_log, 'this message should NOT be printed', /info
IDL>

Furthermore, you can have nested named loggers and configure them independently. The change has been to how the level of the message gets compared to the levels of the nest loggers in this case.

For example, let’s get the logger objects for a named logger and two of its children:

mg_log, name='mg_log_demo', logger=logger
mg_log, name='mg_log_demo/sub1', logger=sub1logger
mg_log, name='mg_log_demo/sub2', logger=sub2logger

Then set their levels in the following manner:

logger->setProperty, level=3 ; warning
sub1logger->setProperty, level=5 ; debug
sub2logger->setProperty, level=1 ; critical

Which messages should now appear? My previous scheme was to use the most restrictive level in the hierarchy. So for example, a message to the “mg_log_demo/sub1” logger would have to have level 3 (warning) or higher to pass. But this made it hard to have a fairly restrictive high level setting, but turn on debug output for the section of the application you were currently working on. So I have reversed the logic: the least restrictive level in the hierarchy is now used. So for our setup:

mg_log, 'should appear', name='mg_log_demo', /warn
mg_log, 'should not appear', name='mg_log_demo', /info
mg_log, 'should appear', name='mg_log_demo/sub1', /debug
mg_log, 'should appear', name='mg_log_demo/sub2', /warn
mg_log, 'should not appear', name='mg_log_demo/sub2', /info
mg_log, 'should appear', name='mg_log_demo/sub3', /warn
mg_log, 'should not appear', name='mg_log_demo/sub3', /info

For the common case of setting your top-level logger to informational and setting a sublogger for the section of code you are working on to debug, this should now work as expected. I also added a “not set” logger level (level 0) which indicates that the logger level should not be used when finding the least restrictive level in the hierarchy. The “not set” level is the default level for loggers. If no loggers have a level set, all messages get logged.

Whew! This seems complicated, but I think it is the natural way to thing about this and the most useful in common situations.