If you try this out, please let me know (via email or submitting a ticket on the IDLdoc project site) if you encounter an issue.

Finally, what new features is IDL still lacking to be considered a truly modern scientific programming language? I’ll have to give this some thought, but feel free to make suggestions in the comments.

As a reference to get all them all listed in a single place, the IDL 8.0 related posts discuss lists and hashes, online help, IDL Workbench changes, distribution/licensing changes, operator overloading, other language changes, and new graphics interface. Also, note that Paulo Penteado updated his MAKE_RT über-installation to IDL 8.0. These articles are not intended to be a complete reference to all the new changes; for more information check out the IDL 8.0 webinar and, of course, the “What’s New” section of the online documentation.

First, the location of the default IDL installation has changed—there is another level added to the hierarchy. For example, by default on a UNIX system, IDL 7.1 was installed in:

/usr/local/itt/idl71

Now, IDL 8.0 is installed in:

/usr/local/itt/idl/idl80

Be careful, this installation location will destroy the link that most of us had at /usr/local/itt/idl to the current IDL version.

Also, system administrators take note: using a network license with IDL 8.0 requires the download and installation of the IDL 8.0 FlexLM License Server. So admins familiar with updating previous IDL installations will have to do something a bit different this time.

I can only just skim the surface of the capabilities of function graphics in this post. See the online help for more details.

To create a plot using function graphics, do:

IDL> p = plot(findgen(10), color=[255, 0, 0])

The plot can be annotated from the GUI interface, but it can also be modified via the return value p, an object which represents the newly created plot. You can get and set properties of the graphics object easily:

IDL> p.color = 'blue'   ; or you could use [0, 0, 255]

The plot line automatically changes to blue.

IDL> print, p.color
   0   0 255

From the command line, to find all the available properties of one of the graphics object returned from a function graphics routine, just print it:

IDL> print, p
PLOT <21317>
  ANTIALIAS                 = 1
  ASPECT_RATIO              = 0.0000000
  ASPECT_Z                  = 0.0000000
  BACKGROUND_COLOR          = 255 255 255
  COLOR                     = 255   0   0
  DEPTH_CUE                 = 0.00000      0.00000
[etc, rest of output omitted for brevity]

By the way, if you want to refer to colors by name in other uses in IDL, you can use the new !color system variable:

IDL> print, !color.blue 
   0   0 255

You can see a listing of all of the color names available by using HELP on !color:

IDL> help, !color, /structures
** Structure !COLOR, 147 tags, length=441, data length=441:
   ALICE_BLUE      BYTE      Array[3]
   ANTIQUE_WHITE   BYTE      Array[3]
   AQUA            BYTE      Array[3]
   AQUAMARINE      BYTE      Array[3]
   [etc, rest of output omitted for brevity]

Note that function graphics can be used programmatically without any graphics windows appearing on the screen:

IDL> c = contour(dist(200), /buffer)
IDL> c->save, 'contour.png'

See the online help on a particular graphics object type to see its available methods, but most seem to have close, convertCoord, copyWindow, getSelect, order, print, refresh, rotate, save, scale, select, and translate.

The WIDGET_WINDOW routine provides a way to add a function graphics window to a widget application.

One of my favorite features in the new graphics API is that LaTeX escape sequences are built-in! For example,

IDL> p = plot(findgen(10), title='Some formula: $\mu = \epsilon^2 + \tau$')

(UPDATE: there is a TEX2IDL routine in the IDL distribution now that can be used independently of the function graphics routines.)

The following functions are part of the function graphics API: BARPLOT, COLORBAR, LEGEND, CONTOUR, ERRORPLOT, IMAGE, MAP, MAPCONTINENTS, MAPGRID, PLOT, PLOT3D, POLARPLOT, SURFACE, VECTOR, AXIS, ELLIPSE, POLYGON, POLYLINE, STREAMLINE, TEXT, and WINDOW.

There is much, much more to be said about function graphics, but hopefully this will get you started exploring. It will take more time to actually do some work using it to determine its shortcomings, but I think it is the most important feature of IDL 8.0.

Stay tuned for one last wrap-up article about IDL 8.0.

A new !null variable was introduced in IDL 8.0. There are several equivalent forms:

IDL> help, !null, [], {}
<Expression>    UNDEFINED = !NULL
<Expression>    UNDEFINED = !NULL
<Expression>    UNDEFINED = !NULL

!null is an undefined variable with a couple special properties:

1. it can be assigned to a variable
2. it can be used in array or structure concatenation

For example, a typical use would be:

IDL> a = []
IDL> a = [a, 1]

This eliminates the need for special cases in code to check for the case of adding the first element, though this use case is generally handled better by the new list objects.

IDL now allows the use of negative subscripts to specify indices relative to the end of an array, with -1 representing the last element:

IDL> a = findgen(10)
IDL> print, a[-1]
      9.00000
IDL> print, a[-2]
      8.00000

Negative indices can also be used in ranges:

IDL> print, a[-2:0:-1]
      8.00000      7.00000      6.00000      5.00000      4.00000
      3.00000      2.00000      1.00000      0.00000

But, negative subscripts are handled in the same manner as in older versions of IDL when placed in an array used to index another array:

IDL> print, a[[-1]]
      0.00000

This behavior can also still be controlled via compile_opt strictarrsubs.

Object-oriented programming has had several important updates. Most important of these features is automatic garbage collection. This will automatically free heap variables, both pointers and objects, which no longer have a reference to them. For example, in the following example, an object is created and then its reference is lost by reassignment to another value:

IDL> o = obj_new('IDL_Container')
IDL> help, /heap
Heap Variables:
    # Pointer: 0
    # Object : 1

<ObjHeapVar1>  refcount=1
                STRUCT    = -> IDL_CONTAINER Array[1]
IDL> o = 2
IDL> help, /heap
Heap Variables:
    # Pointer: 0
    # Object : 0

This situation would have resulted in leaked memory—the help, /heap command would have reported 1 object at the end, but it would not have been accessible through the o variable any longer. Now, memory that can no longer be accessed is automatically freed. This feature can be turned off globally or for a particular variable with the DISABLE keyword to the new HEAP_REFCOUNT function.

There are also two small syntax changes that make objects a bit more like other languages and a bit more convenient:

1. method invocation can be done with the . operator, as well as the old -> operator
2. a function of the same name as the class is automatically generated to create new objects of that class, so the OBJ_NEW function is not required to instantiate an object

These are both shown in the following:

IDL> c = IDL_Container()
IDL> c.add, IDLgrModel()
IDL> help, c.get(position=0)
<Expression>    OBJREF    = <ObjHeapVar5(IDLGRMODEL)>

Note that the . operator and the -> operator a slightly different precedence:

IDL> p = ptr_new(list())
IDL> *p->add, 1
IDL> *p.add, 1

*p.add, 1
         ^
% Syntax error.
IDL> (*p).add, 1

My understanding is that this is not intended and will be fixed in a release soon.

The result is a much faster and nimbler Workbench experience, at the cost of a few features (both non-IDL related Eclipse features and ITT VIS added features from previous versions). For example, perspectives are gone entirely now; instead of “IDL”, “Debug”, and “Visualize” perspectives there is only a single layout for your editors/views.

But other, more useful features, like software update for plugins are also gone. Plugins now must be downloaded directly and copied into the features and plugins folders in the idlde directory of the IDL distribution. I downloaded the Subversion plugin and have it successfully working in the 8.0 Workbench. You now have to manually check for updates and re-install them instead of having the Workbench just automatically check every time it starts.

The best new feature of the Workbench is an integrated command line/output log that looks like a TTY-based command line. Maybe it is my command line bias, but this is much more intuitive than separate views of the command line and output log (I have seen many, many users repeatedly clicking in the output log, trying to enter a new command because they see their last command echoed in the output log).

The Workbench plays a bit nicer on Mac OS X by launching like a normal application instead of creating another icon in my dock (as does the Help system). It’s a small thing, but it’s a bit easier to deal with two IDL-related icons instead of four icons, which was starting to dominate my dock.

I haven’t missed the removed features yet (I used the “Debug” perspective for everything and I can manually install plugins for the few that I use), but I only use the Workbench about 10% of the time I use IDL. Overall, the speed and simplicity gained from the removal of the features seems well worth it for me. I like the focus on improving usability of the Workbench instead of just adding more and more new “features” that I may or may not use.

To make an object which recognizes operators, simply inherit from IDL_Object and write a special method for each operator you wish to use. For example, to make your object interact with the + operator, simply write a method with the signature

function myclass::_overloadPlus, left, right

When an object of your class is one of the operands of an expression using +, as in result = left + right, the special method will be called to compute result.

In addition to the standard unary and binary operators like +, le, unary -, etc., objects can also interact with other special routines and language keywords such as HELP, PRINT, SIZE, and FOREACH. Furthermore, the object can define behavior when it is accessed with []. It also automatically accesses properties when . is used (calling getProperty and setProperty as appropriate).

As an example of using operator overloading, I have some classes and routines for accessing HDF5 files (download). To see these classes in use, checkout the main-level program at the end of mg_h5.pro. Run it with:

ID> .run mg_h5

First, I have a convenience function mg_h5 which creates an object representing an HDF 5 file, an MGffHDF5File object:

IDL> h = mg_h5(file_which('hdf5_test.h5'))

The HELP routine will now present more information:

IDL> help, h
H               MGFFHDF5F = </Applications/itt/idl/idl80/examples/data/hdf5_test.h5>

To make this work, I had to write the following special method:

function mgffhdf5file::_overloadHelp, varname
  compile_opt strictarr

  type = obj_class(self)
  specs = string(self.filename, format='(%"<%s>")')
  return, string(varname, type, specs, format='(%"%-15s %-9s = %s")')
end

An HDF 5 file is hierarchical, so groups in the file can be accessed using []:

IDL> g1 = h['images']

The method to implement this is a bit more complicated, but the signature is:

function mgffhdf5file::_overloadBracketsRightSide, isRange, $
                                                   ss1, ss2, ss3, ss4, $
                                                   ss5, ss6, ss7, ss8

Here, isRange is an array indicating whether each dimension specified using [] is a range (i.e., something like 0:10) or just a particular value (like 5 or 'images'). Then ss1 to ss8 are either 3 element arrays or scalar values, depending on the corresponding value in isRange.

One trick: it is possible to write classes that provide the ability to use operator overloading when used in IDL 8.0, but provide their normal functionality when used with IDL versions before 8.0. Just write your own IDL_Object class (here’s mine). This will not be found when using IDL 8.0, since its IDL_Object is a core built-in class, but will be found in older versions of IDL so that classes which inherit from it will compile.

I think that, used wisely, operator overloading can provide very usable, intuitive objects. But too much operator overloading can lead to overly complicated code which is difficult to debug.

The integration with IDL is still available, using ? on the command line (either from the command line version of IDL or the Workbench) will still bring up the help. In the Workbench, you can select a term and view “Help on Selected Item” (although I couldn’t successfully bind that to a keyboard shortcut). If you launch help from the Workbench, the help viewer will go away when you quit the Workbench, but you can also start the help viewer independently.

My one complaint with the help viewer is that it defaults to the “Search” tab instead of the “Index” tab, which I use almost exclusively.

A list is simply a sequentially ordered container of items of any type. You can add/insert items, remove items, look up an item at a given index, etc. It is similar to an IDL_Container, except that it allows its items to be of any type, not just objects (and the items don’t have to even all be of the same type). For example,

IDL> stations = list()                        
IDL> stations->add, 'Boulder'
IDL> ; try the following without the EXTRACT keyword too                 
IDL> stations->add, ['Fort Collins', 'Denver'], /extract
IDL> stations->add, ['Colorado Springs']                
IDL> print, n_elements(stations) 
           4
IDL> print, stations 
Boulder
Fort Collins
Denver
Colorado Springs
IDL> print, stations[1]
Fort Collins
IDL> foreach s, stations do print, strupcase(s)
BOULDER
FORT COLLINS
DENVER
COLORADO SPRINGS

Hashes are another type of container, but ordered by arbitrary keys instead of by an integer index. For example,

IDL> zip_codes = hash()               
IDL> zip_codes[80303] = 'Boulder'     
IDL> zip_codes[80521] = 'Fort Collins'
IDL> zip_codes[80002] = 'Denver'
IDL> zip_codes[80840] = 'Colorado Springs'
IDL> print, zip_codes
80840: Colorado Springs
80521: Fort Collins
80002: Denver
80303: Boulder
IDL> print, zip_codes[80303]
Boulder
IDL> print, zip_codes->keys()
       80840
       80521
       80002
       80303
IDL> print, zip_codes->values()
Colorado Springs
Fort Collins
Denver
Boulder
IDL> foreach city, zip_codes, code do $
IDL>   print, code, city, format='(%"Zip code %d is in %s")'
Zip code 80840 is in Colorado Springs
Zip code 80521 is in Fort Collins
Zip code 80002 is in Denver
Zip code 80303 is in Boulder

For an example of more extensive use of lists and hashes, I implemented Peter Norvig’s spelling corrector in IDL (download):

IDL> print, mg_spellcorrect('corect')
correct
IDL> print, mg_spellcorrect('correct')
correct
IDL> print, mg_spellcorrect('speling')
spelling

Norvig’s Python implementation uses some list features that IDL does not have like list comprehensions, but IDL’s lists and hashes were up for the task (though it’s regular expressions were not and I had to make a few optimizations to get IDL’s implementation to a reasonable speed, so the first time you run mg_spellcorrect will take longer while it generates the word list and saves it away).

I have my own list and hash classes which provide similar functionality, but require elements of lists and keys/values of hashes to be of a single type. This provides a bit of extra performance (and in practice I never needed to have a single list or hash containing items of differing types) against earlier versions of the IDL list and hash objects, but lists and hashes are faster in the final IDL 8.0 release, as seen in this comparison between adding items to standard arrays, lists, and MGcoArrayLists:

UPDATE 2/23: Fixed x-axis scale on above graph. Code for running the test is available (though the graphics code requires a bunch of my private routines).

Where data geeks ask and answer questions on machine learning, natural language processing, artificial intelligence, text analysis, information retrieval, search, data mining, statistical modeling, and data visualization!

Haven’t asked/answered a question yet, but it looks like a good resource.

Link via FlowingData.

Overall, I thought that the talks were great and I learned something interesting and/or useful from each one. I’m particularly excited about the code for producing perceptually uniform color maps that Paulo Penteado promised to release. It is also great to learn a few new facts about IDL 8.0 at each ITT VIS presentation.

Link via Chart Porn.

Motion charts are essentially animated bubble charts. A bubble chart shows data using the x-axis, y-axis, and the size and color of the bubble. A motion chart displays changes over time by showing movement within the two-dimensional space and changes in the size and color of the bubbles.

Motion charts are used quite effectively by Hans Rosling in his great TED talks.

1957 – John Backus and IBM create FORTRAN. There’s nothing funny about IBM or FORTRAN. It is a syntax error to write FORTRAN while not wearing a blue tie.

and

1972 – Dennis Ritchie invents a powerful gun that shoots both forward and backward simultaneously. Not satisfied with the number of deaths and permanent maimings from that invention he invents C and Unix.

I wonder what IDL’s entry would look like?

Link via BoingBoing.