Harris Geospatial^[1] released IDL 8.5.1 yesterday. The release notes show a few nice additions, appropriate to a patch release version: a new FILE_MODTIME, an ::IsFoldCase method for some of the container classes, routines for handling VGroup attributes in HDF4 files, IDLnetURL::URLEncode/IDLnetURL::URLDecode methods, writing a PNG to a buffer instead of a file, and an update to the HDF4 library.

I migrated my libraries to git, and GitHub, from Subversion over eight and a half years ago. While there is a bit of a learning curve to git, it is possible to handle basic operations fairly easily. The following is an overview of the basic features of git with some comparisons to Subversion. This basic overview completely omits many features of git; for more information check out the git docs.

Setup

It is useful to configure git with your name, email, and editor for all the projects that you will be using it for. For example, the following is my setup for git:

$ git config --global user.name "Michael Galloy"
$ git config --global user.email mgalloy@gmail.com
$ git config --global core.editor "emacs -nw"
$ git config --global core.autocrlf input

You can also do configurations per project. For example, in the case you have a project where you interact with a remote repository where you have a different account name.

Create a new repo

If you have a directory (with possible subdirectories) that you would like to turn into a git repo, use init from the directory:

$ git init

This is a major advantage of git over Subversion: you can very quickly turn any directory into a git repository and start working.

Use clone to make your own local copy of an existing repo, e.g., a repo created on GitHub. For example, the following command will grab the mglib GitHub repo and make a local copy:

$ git clone https://github.com/mgalloy/mglib.git

This is analogous to a Subversion checkout.

Add files and commit them

If you are familiar with Subversion or other version control systems, this is similar. We need to tell git to track a particular file with the add subcommand, then use commit to indicate that you want to save the current contents:

$ git add
$ git commit

There are several differences with Subversion at this stage though:

1. The changes are only saved locally, not pushed to a centralized repository (see next section about communicating with a remote repository).
2. It is necessary to “stage” files with the add subcommand before every commit, not just the first time.

The status subcommand is useful to check which the state of the files in repo:

$ git status

Push/pull to/from another repository

If you cloned from a remote repository, then you have a remote called “origin”. You can push commits from your local repository to this remote repository with the following:

$ git push origin master

The “master” refers to the default branch. Similarly, you can retrieve new commits from origin by pulling:

$ git pull origin master

Look at your history

The log subcommand gives the history of your commits:

$ git log
commit cac05188a6997d4dc8febd7363d706768cd8cd16
Author: Michael Galloy <mgalloy@gmail.com>
Date: Mon Aug 25 18:40:50 2014 -0600

Doing total to find contrast.

commit 4bb170a6a57e53a87b1a5dd4db3ed31774f977af
Author: Michael Galloy <mgalloy@gmail.com>
Date: Mon Aug 25 17:51:22 2014 -0600

Adding example in main-level to compute contrast from GLCM.

Sometimes, I like using a few options to get a history that is easier to glance down through many changes:

$ git log --color --pretty=oneline --abbrev-commit
cac0518 Doing total to find contrast.
4bb170a Adding example in main-level to compute contrast from GLCM.

Note that git commits are referenced by a long hash value, not an incrementing integer value. Because of the distributed nature of git, it would not be possible to keep a simple incrementing counter. The first seven digits of the hash are usually all that is needed to reference a commit.

See your changes

The diff subcommand shows differences between items. To see the changes that have not been added yet do:

$ git diff

To see the changes just for a single filename, do:

$ git diff FILENAME

To see the changes between two commits, do:

$ git diff COMMIT1 COMMIT2

Undo a change

One of the main features of version control is to be able to revert changes and to retrieve old versions of code. There are multiple commands to revert, depending on the stage of the code that is being reverted.

For example, to destroy the changes to a file that has not been added or commited, you can just checkout a fresh copy of it:

$ git checkout --

To “visit” the repo at the state it was on a specific commit, do:

$ git checkout

To make a new commit that undoes existing commits (like a version control inverse function), use the reset subcommand. The first few characters of a hash found from the log subcommand are needed to refer to commits, though there is special notation for the most current commit HEAD and back n commits HEAD~n. So to create a new commit that reverts the last two commits, you can do:

$ git revert HEAD~2..HEAD

To create a new commit that reverts the commits from 0123456 to the HEAD, do:

$ git revert 0123456..HEAD

It is also possible to actually delete commits using the reset subcommand, either “soft” or “hard”. This can be problematic if the commit has been pushed to a remote; it is much simpler if the commit hasn’t been pushed yet.

For example, to delete the last commit, keeping the state of the repo as it was just before the commit, i.e., the code changes are still present:

$ git reset --soft HEAD~1

To really delete the commit and its changes fully, do:

$ git reset --hard HEAD~1

Branching and merging

Branching and merging are fundamental to using git effectively. Branching is useful because there are usually multiple versions of the code that are progressing at the same time. For example, you might create a branch for a specific feature so that large changes can be made without breaking the main branch. Or a branch for a release might be made so that quick fixes for the release can be made as well as progressing along for the next release.

To create a branch named “geometry”, do:

$ git branch geometry

That will create the branch, but you will still be on the default “master” branch. To change to the “geometry” branch, do:

$ git checkout geometry

After doing commits on this branch, when it is time to merge the changes back into the “master” branch, use the merge subcommand:

$ git checkout master
$ git merge geometry

There can be conflicts in a merge, if both branches have made changes. In that case, the merge will indicate the issues. Fix the conflicts and do an add/commit. Use git status to find the remaining conflicts.

If you are done with the geometry branch now, you can delete it with:

$ git branch -d geometry

Great essay about climate change and what you can do about it by Bret Victor:

This is aimed at people in the tech industry, and is more about what you can do with your career than at a hackathon. I’m not going to discuss policy and regulation, although they’re no less important than technological innovation. A good way to think about it, via Saul Griffith, is that it’s the role of technologists to create options for policy-makers.

Also fascinating is the presentation of his argument and the surrounding facts, particularly the interactive “model-driven debate” text in the Media section. He imagines (and has implemented in the article) news articles with input values that the reader can modify to explore different scenarios and output values the reader can click to find an explanation of how they were calculated. Wow!

I wrote an overview almost eight years ago about IDL’s basic routines to visualize a vector field. Not much has changed since then. This paper by Laidlaw, et al, mentioned at the end of my overview, outlines six common visualization techniques and evaluates them on the speed and accuracy of doing common tasks with them.

I think it would be great to have two of the better performing techniques: OSTR (image-guided streamlines, integral curves) and LIC (line-integral convolution) techniques. Between the two of them, they have the best or nearly the best performance in every category. An example OSTR visualization is shown below:

I have tackled LIC and have code to produce images from vector fields in my library.

While interactive graphics can be created in IDL, you are currently limited to distributing an IDL application to provide them to others. It would be useful to be able to create interactive web graphics from IDL — particularly if it was just a click or ->save call from function graphics.

Bokeh is a library for Python capable of generating Javascript to display interactive visualizations. Check out its gallery for examples. Bokeh has bindings for other languages; all that is required for the language is to output the correct JSON to the Javascript part of Bokeh. Bokeh is BSD licensed so it could be used in a commercial product like IDL.

Of course, Bokeh can be used via the Python bridge right now, but that adds an additional burden to its use.

IDL already contains syntax for passing arbitrary (and perhaps unknown beforehand) keywords to a routine using the _EXTRA and _REF_EXTRA keywords, depending on whether you are passing a variable back to the caller through the keyword. For example, you can do something like this:

pro my_wrapper_routine, _extra=e
  my_routine, _extra=e
end

Now, MY_WRAPPER_ROUTINE passes all the keywords that MY_ROUTINE accepts along to it (ignoring any others, use _STRICT_EXTRA on the MY_ROUTINE call to generate an error if an unknown keyword is passed in).

It would be useful to have a similar mechanism for positional parameters, making it possible to write routines in IDL which accept an arbitrary number of keywords or pass along parameters to another routine. For example, passing all parameters after the first two to another routine and printing the first “extra” parameter with the new library routine GET_PARAM:

pro my_wrapper_routine, a, b, _extra
  my_routine, _extra
  print, get_param(_extra, 0)
end

What exactly would the _extra variable be? I’m not sure. Maybe it isn’t even needed in the GET_PARAM call, but it certainly is needed in the MY_ROUTINE call. Python handles this through tuples and the special * notation, maybe IDL can use structures and a bit of new notation?

But I would certainly like to get rid of code like the following:

case n_params() of
  0: _sql_query = ''
  1: _sql_query = sql_query
  2: _sql_query = string(arg1, format='(%"' + sql_query + '")')
  3: _sql_query = string(arg1, arg2, format='(%"' + sql_query + '")')
  4: _sql_query = string(arg1, arg2, arg3, format='(%"' + sql_query + '")')

IDL has several places where callback functions are used to provide a mechanism for IDL library routines to call user-defined functions. For example, in curve fitting routines such as CURVEFIT the callback function allows the user to define custom functions that are evaluated by the fitting routine.

For many callbacks, this interface is perfectly adequate. But for some cases, the callback routine requires some data that is not passed to it in the interface defined by the calling routine. In this case, it is typical for the callback routine to use a technique like common blocks to access the data. But even Fortran recognizes the problems with common blocks; surely there must be a better way to do this in 2015.

IDL 8.5 provides just such a mechanism – function pointers. Objects which inherit from IDL_Object and define an _overloadFunction method can be called like a function. This would allow passing a function pointer with the appropriate user-define data as a callback function. A reference to object could be kept and the data could be modified as needed.

This would provide a much nicer interface in a wide variety of applications which use a callback routine: numeric routines such as for curve fitting; several areas in widget programming such as XMANAGER itself, creating a compound widget, and WIDGET_TREE drag and drop; and some networking classes such as IDLnetURL and IDLnetOGCW[CM]S.

The change should not be difficult, just a check for a string versus and object passed as the callback. Something like:

function mg_call_callback, callback, arg1, arg2, _extra=e
  compile_opt stricter
  if (size(callback, /type) eq 11) then begin
    return, callback(arg1, arg2, _extra=e)
  endif else begin
    return, call_function(callback, arg1, arg2, _extra=e)
  endelse
end

Of course, this is complicated by the fact that the above routine assumes that there are two positional parameters (though it can handle any keywords), but I will save that for another wish.

Hans Rosling discusses the news media view of the world versus one based on data.

I agree. The media’s need for interesting and entertaining news means that stories such as “It’s a Pretty Safe Time for Most” or “Things are Slightly Better Than Last Year” are not written or at least not emphasized. We should build context through data.

Link via FlowingData.

The ternary operator is a handy syntax for creating a result based on a (scalar) condition that is evaluated for truth. For example, I frequently use it to set a default value for an input variable, like the following that uses a default value of 5.0 if x is not present:

_x = n_elements(x) eq 0L ? 5.0 : x

Without the syntax, you would need an if statement:

if (n_elements(x) eq 0L) then _x = 5.0 else _x = x

(Or, in this case, a special purpose routine like default.) But, besides being a bit longer, the if statement hides the possibility that you might not assign to _x in both cases of the if statement, whereas it is clear in the ternary statement that _x is going to receive a value in any case.

The ternary statement is even more useful because it can be chained with other statements. For example,

results[f] = day + (year eq currentYear $
               ? string(time, format='(A6)') $
               : string(year, format='(I6)'))

Without the ternary operator, this would require a new local variable and would be twice as long:

if (year eq currentYear) then begin
  yeartime = string(time, format='(A6)')
endif else begin
  yeartime = string(year, format='(I6)')
endelse
result[f] = day + yeartime

One of the main limitations to the ternary operator is that, like an if statement, the condition must be evaluated to a scalar truth value. I would like to extend the ternary operator to arrays. Let’s see how it would work.

First, let’s create some arrays to use in the examples:

IDL> x = findgen(5)
IDL> y = 2 * findgen(5)
IDL> z = lindgen(5) mod 2
IDL> print, x
0.00000 1.00000 2.00000 3.00000 4.00000
IDL> print, y
0.00000 2.00000 4.00000 6.00000 8.00000
IDL> print, z
0 1 0 1 0

Then, my proposal would be for the following lines to work:

IDL> print, z ? x : y
0.00000 1.00000 4.00000 3.00000 8.00000

One, or both, of the operators could be a scalar:

IDL> print, z ? 1.0 : y
0.00000 1.00000 4.00000 1.00000 8.00000
IDL> print, z ? 1.0 : 2.0
2.00000 1.00000 2.00000 1.00000 2.00000

Now, of course, this can be done already, but it takes a lot more code. For example, the first case expands from one line to:

IDL> result = fltarr(5)
IDL> ind = where(z, count, complement=not_ind, ncomplement=not_count)
IDL> if (count gt 0L) then result[ind] = x[ind]
IDL> if (not_count gt 0L) then result[not_ind] = y[not_ind]
IDL> print, result
0.00000 1.00000 4.00000 3.00000 8.00000

IDL already has rules for type promotion if an operation has operands of two differing types; those could be used here to determine the type of the result since the elements of both value operands would be intermixed in a single array.

There are quite a few new features in IDL 8.5 that I have not covered in previous articles. Here is a summary list:

• There is now a DIALOG_COLORPICKER which you can present to users to choose a color:

• Analogous to the command line utility with the same name, IDL 8.5 adds a WGET routine to retrieve the contents of a URL:

IDL> wget('http://www.google.com/index.html', filename='google.html')<br />/Users/mgalloy/Desktop/google.html

• In function graphics, the BARPLOT, ELLIPSE, and POLYGON routines now support fill patterns. The new keywords PATTERN_BITMAP, PATTERN_ORIENTATION, PATTERN_SPACING, and PATTERN_THICK specify the fill pattern.

• Before IDL 8.5, it was possible to have hashes of hashes of hashes, etc., but it was necessary to manually create each level of the hashes. Elements of a hash of a hash were accessed via h['a', 'b'], which was shorthand for (h['a'])['b']. Now it is possible to have IDL automatically create the necessary levels of the hash during a hash assignment:

IDL> h = hash()
IDL> h['a', 'b', 'c'] = 5
IDL> h
{
  "a": {
    "b": {
      "c": 5
    }
  }
}

• Server-side sockets actually have been present in IDL for awhile, but they are now documented.