I presented a poster at a Space Weather workshop at the Lorentz Center in Leiden, Netherlands last week:

Real-time automated detection of coronal mass ejections using ground-based coronagraph instruments

Coronal mass ejections (CMEs) are dynamic events that eject magnetized plasma from the Sun’s corona into interplanetary space. CMEs are a major driver of solar energetic particle (SEP) events and geomagnetic storms. SEP events and geomagnetic storms pose hazards to astronauts, satellites, communication systems, and power grids. Understanding CME formation and predicting their impacts at Earth are primary goals of the National Space Weather program. St. Cyr et al. (2017) reported on the use of near real-time white light observations of the low corona from the COSMO K-Coronagraph (K- Cor) to provide an early warning of possible SEP events driven by fast CMEs. Following that work, one of us (Thompson) created a new CME detection algorithm adapted from the Solar Eruptive Event Detection System (SEEDS) code for use with K-Cor observations from the Mauna Loa Solar Observatory (MLSO) in Hawaii. We develop performance metrics and report on the success of the algorithm to detect CMEs in the 2017 K-Cor observations. Measures of success include the ability of the algorithm to detect an event and the amount of time between the event onset and its detection. The algorithm successfully detected 20 of the 35 CMEs identified between 1 Jan and 31 August, 2017 in the K-Cor data. There were 10 false positive events during this time period. The threshold for CME detection is discussed as a function of CME visibility, instrument background, and sky noise. The code has been modified to run in an automated mode and is in the process of being integrated into the real-time data processing pipeline at Mauna Loa. We report on current status, real-time alerts, and future upgrades.

Here's a tutorial of how to make an animation of the moon's shadow with GOES imagery during The Great American Eclipse of 2017:

Here is one of the coolest examples that I have created using IDL in a while. For this blog post, I'm going to walk through how I created an animation of the Moon's shadow during the Great American Total Solar Eclipse using several different technologies for accessing, downloading, and visualizing the data.

The video is on Harris Geospatial Solutions' Facebook page.

IDL 8.6.1 was released today¹. Some interesting new features:

• Conditional breakpoints from the Workbench
• Hexadecimal constants, e.g., a = 0xFF3A
• Fix for strings that begin with numerals being confused with the octal notation: "123 is an octal value; "123" used to be a syntax error, but is now a valid string.

See the release notes for details.

Here are some odd, but totally legal, IDL statements. I would suggest staying away from all of them and use more conventional syntax.

Here's one that looks like a string, but is really specifying an octal value:

IDL> x = "12
IDL> help, x
X               INT       =       10

Back when octal values were much more important than now, I suppose it made some sense to have special syntax for entering them. In modern times, I would suggest x = '12'o.

This also means that if you are specifying a string that begins with digits 0-7 with double quotes, you can generate a bewildering syntax error:

IDL> y = "12 monkeys"
              ^
% Syntax error.

I recommend using single quotes for all strings, i.e., y = '12 monkeys'.

Next up is a convenience for the truly lazy:

IDL> s = 'some string

You don't have to put the trailing single or double quote on a string if it is the last character on the line. This will probably make your text editor's syntax highlighting confused. One character is not too much to type for some clarity.

Finally, I just saw this one last week:

IDL> for i = 0, 4 do begin y = i & print, i
       0
       1
       2
       3
       4

There are quite a few problems with this:

• There is a begin with no matching end!
• There is not a & after the begin even though you would normally have to start a new line there.
• I would recommend against using &. It can be useful on the command line (it makes it easier to up arrow to a previous set of commands), but don't do it in a file!

This is the standard syntax for that line (if you really need to put it all on a single line):

IDL> for i = 0, 4 do begin & y = i & print, i & endfor

I might count using parentheses for indexing arrays as a syntax oddity as well, but there are so many IDL programmers still doing it that it counts as commonplace. I still recommend against it.

Licensing has been the most controversial change in IDL 8.6. The release notes say:

IDL licensing is now managed through a 3rd-party solution from Flexera software. You obtain the license through a portal hosted by Flexera, then you can choose to activate the license on a license server or on an individual node-locked machine.

This seems like a more convenient solution, but there are a lot of other changes in the licensing for IDL 8.6.

Limits have been placed on the number of instances of IDL running on a machine. For a local (node-locked) license, the number of IDL instances is limited by:

• IDL command line or Workbench - 4
• Execute compiled save code - 4
• IDL Bridge Processes - 16
• IDL Task Engine - 1

For a served (floating) license:

• IDL command line or Workbench - 1
• Execute compiled .sav code - 1
• IDL Bridge Processes - 8
• IDL Task Engine - 1

The flexible single user license which allowed people to use IDL at work and at home (or lab) with a single license has also been eliminated in IDL 8.6.

Furthermore, the IDL 8.6 Virtual Machine cannot currently be downloaded from the Harris site. In the past, this has allowed IDL developers to release applications to users who did not need the full IDL distribution, or an IDL license, to run the application.

Complaints resulted in a proposed change for the next release of IDL. IDL Project Lead, Chris Torrence wrote on Feb 1:

Starting with IDL 8.6.1 (hopefully mid-April), we will make the following changes:

• An IDL user will be able to run an unlimited number of sessions on their machine. In IDL 8.6 the IDL license was tied to the MAC address + install location + process ID, so each process ID would consume a separate license. In IDL 8.6.1, the IDL license will be tied to the MAC address + install location + user id, so multiple process ID’s will consume just a single IDL license.
• This change will apply to the IDL command line, the IDL Workbench, and the Python bridge, on all platforms.
• This change will not apply to ENVI or other Harris Geospatial products.
• There is no policy change for "flexible single user" (other than allowing multiple IDL sessions on one machine). If you need to use IDL on two machines, you should contact Tech Support or your sales rep for options.
• IDL Virtual Machine will remain unchanged from pre-IDL 8.6 – we just need to tie up some lose ends and release it.

IDL 8.6 also has an automatic check for updates (you can turn off with the "IDL_UPDATE_CHECK" preference) that will tell you when an update is available.

Harris Geospatial released IDL 8.6 at some point in the last couple months—it's hard to pick an actual day. I've heard the release was rolled out to customers in batches since then and it was finally my turn last Friday!

The release notes list the new features. I am very interested in checking out the IDL Task Engine; I think it will be extremely useful. There are quite a few small features and changes that I think I will regularly use.

I will have more details in the coming weeks as I look at individual new features one by one.

I have been doing some reading about machine learning recently, using Python as an implementation language. I lot of the routines used are fairly easy to implement in IDL, so I have started filling out my library with IDL versions.

I have written a scatter plot matrix routine that takes a collection of vectors and makes all the scatter plots between pairs of them. For example, here's a scatter plot matrix produced by the routine for the classic iris dataset:

If you want to use the routine, it's probably easiest to clone my entire library.

Upgrading from XQuartz from 2.7.9 to 2.7.11 on macOS Sierra broke IDL widgets for me:

~$ idl
IDL Version 8.5.1, Mac OS X (darwin x86_64 m64).
(c) 2015, Exelis Visual Information Solutions, Inc., a subsidiary of Harris
Corporation.

IDL> xloadct
Error: attempt to add non-widget child "dsm" to parent "idl" which supports
only widgets

The fix that worked for me were the following two commands:

sudo mv /opt/X11/lib/libXt.6.dylib{,.bak}
sudo cp /opt/X11/lib{/flat_namespace,}/libXt.6.dylib

Downgrading to 2.7.9 (but not 2.7.10) also worked for me.

I plot a lot of data on daily cycles, where there is no data collected at night. Let's mock up some sample data with the following simple code:

IDL> x = [findgen(10), findgen(10) + 25, findgen(10) + 50]
IDL> seed = 0L
IDL> y = randomu(seed, 30)
IDL> plot, x, y

Then I get a plot like this:

This plot doesn't show the nightly breaks in data well. Connecting the last data point collected from a day to the first data point collected the next day emphasizes the trend between these points, which may not be appropriate.

I have been using a fairly simple routine to insert NaNs into the data to break the plot into disconnected sections. For example, modify the above data for plotting with:

IDL> new_y = mg_insert_nan(x, y, [10.0, 35.0], new_x=new_x)
IDL> plot, new_x, new_y

The new plot shows the gaps between the "days" in the data:

James Hague writes:

Though my fascination with Forth is long behind me, I still tend toward minimalist programming, but not in the same, extreme, way. I've adopted a more modern approach to minimalism:

Use the highest-level language that's a viable option.

Lean on the built-in features that do the most work.

Write as little code as possible.

I think this is good advance, but I would add one more point about having as few third party dependencies as possible that tries to balance the last point to write as little code as possible.