When working with data files at my day job, I often come across directories containing a large number of files of several distinct types. It would be useful to produce a listing of the files clustered into these types. I wrote `cls` (Clustered ls) to find patterns in filenames for display.

Continue reading “Clustered ls.”

Circular law states the eigenvalues of a matrix with random entries of mean 0 and variance 1/n are approximately uniformly distributed in the unit disk of the complex plane. To see this, create a random matrix:

n = 1000
x = randomu(seed, n, n) - 0.5
x *= sqrt(12.0 / n)

Find the eigenvalues:

eigenvalues = la_eigenproblem(x, eigenvectors=eigenvectors)

Plot them:

plot, real_part(eigenvalues), imaginary(eigenvalues), $
      psym=3, $
      xstyle=1, xrange=[-1.5, 1.5], $
      ystyle=1, yrange=[-1.5, 1.5]

This gives a plot like below:

Via the excellent John D. Cook blog. I recommend reading his site if you are interested in a combination of mathematics and Python.

[IDLdoc][repo] 3.6.2 has been released! New features include:

* Bug fix for image directive for image files in other directories specified
with a relative path (fix by Dave Gellman).

* Only copying MathJax for LaTeX-style equations if not already present.

* Fixed crash when invalid format/markup was specified on the docformat line of
a `.pro` file.

[repo]: https://github.com/mgalloy/idldoc/ “mgalloy/idldoc”
[download]: https://github.com/mgalloy/idldoc/wiki/Releases “IDLdoc releases”

[mgunit][repo] 1.6 has been released! New features include:

* Fix for bug when no filename with jUnit output format

* Recursively search directories below test suite home directory for `*_ut.pro`
and `*_uts.pro` files

* Fixed for bug in `mguttestsuite_define::addTestingFolder` that did not add
absolute paths correctly

* Add superclasses of test classes recursively

You can [download] a distribution with a `.sav` file and documentation, or just access the [repo] as needed.

[repo]: https://github.com/mgalloy/mgunit/ “mgalloy/mgunit”
[download]: https://github.com/mgalloy/mgunit/wiki/Releases “mgunit releases”

IDL 8.7 was released recently. The listed features are:

– `ROUTINE_DIR` function that returns the directory for the file containing the calling routine
– asynchronous job classes
– a few miscellaneous other updates

The asynchronous job classes look interesting:

> The IDLAsync classes allow you to specify units of work to execute asynchronously outside the main IDL session. To do this, create an IDLAsyncQueue and one or more IDLAsyncJob objects that encapsulate the work to be performed. As jobs are added to the queue, they will be executed at some point in the future as resources are available. When a job is complete, you can retrieve its results for further use. You can also construct an IDLAsyncJoin object and pass it into the jobs on creation. If you do this, you can wait on the join object for all of the jobs it observes to be finished before continuing.

Check the [release notes] for more details.

[release notes]: http://www.harrisgeospatial.com/docs/whatsnew.html

The IDL Usenet newsgroup has moved to a Google Group:

This Google Group is a continuation of the Usenet group comp.lang.idl-pvwave, but allows for better spam filtering. It is for discussion of the Interactive Data Language (IDL), developed by Harris Geospatial Corporation. Questions about ENVI, a geospatial analytics software written in IDL are welcome. Discussion of the similar PV-WAVE language is also allowed.

The big question now is how to save the posts from the old newsgroup.

I presented a [poster] at a [Space Weather workshop] at the Lorentz Center in Leiden, Netherlands last week:

> **Real-time automated detection of coronal mass ejections using ground-based coronagraph instruments**
>
> Coronal mass ejections (CMEs) are dynamic events that eject magnetized plasma from the Sun’s corona into interplanetary space. CMEs are a major driver of solar energetic particle (SEP) events and geomagnetic storms. SEP events and geomagnetic storms pose hazards to astronauts, satellites, communication systems, and power grids. Understanding CME formation and predicting their impacts at Earth are primary goals of the National Space Weather program. St. Cyr et al. (2017) reported on the use of near real-time white light observations of the low corona from the COSMO K-Coronagraph (K- Cor) to provide an early warning of possible SEP events driven by fast CMEs. Following that work, one of us (Thompson) created a new CME detection algorithm adapted from the Solar Eruptive Event Detection System (SEEDS) code for use with K-Cor observations from the Mauna Loa Solar Observatory (MLSO) in Hawaii. We develop performance metrics and report on the success of the algorithm to detect CMEs in the 2017 K-Cor observations. Measures of success include the ability of the algorithm to detect an event and the amount of time between the event onset and its detection. The algorithm successfully detected 20 of the 35 CMEs identified between 1 Jan and 31 August, 2017 in the K-Cor data. There were 10 false positive events during this time period. The threshold for CME detection is discussed as a function of CME visibility, instrument background, and sky noise. The code has been modified to run in an automated mode and is in the process of being integrated into the real-time data processing pipeline at Mauna Loa. We report on current status, real-time alerts, and future upgrades.

[Space Weather workshop]: http://lorentzcenter.nl/lc/web/2017/921/info.php3?wsid=921&venue=Oort “Lorentz Center – Space Weather: A Multi-Disciplinary Approach ”

[poster]: http://michaelgalloy.com/wp-content/uploads/2017/10/mgalloy-lorentz-workshop-2017.pdf “Real-time automated detection of coronal mass ejections using ground-based coronagraph instruments”

Here’s a [tutorial] of how to make an animation of the moon’s shadow with GOES imagery during The Great American Eclipse of 2017:

> Here is one of the coolest examples that I have created using IDL in a while. For this blog post, I’m going to walk through how I created an animation of the Moon’s shadow during the Great American Total Solar Eclipse using several different technologies for accessing, downloading, and visualizing the data.

The is on Harris Geospatial Solutions’ Facebook page.

[tutorial]: http://www.harrisgeospatial.com/Learn/Blogs/Blog-Details/TabId/2716/ArtMID/10198/ArticleID/21275/The-Eclipse-Tracking-where-the-Moons-shadow-GOES.aspx?utm_source=twitter&utm_medium=social&utm_campaign=blog “The Eclipse: Tracking where the Moon’s shadow GOES”
: https://www.facebook.com/HarrisGeospatialSolutions/videos/10155456330801006/ “Our IDL 8.6.1 created an animation of the 2017 solar eclipse”

IDL 8.6.1 was released today[^1]. Some interesting new features:

* Conditional breakpoints from the Workbench
* Hexadecimal constants, e.g., `a = 0xFF3A`
* Fix for strings that begin with numerals being confused with the octal notation: `”123` is an octal value; `”123″` used to be a syntax error, but is now a valid string.

See the [release notes] for details.

[release notes]: http://harrisgeospatial.com/Support/Maintenance/TabId/2350/ArtMID/10427/ArticleID/21253/Whats-New-in-IDL-861.aspx “What’s New in IDL 8.6.1”

[^1]: Really sometime in the last week or so. The announcement on the newsgroup was today, but the release notes was posted 7/27.

Here are some odd, but totally legal, IDL statements. I would suggest staying away from all of them and use more conventional syntax.

Here’s one that looks like a string, but is really specifying an octal value:

IDL> x = “12
IDL> help, x
X INT = 10

Back when octal values were much more important than now, I suppose it made some sense to have special syntax for entering them. In modern times, I would suggest `x = ’12’o`.

This also means that if you are specifying a string that begins with digits 0-7 with double quotes, you can generate a bewildering syntax error:

IDL> y = “12 monkeys”
^
% Syntax error.

I recommend using single quotes for all strings, i.e., `y = ’12 monkeys’`.

Next up is a convenience for the truly lazy:

IDL> s = ‘some string

You don’t have to put the trailing single or double quote on a string if it is the last character on the line. This will probably make your text editor’s syntax highlighting confused. One character is not too much to type for some clarity.

Finally, I just saw this one last week:

IDL> for i = 0, 4 do begin y = i & print, i
0
1
2
3
4

There are quite a few problems with this:

* There is a `begin` with no matching `end`!
* There is not a `&` after the `begin` even though you would normally have to start a new line there.
* I would recommend against using `&`. It can be useful on the command line (it makes it easier to up arrow to a previous set of commands), but don’t do it in a file!

This is the standard syntax for that line (if you really need to put it all on a single line):

IDL> for i = 0, 4 do begin & y = i & print, i & endfor

I might count using parentheses for indexing arrays as a syntax oddity as well, but there are so many IDL programmers still doing it that it counts as commonplace. I still recommend against it.