When working with data files at my day job, I often come across directories containing a large number of files of several distinct types. It would be useful to produce a listing of the files clustered into these types. I wrote cls (Clustered ls) to find patterns in filenames for display.

For example, in the directory with 7552 files, the ls output is difficult to parse easily. But cls produces a much more compact listing that can be scanned:

$ cls
*.ls [4 files]
20180101_*_kcor_l1.5.fts.gz [1335 files]
20180101_*_kcor_l1.5.gif [1335 files]
20180101_*_kcor_l1.5_avg.fts.gz [167 files]
20180101_*_kcor_l1.5_avg.gif [167 files]
20180101_*_kcor_l1.5_avg_cropped.gif [167 files]
20180101_*_kcor_l1.5_cropped.gif [1335 files]
20180101_*_kcor_l1.5_nrgf.fts.gz [167 files]
20180101_*_kcor_l1.5_nrgf.gif [167 files]
20180101_*_kcor_l1.5_nrgf_cropped.gif [167 files]
20180101_180511_kcor_l1.5_extavg.fts.gz
20180101_180511_kcor_l1.5_extavg.gif
20180101_180511_kcor_l1.5_extavg_cropped.gif
20180101_180511_kcor_l1.5_nrgf_extavg.fts.gz
20180101_180511_kcor_l1.5_nrgf_extavg.gif
20180101_180511_kcor_l1.5_nrgf_extavg_cropped.gif
20180101_kcor_l1.5.{mp4,tarlist,tgz}
20180101_kcor_l1.5_nrgf_cropped.mp4
20180101_kcor_l1.5_{cropped,nrgf}.mp4
20180101_kcor_minus.mp4
20180102_*_kcor_l1.5.fts.gz [675 files]
20180102_*_kcor_l1.5.gif [675 files]
20180102_*_kcor_l1.5_avg.fts.gz [84 files]
>20180102_*_kcor_l1.5_avg.gif [84 files]
20180102_*_kcor_l1.5_avg_cropped.gif [84 files]
20180102_*_kcor_l1.5_cropped.gif [675 files]
20180102_*_kcor_l1.5_nrgf.fts.gz [84 files]
20180102_*_kcor_l1.5_nrgf.gif [84 files]
20180102_*_kcor_l1.5_nrgf_cropped.gif [84 files]

The arguments for the script are quite simple right now:

usage: cls [-h] [-m MAX_LISTED] [files [files ...]]
Clustered ls 0.0.1

positional arguments:<br />files path specification to check

optional arguments:
-h, --help show this help message and exit
-m MAX_LISTED, --max-listed MAX_LISTED
max number of files to explicitly list

The -m argument indicates the cutoff number of files to use “*” for, by default 3. For example, the line in the above output:

*.ls [4 files]

is changed to:

{okcgif,okfgif,okl1gz,oknrgf}.ls

if you use:

$ cls -m 4

The current code for cls uses a naive method for determining the clusters which does not try to produce optimal clusters. This can change the output because there can be multiple ways to cluster a list of files. For example, in the directory with the following files:

$ ls
a-1.log a-2.log b-1.log b-2.log c-1.log c-2.log

cls can make two different clustering depending on how the original files are ordered:

$ cls *
a-{1,2}.log
b-{1,2}.log
c-{1,2}.log
$ cls *-{1,2}.log
{a,b,c}-1.log
{a,b,c}-2.log

See my scripts repo for code for cls. This code is a proof of concept prototype written in Python 3.

IDLdoc 3.6.2 has been released! New features include:

• Bug fix for image directive for image files in other directories specified with a relative path (fix by Dave Gellman).
• Only copying MathJax for LaTeX-style equations if not already present.
• Fixed crash when invalid format/markup was specified on the docformat line of a .pro file.

mgunit 1.6 has been released! New features include:

• Fix for bug when no filename with jUnit output format
• Recursively search directories below test suite home directory for *_ut.pro and *_uts.pro files
• Fixed for bug in mguttestsuite_define::addTestingFolder that did not add absolute paths correctly
• Add superclasses of test classes recursively

You can download a distribution with a .sav file and documentation, or just access the repo as needed

IDL 8.7 was released recently. The listed features are:

• ROUTINE_DIR function that returns the directory for the file containing the calling routine
• asynchronous job classes
• a few miscellaneous other updates

The asynchronous job classes look interesting:

The IDLAsync classes allow you to specify units of work to execute asynchronously outside the main IDL session. To do this, create an IDLAsyncQueue and one or more IDLAsyncJob objects that encapsulate the work to be performed. As jobs are added to the queue, they will be executed at some point in the future as resources are available. When a job is complete, you can retrieve its results for further use. You can also construct an IDLAsyncJoin object and pass it into the jobs on creation. If you do this, you can wait on the join object for all of the jobs it observes to be finished before continuing.

Check the release notes for more details.

I presented a poster at a Space Weather workshop at the Lorentz Center in Leiden, Netherlands last week:

Real-time automated detection of coronal mass ejections using ground-based coronagraph instruments

Coronal mass ejections (CMEs) are dynamic events that eject magnetized plasma from the Sun’s corona into interplanetary space. CMEs are a major driver of solar energetic particle (SEP) events and geomagnetic storms. SEP events and geomagnetic storms pose hazards to astronauts, satellites, communication systems, and power grids. Understanding CME formation and predicting their impacts at Earth are primary goals of the National Space Weather program. St. Cyr et al. (2017) reported on the use of near real-time white light observations of the low corona from the COSMO K-Coronagraph (K- Cor) to provide an early warning of possible SEP events driven by fast CMEs. Following that work, one of us (Thompson) created a new CME detection algorithm adapted from the Solar Eruptive Event Detection System (SEEDS) code for use with K-Cor observations from the Mauna Loa Solar Observatory (MLSO) in Hawaii. We develop performance metrics and report on the success of the algorithm to detect CMEs in the 2017 K-Cor observations. Measures of success include the ability of the algorithm to detect an event and the amount of time between the event onset and its detection. The algorithm successfully detected 20 of the 35 CMEs identified between 1 Jan and 31 August, 2017 in the K-Cor data. There were 10 false positive events during this time period. The threshold for CME detection is discussed as a function of CME visibility, instrument background, and sky noise. The code has been modified to run in an automated mode and is in the process of being integrated into the real-time data processing pipeline at Mauna Loa. We report on current status, real-time alerts, and future upgrades.

Here’s a tutorial of how to make an animation of the moon’s shadow with GOES imagery during The Great American Eclipse of 2017:

Here is one of the coolest examples that I have created using IDL in a while. For this blog post, I’m going to walk through how I created an animation of the Moon’s shadow during the Great American Total Solar Eclipse using several different technologies for accessing, downloading, and visualizing the data.

The video is on Harris Geospatial Solutions’ Facebook page.

IDL 8.6.1 was released today^[1]. Some interesting new features:

• Conditional breakpoints from the Workbench
• Hexadecimal constants, e.g., a = 0xFF3A
• Fix for strings that begin with numerals being confused with the octal notation: "123 is an octal value; "123" used to be a syntax error, but is now a valid string.

See the release notes for details.