iTerm2 is a macOS terminal emulator with a lot of extra features. In particular, it has a simple protocol for displaying images inline. It comes with a program imgcat that will display common image formats such as PNG, JPEG, GIF, etc. Most of the images I deal with are FITS, though. I wrote fitscat to be a handy utility to display FITS images, as well as to print basic information about the FITS file such as a listing of extensions or an extension header.

For example, fitscat can display an image in an extension, as seen below:

There are options to specify a minimum and maximum value for scaling, as well as to use a simple filter such square root.

Also, fitscat can also print basic information about a FITS file, such as a listing of extensions:

CoMP$ fitscat --list 20150624.170419.comp.1074.iqu.5.fts
Filename: 20150624.170419.comp.1074.iqu.5.fts
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      67   ()
  1  I, 1074.38    1 ImageHDU        33   (620, 620)   float32
  2  I, 1074.50    1 ImageHDU        33   (620, 620)   float32
  3  I, 1074.62    1 ImageHDU        33   (620, 620)   float32
  4  I, 1074.74    1 ImageHDU        33   (620, 620)   float32
  5  I, 1074.86    1 ImageHDU        33   (620, 620)   float32
  6  Q, 1074.38    1 ImageHDU        33   (620, 620)   float32
  7  Q, 1074.50    1 ImageHDU        33   (620, 620)   float32
  8  Q, 1074.62    1 ImageHDU        33   (620, 620)   float32
  9  Q, 1074.74    1 ImageHDU        33   (620, 620)   float32
 10  Q, 1074.86    1 ImageHDU        33   (620, 620)   float32
 11  U, 1074.38    1 ImageHDU        33   (620, 620)   float32
 12  U, 1074.50    1 ImageHDU        33   (620, 620)   float32
 13  U, 1074.62    1 ImageHDU        33   (620, 620)   float32
 14  U, 1074.74    1 ImageHDU        33   (620, 620)   float32
 15  U, 1074.86    1 ImageHDU        33   (620, 620)   float32

Or display a header:

CoMP$ fitscat --header -e 3 20150624.170419.comp.1074.iqu.5.fts
XTENSION= 'IMAGE   '           /extension type
BITPIX  =                  -32 /bits per data value
NAXIS   =                    2 /number of axes
NAXIS1  =                  620 /
NAXIS2  =                  620 /
PCOUNT  =                    0 /
GCOUNT  =                    1 /
EXTNAME = 'I, 1074.62'         /
WAVELENG=             1074.620 / WAVELENGTH OF OBS (NM)
POLSTATE= 'I       '           / POLARIZATION STATE
EXPOSURE=               250.00 / EXPOSURE TIME (MILLISEC)
NAVERAGE=                   16 / Number of images averaged together
FILTER  =                    1 / FILTER WHEEL POSITION (1-8)
DATATYPE=               'DATA' / DATA, DARK OR FLAT
LCVR1TMP=            29.639999 / DEGREES CELSIUS
LCVR2TMP=            33.429001 /
LCVR3TMP=            33.715000 /
LCVR4TMP=            33.738998 /
LCVR5TMP=            33.618999 /
LCVR6TMP=            28.847000 /
NDFILTER=                    8 / ND 1=.1, 2=.3, 3=.5, 4=1, 5=2, 6=3, 7=4, 8=cle
BACKGRND=               13.154 / Median of masked line center background
BODYTEMP=               34.023 / TEMPERATURE OF FILTER BODY (C)
BASETEMP=               33.599 / BASE PLATE TEMP (C)
RACKTEMP=               25.012 / COMPUTER RACK AMBIENT AIR TEMP (C)
OPTRTEMP=               33.306 / OPTICAL RAIL TEMP (C)
DEMULT  =                    1 / 1=DEMULTIPLEXED, 0=NOT DEMULTIPLEXED
FILTTEMP=               35.000 / ILX FILTER TEMPERATURE (C)
FLATFILE= '20150624.070023.FTS' / Name of flat field file
INHERIT =                    T /
DISPMIN =                 0.00 / Minimum data value
DISPMAX =                 5.00 / Maximum data value
DISPEXP =                 0.50 / Exponent value for scaling

The full interface of fitscat is shown below:

$ fitscat --help
usage: fitscat [-h] [--min MIN] [--max MAX] [--debug] [-d] [-l] [-r]
               [-e EXTEN_NO] [-f FILTER] [-s SLICE]
               filename

fitscat - a FITS query/display program

positional arguments:
  filename              FITS file to query

optional arguments:
  -h, --help            show this help message and exit
  --min MIN             min for scaling
  --max MAX             max for scaling
  --debug               set to debug
  -d, --display         set to display
  -l, --list            set to list HDUs
  -r, --header          set to display header
  -e EXTEN_NO, --exten_no EXTEN_NO
                        specify extension
  -f FILTER, --filter FILTER
                        specify filter (default: none)
  -s SLICE, --slice SLICE
                        specify slice of data array to display

Source code for the Python script is available on GitHub. The script is compatible with Python 2 and 3, but requires standard scientific Python packages AstroPy, NumPy, and PIL.

NIST has published new standards for digital identifies. Highlights, via Bruce Schneier, for passwords:

  1. No password rules! Use pass phrases.
  2. Don’t expire passwords.
  3. Allow password managers.

I have written about this before, where I said my personal pet peeve was forced password expiration (#2). I hope organizations start using the new standards quickly!

I presented a poster at a Space Weather workshop at the Lorentz Center in Leiden, Netherlands last week:

Real-time automated detection of coronal mass ejections using ground-based coronagraph instruments

Coronal mass ejections (CMEs) are dynamic events that eject magnetized plasma from the Sun’s corona into interplanetary space. CMEs are a major driver of solar energetic particle (SEP) events and geomagnetic storms. SEP events and geomagnetic storms pose hazards to astronauts, satellites, communication systems, and power grids. Understanding CME formation and predicting their impacts at Earth are primary goals of the National Space Weather program. St. Cyr et al. (2017) reported on the use of near real-time white light observations of the low corona from the COSMO K-Coronagraph (K- Cor) to provide an early warning of possible SEP events driven by fast CMEs. Following that work, one of us (Thompson) created a new CME detection algorithm adapted from the Solar Eruptive Event Detection System (SEEDS) code for use with K-Cor observations from the Mauna Loa Solar Observatory (MLSO) in Hawaii. We develop performance metrics and report on the success of the algorithm to detect CMEs in the 2017 K-Cor observations. Measures of success include the ability of the algorithm to detect an event and the amount of time between the event onset and its detection. The algorithm successfully detected 20 of the 35 CMEs identified between 1 Jan and 31 August, 2017 in the K-Cor data. There were 10 false positive events during this time period. The threshold for CME detection is discussed as a function of CME visibility, instrument background, and sky noise. The code has been modified to run in an automated mode and is in the process of being integrated into the real-time data processing pipeline at Mauna Loa. We report on current status, real-time alerts, and future upgrades.

Here’s a tutorial of how to make an animation of the moon’s shadow with GOES imagery during The Great American Eclipse of 2017:

Here is one of the coolest examples that I have created using IDL in a while. For this blog post, I’m going to walk through how I created an animation of the Moon’s shadow during the Great American Total Solar Eclipse using several different technologies for accessing, downloading, and visualizing the data.

The video is on Harris Geospatial Solutions’ Facebook page.

The dataviz.tools site is an annotated and categorized catalog of good visualization tools.

This site features a curated selection of data visualization tools meant to bridge the gap between programmers/statisticians and the general public by only highlighting free/freemium, responsive and relatively simple-to-learn technologies for displaying both basic and complex, multivariate datasets.

via FlowingData

Some great tips for spotting misleading visualizations:

By using dual axes, the magnitude can shrink or expand for each metric. This is typically done to imply correlation and causation. “Because of this, this other thing happened. See, it’s clear.”

There are some great links as examples of these problems, like the spurious correlations project by Tyler Vigen to automatically find correlations.

I think “machine learning” in this paper applies fairly well to any type of scientific pipeline code:

Using the framework of technical debt, we note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning.

The authors argue that machine learning systems have the regular issues of a code, but also have other complexities that are not necessary addressed in the normal way of refactoring libraries, adding unit tests, etc.

IDL 8.6.1 was released today1. Some interesting new features:

  • Conditional breakpoints from the Workbench
  • Hexadecimal constants, e.g., a = 0xFF3A
  • Fix for strings that begin with numerals being confused with the octal notation: "123 is an octal value; "123" used to be a syntax error, but is now a valid string.

See the release notes for details.


  1. Really sometime in the last week or so. The announcement on the newsgroup was today, but the release notes was posted 7/27. 

Travis Oliphant, creator of NumPy the array package for Python, wrote a analog to the Zen of Python for NumPy:

Strided is better than scattered
Contiguous is better than strided
Descriptive is better than imperative (use data-types)
Array-oriented is often better than object-oriented
Broadcasting is a great idea — use where possible
Vectorized is better than an explicit loop
Unless it’s complicated — then use numexpr, weave, or Cython
Think in higher dimensions

I tried something for IDL last year.

I will be posting more about the Great American Eclipse of 2017 this summer. To start off with, NASA has made some great maps of where the total eclipse will be visible from:

On August 21, 2017, the moon will pass between Earth and the sun in a total solar eclipse that will be visible on a path from Oregon to South Carolina across the continental United States. This path of totality will occur in a little over 90 minutes, while observers on the ground will see the eclipse for about two and a half minutes. Standing at the edge of the moon’s shadow, or umbra, the difference between seeing a total eclipse and a partial eclipse comes down to elevation – mountains and valleys both on Earth and on the moon – which affect where the shadow lands. In this visualization, data from NASA’s Lunar Reconnaissance Orbiter account for the moon’s terrain that creates a jagged edge on its shadow. This data is then combined with elevation data on Earth as well as information on the sun angle to create the most accurate map of the eclipse path to date. Watch the video to learn more.

NASA also has downloadable, detailed state maps of the eclipse path.

via kottke

older posts »