HoloViews is yet another Python visualization library, but it has a different approach:

With HoloViews, instead of building a plot using direct calls to a plotting library, you first describe your data with a small amount of crucial semantic information required to make it visualizable, then you specify additional metadata as needed to determine more detailed aspects of your visualization. This approach provides immediate, automatic visualization that can be effortlessly requested at any time as your data evolves, rendered automatically by one of the supported plotting libraries (such as Bokeh or Matplotlib).

It can produce a variety of visualizations — check out its gallery to see more examples.

iTerm2 is a macOS terminal emulator with a lot of extra features. In particular, it has a simple protocol for displaying images inline. It comes with a program imgcat that will display common image formats such as PNG, JPEG, GIF, etc. Most of the images I deal with are FITS, though. I wrote fitscat to be a handy utility to display FITS images, as well as to print basic information about the FITS file such as a listing of extensions or an extension header.

For example, fitscat can display an image in an extension, as seen below:

There are options to specify a minimum and maximum value for scaling, as well as to use a simple filter such square root.

Also, fitscat can also print basic information about a FITS file, such as a listing of extensions:

CoMP$ fitscat --list 20150624.170419.comp.1074.iqu.5.fts
Filename: 20150624.170419.comp.1074.iqu.5.fts
No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 67 ()
1 I, 1074.38 1 ImageHDU 33 (620, 620) float32
2 I, 1074.50 1 ImageHDU 33 (620, 620) float32
3 I, 1074.62 1 ImageHDU 33 (620, 620) float32
4 I, 1074.74 1 ImageHDU 33 (620, 620) float32
5 I, 1074.86 1 ImageHDU 33 (620, 620) float32
6 Q, 1074.38 1 ImageHDU 33 (620, 620) float32
7 Q, 1074.50 1 ImageHDU 33 (620, 620) float32
8 Q, 1074.62 1 ImageHDU 33 (620, 620) float32
9 Q, 1074.74 1 ImageHDU 33 (620, 620) float32
10 Q, 1074.86 1 ImageHDU 33 (620, 620) float32
11 U, 1074.38 1 ImageHDU 33 (620, 620) float32
12 U, 1074.50 1 ImageHDU 33 (620, 620) float32
13 U, 1074.62 1 ImageHDU 33 (620, 620) float32
14 U, 1074.74 1 ImageHDU 33 (620, 620) float32
15 U, 1074.86 1 ImageHDU 33 (620, 620) float32

Or display a header:

CoMP$ fitscat --header -e 3 20150624.170419.comp.1074.iqu.5.fts
XTENSION= 'IMAGE ' /extension type
BITPIX = -32 /bits per data value
NAXIS = 2 /number of axes
NAXIS1 = 620 /
NAXIS2 = 620 /
PCOUNT = 0 /
GCOUNT = 1 /
EXTNAME = 'I, 1074.62' /
WAVELENG= 1074.620 / WAVELENGTH OF OBS (NM)
POLSTATE= 'I ' / POLARIZATION STATE
EXPOSURE= 250.00 / EXPOSURE TIME (MILLISEC)
NAVERAGE= 16 / Number of images averaged together
FILTER = 1 / FILTER WHEEL POSITION (1-8)
DATATYPE= 'DATA' / DATA, DARK OR FLAT
LCVR1TMP= 29.639999 / DEGREES CELSIUS
LCVR2TMP= 33.429001 /
LCVR3TMP= 33.715000 /
LCVR4TMP= 33.738998 /
LCVR5TMP= 33.618999 /
LCVR6TMP= 28.847000 /
NDFILTER= 8 / ND 1=.1, 2=.3, 3=.5, 4=1, 5=2, 6=3, 7=4, 8=cle
BACKGRND= 13.154 / Median of masked line center background
BODYTEMP= 34.023 / TEMPERATURE OF FILTER BODY ©
BASETEMP= 33.599 / BASE PLATE TEMP ©
RACKTEMP= 25.012 / COMPUTER RACK AMBIENT AIR TEMP ©
OPTRTEMP= 33.306 / OPTICAL RAIL TEMP ©
DEMULT = 1 / 1=DEMULTIPLEXED, 0=NOT DEMULTIPLEXED
FILTTEMP= 35.000 / ILX FILTER TEMPERATURE ©
FLATFILE= '20150624.070023.FTS' / Name of flat field file
INHERIT = T /
DISPMIN = 0.00 / Minimum data value
DISPMAX = 5.00 / Maximum data value
DISPEXP = 0.50 / Exponent value for scaling

The full interface of fitscat is shown below:

usage: fitscat [-h] [--min MIN] [--max MAX] [--debug] [-d] [--resize-factor RESIZE_FACTOR] [-l]
               [-r] [-e EXTEN_NO] [-f FILTER] [-s SLICE] [--validate]
               filename

fitscat - a FITS query/display program

positional arguments:
  filename              FITS file to query

options:
  -h, --help            show this help message and exit
  --min MIN             min for scaling
  --max MAX             max for scaling
  --debug               set to debug
  -d, --display         set to display
  --resize-factor RESIZE_FACTOR
                        resize by factor
  -l, --list            set to list HDUs
  -r, --header          set to display header
  -e EXTEN_NO, --exten_no EXTEN_NO
                        specify extension
  -f FILTER, --filter FILTER
                        specify filter (default: none)
  -s SLICE, --slice SLICE
                        specify slice of data array to display
  --validate            check keywords against FITS standard

Source code for the Python script is available on GitHub. The script is compatible with Python 2 and 3, but requires standard scientific Python packages AstroPy, NumPy, and PIL.

NIST has published new standards for digital identifies. Highlights, via Bruce Schneier, for passwords:

1. No password rules! Use pass phrases.
2. Don’t expire passwords.
3. Allow password managers.

I have written about this before, where I said my personal pet peeve was forced password expiration (#2). I hope organizations start using the new standards quickly!

I presented a poster at a Space Weather workshop at the Lorentz Center in Leiden, Netherlands last week:

Real-time automated detection of coronal mass ejections using ground-based coronagraph instruments

Coronal mass ejections (CMEs) are dynamic events that eject magnetized plasma from the Sun’s corona into interplanetary space. CMEs are a major driver of solar energetic particle (SEP) events and geomagnetic storms. SEP events and geomagnetic storms pose hazards to astronauts, satellites, communication systems, and power grids. Understanding CME formation and predicting their impacts at Earth are primary goals of the National Space Weather program. St. Cyr et al. (2017) reported on the use of near real-time white light observations of the low corona from the COSMO K-Coronagraph (K- Cor) to provide an early warning of possible SEP events driven by fast CMEs. Following that work, one of us (Thompson) created a new CME detection algorithm adapted from the Solar Eruptive Event Detection System (SEEDS) code for use with K-Cor observations from the Mauna Loa Solar Observatory (MLSO) in Hawaii. We develop performance metrics and report on the success of the algorithm to detect CMEs in the 2017 K-Cor observations. Measures of success include the ability of the algorithm to detect an event and the amount of time between the event onset and its detection. The algorithm successfully detected 20 of the 35 CMEs identified between 1 Jan and 31 August, 2017 in the K-Cor data. There were 10 false positive events during this time period. The threshold for CME detection is discussed as a function of CME visibility, instrument background, and sky noise. The code has been modified to run in an automated mode and is in the process of being integrated into the real-time data processing pipeline at Mauna Loa. We report on current status, real-time alerts, and future upgrades.

Here’s a tutorial of how to make an animation of the moon’s shadow with GOES imagery during The Great American Eclipse of 2017:

Here is one of the coolest examples that I have created using IDL in a while. For this blog post, I’m going to walk through how I created an animation of the Moon’s shadow during the Great American Total Solar Eclipse using several different technologies for accessing, downloading, and visualizing the data.

The video is on Harris Geospatial Solutions’ Facebook page.

The dataviz.tools site is an annotated and categorized catalog of good visualization tools.

This site features a curated selection of data visualization tools meant to bridge the gap between programmers/statisticians and the general public by only highlighting free/freemium, responsive and relatively simple-to-learn technologies for displaying both basic and complex, multivariate datasets.

via FlowingData

Some great tips for spotting misleading visualizations:

By using dual axes, the magnitude can shrink or expand for each metric. This is typically done to imply correlation and causation. “Because of this, this other thing happened. See, it’s clear.”

There are some great links as examples of these problems, like the spurious correlations project by Tyler Vigen to automatically find correlations.

I think “machine learning” in this paper applies fairly well to any type of scientific pipeline code:

Using the framework of technical debt, we note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning.

The authors argue that machine learning systems have the regular issues of a code, but also have other complexities that are not necessary addressed in the normal way of refactoring libraries, adding unit tests, etc.

IDL 8.6.1 was released today^[1]. Some interesting new features:

• Conditional breakpoints from the Workbench
• Hexadecimal constants, e.g., a = 0xFF3A
• Fix for strings that begin with numerals being confused with the octal notation: "123 is an octal value; "123" used to be a syntax error, but is now a valid string.

See the release notes for details.