Sometimes it is useful to selectively run tests from a suite of mgunit tests. For example, in the GPULib unit test suite, certain tests required hardware capable of performing them, e.g., double precision computations, streaming, etc. There is a SKIP keyword to ASSERT to handle these situations. For example, a GPULib unit test requiring double precision hardware might do something like:

assert, gpuDoubleCapable(), 'CUDA device not double capable', /skip

This would skip the test, so the test would not count as either passed or failed. Here, gpuDoubleCapable can perform a check to determine whether the hardware is double capable.

But this requires some type of global setting to be checked. It would be useful to be able pass arguments to mgunit when starting the tests that could be checked during a test. I’ve added this feature to the master branch of mgunit.

To use this, define a super class for your tests that inherits from MGutTestCase which accepts a keyword for your property, say MY_PROPERTY, in its ::init method. Store that value in some manner, probably as an instance variable of your object, so that you can check it in your test. Then call mgunit like this:

mgunit, 'my_tests_uts', MY_PROPERTY=1

This provides a convenient way to run your tests in various modes, skip certain tests, or pass other information to your tests.

This feature is only available in the master branch of mgunit right now, but should be in the next release.

Recently, I have been working on a data pipeline that updates a MySQL database. We have been using Java simply to access the database and I thought I would simplify our toolchain by removing the dependency on Java by updating the database directly from IDL where we already have all the data that is needed to populate the tables.

So I have been adding MySQL bindings to mglib in the last week. I have also added a higher-level interface to the straight wrappers of the MySQL C routines.

Continue reading “MySQL bindings.”

I sold the last print copy of Modern IDL that I had this morning. At this time, I do not intend to make another print run; the PDF will be the only version of the book from now on1.

While I continually update Modern IDL for new versions of IDL, only the PDF purchasers get the full benefit of the updates. The hardcopy version was often a version of two behind because of the way that I placed orders to my printer. PDF purchasers also got future updates2, which was, of course, impossible for the hardcopy version.

For those who still really want a hardcopy version, you have permission to print your PDF. While higher, I think the total cost of the PDF plus printing/binding at Kinkos shouldn’t be too much more than I was charging for the hardcopy.


  1. Although, I don’t have anything against other electronic formats. If you have a favorite electronic format that you would like to see Modern IDL available in, please let me know. 

  2. PDF purchasers will receive updates until IDL 9.0, when you will have to purchase again to receive further updates. 

I have long implored others to abandon the use of common blocks for a myriad of reasons. They create a web of interconnections between the routines that use the common block (and which routines, exactly, are those? You can find out by grepping, but it is not immediately obvious). It is not clear at all which routine is responsible for changing the value of some variable stored in a common block. It is easy for the name of a variable intended to be local to clash with the name of a common block variable without realizing it.

But, sometimes, they provide a nice, quick solution for a problem that would otherwise require building a lot of infrastructure. I have been doing various types of profiling of a code recently. After finding the “hot” routines using PROFILER, I have been trying to narrow down to (and quantify the runtime of) the particular lines using up an appreciable amount of time. The problem is that I want the cumulative runtime for the lines over the course of executing many times during a run.

So I created a “profiling” common block with a time variable, initialized at the beginning of the program, which is incremented after the lines run each time. This can be reported at the end of the program. I have been moving the common block around to various routines to get times until I found the “hot” lines within the “hot” routines.

Full disclosure: this is not the only place I have used common blocks. MG_LOG, a routine that I use a lot, has a common block.

After spending awhile last Friday trying to vectorize a loop of a small matrix-vector multiplication for every pixel of an image, I gave up and decided to just write it as a DLM. For my image sizes of 1024 by 1024 pixels (actually two images of that size), the run time went from 3.15 seconds to 0.26 seconds on my MacBook Pro. That’s not a lot of time to save, but since we acquire imagery every 15 seconds, it was useful.

Check out analysis.c for source code. There are also unit tests showing how to use it.

The “Zen of Python” provides the basic philosophy of Python. From PEP 20:

There should be one — and preferably only one — obvious way to do it.

I doubt there is a single area in Python that violates this more than making a simple HTTP request. Python provides at least six builtin libraries to do this: httplib, httplib2, urllib, urllib2, urllib3, and pycurl. There are several reviews comparing the various libraries.

But there is a third party library, requests, that might be the “obvious way to do it” now:

import requests, json

url = 'https://api.github.com/repos/mgalloy/mglib'
r = requests.get(url, auth=('mgalloy', 'my_password'))
print r.json()['updated_at']

requests is installed as part of Anaconda, which an easy way to get all the core scientific programming packages for Python.

After nearly six and a half years, today is my last day at Tech-X. Yesterday, I finished all my tasks for the GPULib 1.8 release and the transitioning to the new GPULib product manager.

GPULib will continue on at Tech-X. I will continue to use it and contribute to it. I will also help Tech-X support the product and do my best to make the transition seamless for GPULib users.

I am excited to start my next job at NCAR, but equally excited to have a couple weeks between jobs to explore other things.

GPULib 1.8 has been released with updates to the underlying libraries as well as many other features in many areas of the library. It has been updated to use the most recent versions of IDL and CUDA, IDL 8.4 and CUDA 6.5. The new features are:

  • Support for integer data types. I have been wanting to support integer types in GPULib for awhile and now GPULib supports all the numeric types provided by IDL! We can finally do:

    dx = gpuIndgen(10)
    
  • Added GPUREPMAT routine. This is a handy routine to create a new array by repeating a 2-dimensional array in a grid.

  • Added GPUCREATEKERNEL routine to create the source code of a simple kernel. This is a code generation routine that can be loaded with GPULOADMODULE/GPULOADFUNCTION and executed with GPUEXECUTEFUNCTION.

  • Added GPUFINITE routine similar to IDL’s library routine.

  • Added linear algebra routines GPULUDC, GPULUSOL, and GPULEAST_SQUARES. This fills out more of the GPU equivalent of the convenience routines provided by IDL so that the LAPACK interface of MAGMA is not required to perform linear algebra computations.

  • Added support for RHO and THETA keywords in GPURADON.

  • Added GPUMEAN routine. This routine has DIMENSION and NAN keywords with the same functionality as IDL’s library routine.

Full disclosure: I work for Tech-X and I am the product manager/developer for GPULib.

I’ll be at the AGU Fall Meeting this year presenting work about SatelliteDL:

SatelliteDL is an IDL toolkit for the analysis of satellite Earth observations from a diverse set of platforms and sensors. The core function of the toolkit is the spatial and temporal alignment of satellite swath and geostationary data.

It’s in the Informatics session on Tuesday afternoon, IN23D-3748. Check out the poster!

The biggest change introduced by IDL 8.4 is the treatment of all variables as objects. Every variable now has attributes that you would normally get returned from the SIZE function: length, ndim, dim, tname, typecode, and typename. For example:

IDL> a = bindgen(2, 3)
IDL> print, a.length
           6
IDL> print, a.ndim
           2
IDL> print, a.dim
           2           3
IDL> print, a.tname
BYTE

There are also static methods available for all variables:

IDL> n = 50
IDL> print, n->toString()
50

Strings, numbers, integers, and pointers have their own set of special methods appropriate for their respective types. For example integers have some handle base conversion methods:

IDL> print, n->toHex()
32
IDL> print, n->toBinary()
110010

Strings have some methods that are not available through other IDL library routines, including the very useful replace method:

IDL> s = 'ITT VIS'
IDL> print, s.replace('ITT', 'Exelis')
Exelis VIS

older posts »