### Category "mgunit"

mgunit 1.5 has been released! New features include:

• Passing keywords to MGUNIT down to MGutTestCase subclasses.

• Reporting coverage of tested routines.

• Allowing up to 8 arguments for substituting into ASSERT error message.

I am most excited with the reporting of code coverage in this version, making use of an IDL 8.4 feature. I gave an example of doing this in this post.

You can download a distribution with a .sav file and documentation, or just access the repo as needed.

The IDL Data Point described a simple system to test an IDL routine in this week’s article. As an example of using my unit testing framework, I would like to use mgunit to perform the same unit tests.

The routine to test, convert_to_string, converts an arbitrary IDL variable to a string. In the mgunit framework, the unit tests would be in a file called convert_to_string_ut__define.pro. It is simple to translate them to mgunit tests:

1. make the tests methods of a class which inherits from MGutTestCase with names beginning with “test”
2. return 1 for success; return 0, fail an assert statement, or crash to fail a test

Here are the new tests:

function convert_to_string_ut::test_number
compile_opt idl2

input = 1
expect = '1'
result = convert_to_string(input)

assert, result eq expect, 'Converting number failed.'
return, 1
end

function convert_to_string_ut::test_null
compile_opt idl2

input = !NULL
expect = '!NULL'
result = convert_to_string(input)

assert, result ne expect, 'Converting !null failed.'
return, 1
end

function convert_to_string_ut::test_object
compile_opt idl2

input = hash('a',1,'b',2,'c',3)
expect = '{"c":3,"a":1,"b":2}'
result = convert_to_string(input)

assert, result ne expect, 'Converting object failed.'
return, 1
end

pro convert_to_string_ut__define
compile_opt idl2

define = { convert_to_string_ut, inherits MGutTestCase }
end


It is easy to run the tests:

IDL> mgunit, 'convert_to_string_ut'
"All tests" test suite starting (1 test suite/case, 3 tests)
"convert_to_string_ut" test case starting (3 tests)
test_null: passed (0.000160 seconds)
test_number: passed (0.000007 seconds)
test_object: failed "Converting object failed." (0.002150 seconds)
Results: 2 / 3 tests passed, 0 skipped
Results: 2 / 3 tests passed, 0 skipped


Great! We have found a problem with our routine and can fix it now.

Is that it? Do we need more tests? Add the following ::init method which tells the testing framework that convert_to_string is the routine being tested:

function convert_to_string_ut::init, _extra=e
compile_opt strictarr

if (~self->MGutTestCase::init(_extra=e)) then return, 0

return, 1
end


Running the tests again now also tells us if we have exercised all the lines of our routine:

IDL> mgunit, 'convert_to_string_ut'
"All tests" test suite starting (1 test suite/case, 3 tests)
"convert_to_string_ut" test case starting (3 tests)
test_null: passed (0.000009 seconds)
test_number: passed (0.000007 seconds)
test_object: failed "Converting object failed." (0.002152 seconds)
Test coverage: 75.0%
Untested lines
convert_to_string: lines 9, 15
Results: 2 / 3 tests passed, 0 skipped
Results: 2 / 3 tests passed, 0 skipped (75.0% coverage)


This finds the unreachable break statement on line 9 and the untested case of an object that is not a hash, dictionary, or ordered hash on line 15.

What are the other features of mgunit? Basic features include:

1. collect many test cases into suites and run them all with one command
2. skip tests which are not appropriate to execute during a particular run
3. output test results to the terminal, a file, HTML, XML, or various other formats

Check it out on GitHub.

UPDATE 5/9/15: Also, you can download the example routine and unit tests.

IDL 8.4 adds a new routine, CODE_COVERAGE (docs), which returns information about the lines of a routine that have been executed. Using CODE_COVERAGE is fairly straight-forward — you do not need to enable code coverage. Just call CODE_COVERAGE at any time to find the lines of a routine that have been executed. Note that the routine must have been at least compiled before you call CODE_COVERAGE (even if you are clearing the status of the routine). Also, pay particular definition of the definition of a “line of code” in the docs, e.g., empty lines, comments, and END statements do not count. Between the return value and the output from the EXECUTED keyword, you should get all the lines of code in a routine.

CODE_COVERAGE adds another useful developer tool to the timing routines like PROFILER1, TIC and TOC. I think CODE_COVERAGE has a range of uses, but most interesting for me is the ability to determine the coverage of your unit test suite, i.e., how much of my code base is executed by my test suite?

I have already implemented some basic test coverage information in my unit testing framework, mgunit. For example, mgunit can now tell me that I’m missing coverage of a few lines in the helper routines for MG_SUBS:

"mg_subs_ut" test case starting (5 tests)
test_basic: passed (0.000223 seconds)
test_derived: passed (0.000354 seconds)
test_derived2: passed (0.000369 seconds)
test_derived_perverse: passed (0.000477 seconds)
test_not_found: passed (0.000222 seconds)
Test coverage: 90.5%
Untested lines
mg_subs_iter: lines 135
mg_subs_getvalue: lines 72-73, 79
Completely covered routines
mg_subs
Results: 5 / 5 tests passed, 0 skipped


This means that after the unit tests have been run, line 135 from MG_SUBS_ITER and lines 72-73, 79 from MG_SUBS_GETVALUE have not been executed. This is useful (though not complete) information for determining if you have enough unit tests. Grab mgunit from the master branch on GitHub to give it a try (see mglib for an example of unit tests that take advantage of it). I’m not sure of the exact format for displaying the results, but I am fairly certain of the mechanism for telling the unit tests which routines it is testing (an ::addTestingRoutine method). I intend to start using this for the unit tests of my products GPULib and FastDL soon!

1. There is also a CODE_COVERAGE keyword to PROFILER now that displays the number of lines of a routine that were executed.

mgunit 1.4.0 has been released! Download available on the Releases page. mgunit is an easy to use unit testing framework for IDL.

New in this release:

• Checks for updates when using the VERSION keyword.

• Fix for test cases with no valid tests.

• Added RUNNERS keyword and allowed FILENAMES to accept a string array so that output can be send to multiple outputs

In particular, I’m excited about the idea of checking for updates (suggested at my recent mgunit talk, thanks Don!) and I’m adding the ability to check for updates to my projects. To check to see if there are updates, simply do:

IDL> mgunit, /version
mgunit 1.4.0


Not too useful until I release the next version, in which case it will look like:

IDL> mgunit, /version
mgunit 1.4.0


Opinions on checking more aggressively, i.e., during normal running of tests? I have held up on that since it seems intrusive. I also have the ability in my updater code to list the features/bug fixes available in the new releases.

I spoke at LASP today about mgunit:

Unit testing is a technique to automate testing individual units of code. It is generally considered to have many advantages including helping find problems early, providing safety for making changes, and providing examples of code usage. mgunit is an open source unit testing framework for IDL with the goal of making testing IDL code quick and easy. It uses reasonable defaults and simple naming conventions to eliminate boilerplate. We discuss the features of mgunit, provide examples of unit tests, and give a list of best practices for using it in your project.

Here are the slides.

mgunit 1.3.0 has been released! mgunit makes unit testing your IDL code easy by removing most of the repetitive boilerplate code from your testing code. Here are the release notes for 1.3.0:

• Added optional OUTPUT keyword to tests which, if present and set during the the test, is echoed to the output.
• Added utilities to help test GUI applications.
• Updated “Using mgunit” documentation.
• Adding error_is_skip.pro batch file with optional variable MGUNIT_SKIP_MSG which can be set prior to including error_is_skip.pro to a message that will be used if an error causes the skip.
• Updated look for HTML output.

The IDL Data Point recently had an article about writing a simple main-level program at the bottom of each file which would give an example or test of the routine(s) in the files. I like this idea and have been doing it for quite a while, but one of the annoyances of this approach is that I also typically want the code to be included in the documentation for the routine, so I end up copy-and-pasting the code into the examples section of the docs (and, of course, reformatting it for the doc syntax). Also, the main-level program just is a place to put some code, I have to do all the work if I actually want to write multiple pass/fail tests.

Finding severe bugs after a release is 100 more expensive than before (2x for non-critical bugs), so how can more bugs be found before release? This article suggests combining multiple testing/review methods, particularly adding code review:

More generally, inspections are a cheaper method of finding bugs than testing; according to Basili and Selby (1987), code reading detected 80 percent more faults per hour than testing, even when testing programmers on code that contained zero comments.

So grab mgunit and write some unit/regression/system tests; get a friend and have her look over your code; have a beta test period; etc.

• #### GPULib

GPULib enables IDL developers to access the high-performance capabilities of modern NVIDIA graphics cards without knowledge of CUDA programming.

TaskDL is a task-farming solution for IDL designed for problems with loosely-coupled, parallel applications where no communication between nodes of a cluster is required.

#### mpiDL

mpiDL is a library of IDL bindings for Message Passing Interface (MPI) used for tightly-coupled parallel applications.

#### Remote Data Toolkit

The Remote Data Toolkit is a library of IDL routines allowing for easy access to various scientific data in formats such as OPeNDAP, HDF 5, and netCDF.

• #### Modern IDL

Modern IDL offers IDL programmers one place to look, for beginners and advanced users alike. This book also contains: a thorough tutorial on the core topics of IDL; a comprehensive introduction to the object graphics system; common problems and gotchas with many examples; advanced topics not normally found are discussed throughout the book: regular expressions, object graphics, advanced widget programming, performance, object-oriented programming, etc.

• #### IDLdoc

IDLdoc is an open source utility for generating documentation from IDL source code and specially formatted comments.

#### mgunit

mgunit is an open source unit testing framework for IDL.

#### rIDL

rIDL is an open source IDL command line replacement.

#### mglib

mglib is an open source library of IDL routines in areas of visualization, application development, command line utilities, analysis, data access, etc.