Software executable dry run

Developers covering multiple platforms and archs can benefit from including a self-contained dry run. We define a software dry run as a fast self-contained run of the executable, exercising most or all of the program using actual input files. The concept of dry run is used by popular programs that rely on several components and connections including rsync.

Benefits

A dry run self-check can be used from Python or any other script calling the executable to ensure the binary is compatible with the current platform environment. The dry run helps mitigate confusing error messages by checking that the executable runs on the platform before making a large program run.

The dry run can catch platform-specific issues like:

  • incompatible executable format (running a executable built for another platform)
  • executable built for incompatible arch (using CPU feature not available on this platform)
  • shared library (DLL) path / arch issues

Implementation

The dry run does not output any files besides temporary files. For example, in a simulation, the dry run might run one complete time step. To test file I/O, optionally write temporary file(s) using the same file format. An advanced dry run might read in those temporary files and do a basic sanity check.

By our definition, a dry run is distinct from an integration test. A dry run of the program just checks that the platform environment is OK to run with this binary. The dry run checks simply that the code executes without crashing. The dry run does not emphasize deep checks of program output as an integration test would. Consider making the dry run return code be 0 for compatibility with CMake and other high level build systems.

CMake dry run test

Assuming you have configured the project executable code as above, implement a check of the dry run with CMake.

project(Foo LANGUAGES C)
enable_testing()

add_executable(foo foo.c)

add_test(NAME check_foo COMMAND $<TARGET_FILE:foo> -dryrun <other command line flags>)
set_tests_properties(check_foo PROPERTIES PASS_REGULAR_EXPRESSION "OK: myprogram")

Here we make optional use of PASS_REGULAR_EXPRESSION to verify the special dry run text you put in the executable code. The dry run test should have return code zero.