Cross platform unit testing with Wine and Autotools

My current project targets the Windows and Linux operating systems. Development is primarily under Linux and uses the Autotools build system. Windows binaries are generated with the mingw cross compilers. This setup isn’t for everyone, but it suits me quite well.

For my testing needs I use Automake’s simple builtin test harness and the simple TAP driver. Automake will add Makefile targets for each executable listed in TESTS. The LOG_DRIVER command will be used to generate test logs and test statuses using any LOG_COMPILER wrapper provided. They can be collectively run with make check (and prior to distribution with make distcheck).

Unfortunately, directly executing cross-compiled binaries is not terrifically successful in my development environment. The immediate and `simple’ answer to this issue is to just use an emulation layer like Wine to run the binaries on Automake’s behalf.

Custom LOG_COMPILER

My first attempt at this technique used the per-file-extension LOG_COMPILER feature of Automake. Each file-extension can have its own LOG_COMPILER which will run the test executable.

PY_LOG_COMPILER=python
RB_LOG_COMPILER=ruby23
EXE_LOG_COMPILER=wine

check_PROGRAMS=foo
TESTS=foo bar.py qux.rb

This works just fine for the python and ruby examples (although it doesn’t prove terrifically useful given they are typically executed just fine without additional configuration).

However EXE_LOG_COMPILER is never used because Automake doesn’t use the host system’s executable extension, EXEEXT, in the test targets. As a quick fix I experimented with transforming the check_PROGRAM list to include EXEEXT, but the complexity grew rather quickly for my particular case.

A global wrapper

Given I was getting sick of buildsystems after a week of assorted modifications I decided to use the global LOG_COMPILER hook and hand off to wine inside a trivial shell script.

#!/bin/sh

case "$1" in
*.exe)
    /usr/bin/env wine $@
    ;;

*)
    $@
    ;;
esac

I just trivially forward all arguments to wine if we’re being called with an .exe, otherwise I call the target as normal. Easy.

Line endings

Testing quickly with make check showed up errors immediately. All tests execute with successful exit-codes and the output succeeds visually, but Automake reports the test output for lacks a required final output line.

Inspecting the test’s .log file doesn’t throw any red flags, but given the Linux binaries are executing correctly and even the most trivial test fails to parse correctly under wine I have some suspicion that line endings might be a problem.

Different systems use different character sequences to denote a newline. UNIX uses LF, MacOS 9 uses CR, Windows uses CRLF, and there are various other quirky flavours. After a quick modification to my TAP code, which forced the use of LF rather than CRLF, the tests pass.

Modifying tap-driver.sh

Manually specifying the `correct’ line ending would prove quite fragile in the future, and it’s probably best to avoid hacks like these in our test framework of all places. So we look deeper into Automake’s testing machinery.

The only area of Automake’s testing framework I’m aware of that actually cares about the test output (in my case) is tap-driver.sh. This handles TAP parsing, output, and reporting. It comprises of a small amount of bash for argument parsing and a few hundred lines of embedded gawk.

Not knowing anything about gawk I wrangled a `fix’ which used a regex to replace CRLF pairs with LF characters just before parsing the test output. It works. However, it’s a terrible hack, and I desperately want to avoid maintaining a patch for bash/gawk/whatever.

The correct way to solve this problem seems to be changing gawk’s record delimiter from LF to CRLF by supplying a value for the `RS’ variable. Unfortunatetly gawk settings do not appear to be exposed to client scripts. Which, on later reflection, is probably for the best.

Fleshing out the global wrapper

After a little thought, and backing up a few steps, I recall the tiny wine-wrapper.sh script. What if I just transform the test output there before it hits Automake and tap-driver.sh?

#!/bin/sh

case "$1" in
*.exe)
    OUTPUT="$(/usr/bin/env wine $@)"
    STATUS=$?

    echo "$OUTPUT" | sed 's/\r$//'
    exit $STATUS
    ;;

*)
    $@
    exit $?
    ;;
esac

The above script could be more efficient or compact, but I’m not sufficiently familiar with POSIX shell scripting and handling pipe exit codes in a cross-platform way is beyond my current skills.

Sure, it gives us inconsistent line endings if we’re running under Windows proper, but I’m not planning on running these tests there en-masse just yet.

Also, I had lost any trace of motivation for buildsystems at this point.

EXTRA_DIST = ${top_srcdir}/build-aux/wine-wrapper.sh
LOG_COMPILER = ${top_srcdir}/build-aux/wine-wrapper.sh

check_PROGRAMS = foo
TESTS = foo

After installing the new LOG_COMPILER we now have consistent line endings (under Linux) for all unit tests, no patches to external code, and a suite of Windows tests passing with full TAP output under Linux.

I call that a win.