RunTests Cluster testing

Our cluster testing has traditionally been targetted at testing where we build something, and then run lots of test files through it, and watch for changes in the 'bitmaps' produced.

As we expand our software to offer more APIs etc, we'd like to be able to do something a bit more akin to unit testing. The proposed solution to this is to expand the cluster to offer a 'runtests' test.

Rather than doing a single build, and then firing multiple test files through it, this drives the cluster differently. For every test we want to run, we have a shell script (foo.runtest) in tests_private/runtests/ that performs both any building required, plus any testing.

This test will then output results in a specific format so that the cluster can pick them up.

How to run such a test.

Use "clusterpush.pl runtests" (or "gitpush.pl runtests").

These will be run nightly too. Possibly we may want to run them on every gs commit.

How to write such a test

You need to write a shellscript, and check it in as "tests_private/runtests/whatever.runtest".

How this script is invoked

For every test script to be run, the cluster creates a temporary directory, and populates it with:

  • ghostpdl - a symbolic link to the ghostpdl source to be tested.
  • tests - a symbolic link to the tests directory
  • tests_private - a symbolic link to the tests_private directory
  • test.sh - a copy of the whatever.runtest file (YES, it is copied and renamed to be test.sh)

The cluster then changes directory into this temporary directory, and runs test.sh, capturing the stdout, and the return code. A return code of non-0 indicates error. Somewhere in the stdout, we expect to see a line like:

   367c3858886d1c43b75e3ea48f770d80db800d15 -

i.e. the output from a command such as "cat testlog | md5sum". NOTE, "md5sum testlog" will NOT work.

This md5sum is what is used as the 'result' of the test, and changes to this are what triggers things being spotted as changing in the test reports.

Any temporary files required can be created within the temporary directory, and all will be tidied up automatically at the end when the directory is deleted.

An example test

Our first runtest file is for testing the Ghostscript C API by using ghostpdl/demos/c/api_test.c.

The script looks as follows:

#!/bin/bash

Thou shalt use bash. Seriously. Don't use any other shells.

cd ghostpdl/demos/c

We do all our running within the ghostpdl tree, including making temp files there. We tidy those up as part of the test.

echo BUILD: > buildlog
echo TEST STDOUT: > testlog
echo TEST STDERR: > testerr

We're going to collect the output from different phases of the test into different files. We label each one internally, because all of them are going to be concatenated in the logs.

make api_test &>> buildlog

Make the source using the makefile in the ghostpdl tree. We'll look at the contents of this later. Keep both stdout and stderr from it in buildlog. Note that while stdout and stderr might get interleaved differently in different builds, and changes in the gs build system may change the output. As such the output here is NOT considered for the md5sum 'result' later.

RESULT= make run_api_test 1>> testlog 2>> testerr

Run the actual test, keeping the return code in RESULT. This will generate various output files within the directory that we will test in a bit.

echo POSTTEST STDOUT: >> testlog
echo POSTTEST STDERR: >> testerr
make post_api_test 1>> testlog 2>>testerr

Label that we have moved on to the 'post test', and run the post test. For this particular runtest, we md5sum the (many) bitmaps that the test produced. These md5sums then end up in the testlog, so any changes in them will invalidate the test.

cat testerr >> testlog

Concatenate the testerr onto testlog. We do this explicitly at the end, rather than just collecting both stderr and stdout together into testlog, to avoid the indeterminate interleaving of data we'd get doing that.

cat testlog | md5sum

That generates the actual result we use.

cat buildlog testlog

We display the logs. This causes all the jobs output to be made visible in the log that the cluster collects, so any changes can be examined on the web.

rm buildlog testlog testerr

Tidy up the files we made.

And that's it.

The test script makes use of a Makefile stored in the gs tree (ghostpdl/demos/c/Makefile).

api_test: api_test.c
        (cd ../.. && ./autogen.sh)
        (cd ../.. && make so)
        pwd
        gcc -fPIC -I../.. api_test.c -DGHOSTPDL=1 -lgpdl -L../../sobin -o api_test

run_api_test: api_test
        LD_LIBRARY_PATH=../../sobin ./api_test

post_api_test:
        md5sum apitest*
        rm apitest*

api_test is the standard build target to make the test program. It configures gs, makes it, and builds it.

run_api_test runs the build program, and produces many 'apitest*' bitmaps.

post_api_test md5sums those bitmaps, and then deletes them.

-- Robin Watts - 2020-08-27

Comments

Topic revision: r1 - 2020-08-27 - RobinWatts
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright 2014 Artifex Software Inc