How to test a test file

Well, you could just commit it to tests_private and hope

Or you can simulate what the cluster does on your local machine.

What the cluster does

Currently, for a given test file, the cluster does the following:

mkdir temp.dir

(Makes a directory for us to work in)

OLD=`pwd`
ln -s $OLD/tests_private temp.dir/tests_private
ln -s $OLD/tests temp.dir/tests
cp YOURTESTSCRIPT.js temp.dir/test.js

Remember where we were, and populate the test directory with 'tests_private', 'tests', and a copy of the script to be tested, renamed to 'test.js'.

cd temp.dir
mutool run test.js > stdout 2> stderr

Change into the directory, and run the test, collecting the outputs.

/bin/echo 'return:' $?
echo STDOUT:
cat stdout
echo STDERR:
cat stderr

The cluster then carefully runs all the output to stdout, first the error code, then the collected stdout, and then the collected stdin. This is so the output appears in the logs for the cluster.

rm test.js tests tests_private

Then the cluster deletes test.js, tests, tests_private.

Any files left in the directory at this point were created by the script, and should be counted towards the md5sum.

echo OUTPUTS:
ls | sort

Next, list the names of all the output files, then

$OLD/strip_contents.pl `ls | sort` | md5sum > $OLD/temp.md5

Basically, this 'cat's each output file in turn through md5sum, storing $OLD/temp.md5 (the result). The reason it's not simply a cat is that the "Contents" section of each PDF varies by time. We can't easily allow for that, so we hackily skip it for the purposes of md5summing.

cd $OLD
rm -rf temp.dir

Delete temp.dir so we are back where we started, but with a "temp.md5" result.

--+++ What you can do:

• Make a temp directory.
• symlink tests_private and tests within it.
• Put YOURTESTFILE.js in there.
• mutool run YOURTESTFILE.js > stdout >& stderr
• Check that stdout, stderr and any generated files look reasonable, and don't vary from run to run (other than .pdf files varying in the Contents section).

Murun Cluster testing

As from 1 April 2020 (no, not an April Fool's joke) the cluster will perform tests on javascript files fed through "mutool run". These tests are known as "murun" tests.

A murun test is triggered on every commit to the mupdf repo, or by "clusterpush.pl murun".

Where do the scripts live?

The cluster looks for scripts (named "*.js") in tests_private/pdf/js/.

For each of these scripts, a temporary directory is created, containing "tests_private" and "tests". The script is copied into the directory as test.js, and is run using "mutool run test.js". Both stdout and stderr are captured. Scripts can save documents to the current directory (please, nowhere else), and reload documents from that same directory.

Any other resources (such as key files) should be checked into the same directory as the script (e.g. tests_private/pdf/js/) and should be loaded using relative paths (e.g. "tests_private/pdf/js/ArthurDent.pfx").

The log captures both stdout and stderr from the script.

The "result" of the script is the md5sum of all the files that end up in the test directory (after test.js, tests_private, tests etc have been removed). Thus any change in the stdout, stderr, output bitmaps and pdf outputs of the script will cause the md5 to change. (The one exception to this is that PDF files have their signing digests ignored for the purposes of the md5summing to avoid differences between signed PDF files due to randomness introduced as part of the cryptographic processes performed by libcrypto).

What if I run a file, and a difference is reported?

Firstly, you can click the "log" link from the results webpage to see the captured output from stdout/stderr of each run. This may show you errors where the internal tests within the scripts have picked up differences.

Alternatively, you can run a bmpcmp. This should show you the differences between the bitmaps produced as the script runs, in both old and new mutools.

If all else fails, you'll need to reproduce the runs locally.

How to author a test file

Manually.

You can, of course, write scripts manually. Anything acceptable to "mutool run" can be used.

Remember that the output of the script can be bitmaps or stdout/stderr changes.

Using mupdf-gl

One major driver of this testing is a desire to test interactive features as used within a viewer. These could be scripted manually, but that's likely to be very tedious. Therefore, mupdf-gl has been extended with a facility to dump out the actions that happen in its UI to a test file.

If you run:

• mupdf-gl -T test.js input.pdf

Then it will load input.pdf as normal. Any interaction with the document (changing page, clicking in form fields etc), will cause javascript to do that same action to be written into test.js.

When a document is signed, the equivalent code to sign, save out, and reload the document is written to test.js.

When a document is loaded (either originally, or as a reload after signing), javascript is written to test.js that checks the status of signature and data fields, and prints errors if it is not as expected.

Once you've completed all the steps you want to test, exit mupdf-gl.

You will almost certainly need to fixup the paths in the resultant test.js manually.

Hopefully, all being well, this script can be tested in "mutool run test.js", and once you are happy with it, it can be committed (with a more meaningful name) into tests_private/pdf/js/.

-- Robin Watts - 2020-04-01

Comments

<--/commentPlugin-->