Murun Cluster testing

As from 1 April 2020 (no, not an April Fool's joke) the cluster will perform tests on javascript files fed through "mutool run". These tests are known as "murun" tests.

A murun test is triggered on every commit to the mupdf repo, or by "clusterpush.pl murun".

Where do the scripts live?

The cluster looks for scripts (named "*.js") in tests_private/pdf/js/.

For each of these scripts, a temporary directory is created, containing "tests_private" and "tests". The script is copied into the directory as test.js, and is run using "mutool run test.js". Both stdout and stderr are captured. Scripts can save documents to the current directory (please, nowhere else), and reload documents from that same directory.

Any other resources (such as key files) should be checked into the same directory as the script (e.g. tests_private/pdf/js/) and should be loaded using relative paths (e.g. "tests_private/pdf/js/ArthurDent.pfx").

The log captures both stdout and stderr from the script.

The "result" of the script is the md5sum of all the files that end up in the test directory (after test.js, tests_private, tests etc have been removed). Thus any change in the stdout, stderr, output bitmaps and pdf outputs of the script will cause the md5 to change. (The one exception to this is that PDF files have their signing digests ignored for the purposes of the md5summing to avoid differences between signed PDF files due to randomness introduced as part of the cryptographic processes performed by libcrypto).

What if I run a file, and a difference is reported?

Firstly, you can click the "log" link from the results webpage to see the captured output from stdout/stderr of each run. This may show you errors where the internal tests within the scripts have picked up differences.

Alternatively, you can run a bmpcmp. This should show you the differences between the bitmaps produced as the script runs, in both old and new mutools.

If all else fails, you'll need to reproduce the runs locally.

How to author a test file

Manually.

You can, of course, write scripts manually. Anything acceptable to "mutool run" can be used.

Remember that the output of the script can be bitmaps or stdout/stderr changes.

Using mupdf-gl

One major driver of this testing is a desire to test interactive features as used within a viewer. These could be scripted manually, but that's likely to be very tedious. Therefore, mupdf-gl has been extended with a facility to dump out the actions that happen in its UI to a test file.

If you run:

  • mupdf-gl -T test.js input.pdf

Then it will load input.pdf as normal. Any interaction with the document (changing page, clicking in form fields etc), will cause javascript to do that same action to be written into test.js.

When a document is signed, the equivalent code to sign, save out, and reload the document is written to test.js.

When a document is loaded (either originally, or as a reload after signing), javascript is written to test.js that checks the status of signature and data fields, and prints errors if it is not as expected.

Once you've completed all the steps you want to test, exit mupdf-gl.

You will almost certainly need to fixup the paths in the resultant test.js manually.

Hopefully, all being well, this script can be tested in "mutool run test.js", and once you are happy with it, it can be committed (with a more meaningful name) into tests_private/pdf/js/.

-- Robin Watts - 2020-04-01

Comments

Topic revision: r1 - 2020-04-01 - RobinWatts
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright 2014 Artifex Software Inc