Regression tests

ispc doesn't have stand-alone unit tests; this is unfortunate and should be remedied at some point. There are, however, nearly 500 short test programs written in ispc that do a pretty good job of exercising the compiler's functionality. These tests are found in the tests/ directory of the ispc distribution.

Running the Tests

To run the tests, run the run_tests.sh shell script in the top-level ispc source directory. The tests run for a few minutes. (It would probably be worthwhile to write a little job system to run the tests in parallel across the system's multiple cores.)

If successful, the test script will print output like:

    % ./run_tests.sh
    Running correctness tests
    Running failing tests
    No surprises.
    %

If some tests fail, the test system will print output like:

    Running failing tests

And will also return a non-zero exit code.

Note: currently some of the tests fail under Windows; see https://github.com/ispc/ispc/issues/55.

Test System Design

Each test is in a self-contained ispc source file; it must define three functions:

• width(), which returns the number of result values that the test computes, this is almost always programCount.
• result(uniform float[]), which returns the result that the test function should return
• A test function, named one of f_v, f_f, f_fu, f_di, f_du, or f_duf. These various names encode the signature of the test function.

The test script compiles each test function to LLVM bitcode and runs ispc_test with the name of the bitcode file. ispc_test uses LLVM JIT compiler to compile executable code for the test and then runs the three functions above, passing the test function particular values of particular types based on its signature. It then checks to make sure that the values returned from the call to result() match the values returned from the call to the test function; if they don't the differing values are printed along with an error message.

To make this concrete, here is a example of a test (a cleaned-up version of tests/bool-float-typeconv.ipsc). It does a quick sanity check of bool to float type conversion.

    export void f_f(uniform float RET[], uniform float aUniform[]) {
        float a = aUniform[programIndex];
        RET[programIndex] = a < 3.;
    }
    
    export void result(uniform float RET[]) { 
        RET[programIndex] = 0; RET[0] = RET[1] = 1; 
    }

First, notice that the test function defined here is f_f. In addition to the array in which to store the result values computed by the function, the RET parameter, functions with the name f_f are also passed an array of floats, with values {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}. The test function converts this to a varying value a by directly indexing into this array with programIndex, giving a the value one in the first program instance and so forth. By inspection, we can see that the boolean test in the last line of f_f should evaluate to true for the first two program instances running, and false for all of the rest, and that the conversion of those bools to floats should put 1 in the first two program instances result values and zero in the rest. These, in turn, are the values that result() reports are expected.

Here are the types and values passed as parameters by ispc_test for functions with the various signatures listed above:

    export void f_v(uniform float RET[]);  // i.e. no parameters passed other than the output array

    // a[] = { 1, 2, 3, ... };
    export void f_f(uniform float RET[], uniform float a[]);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    export void f_fu(uniform float RET[], uniform float a[], float b);

    // a[] = { 1, 2, 3, ... };
    // b[] = { 2, 4, 6, ... };
    export void f_fi(uniform float RET[], uniform float a[], int b[]);

    // a[] = { 1, 2, 3, ... };
    // b[] = { 5, 6, 7, ... };
    export void f_di(uniform float RET[], uniform double a[], int b[]);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    export void f_du(uniform float RET[], uniform double a[], double b);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    export void f_duf(uniform float RET[], uniform double a[], float b);

Writing New Tests

New functionality should have targeted tests that exercise the set of features that the functionality introduces. If the functionality is in any way dependent on the mask, it's important to exercise a few cases like 'mask all on', 'mask all off', and 'mixed mask'.