Regression tests

ispc doesn't have stand-alone unit tests; this is unfortunate and should be remedied at some point. There are, however, over 600 short test programs written in ispc that do a pretty good job of exercising the compiler's functionality. These tests are found in the tests/ directory of the ispc distribution.

Table of Contents Running the Tests Known Failures Test System Design Writing New Tests

Running the Tests

To run the tests, run the run_tests.py python script in the top-level ispc source directory. The tests run for a minute or two. (Tests run in parallel across the CPU cores of the system, so the more cores, the faster they go.)

If successful, the test script will print output like:

% ./run_tests.py
Found 4 CPUs. Running 630 tests.
Done 630 / 630 [failing_tests/scatter-vector.ispc]                              
%

If some tests fail, the test system will generate additional output indicating which test failed and how it failed. The exit code is equal to the number of tests that failed. Thus, if all pass, it generates a regular exit code of 0.

Known Failures

With the SSE4x2 target, a number of the tests fail with LLVM 2.9, printing errors like:

SplitVectorResult #0: 0x10103c710: v8f32 = fp_round 0x10103c610, 0x101028610 [ORD=28] [ID=0]
    
Do not know how to split the result of this operator!
UNREACHABLE executed at LegalizeVectorTypes.cpp:408!
Stack dump:
0.	Running pass 'X86 DAG->DAG Instruction Selection' on function '@f_fu'

Furthermore, also with the SSE4x2 target, with both LLVM2.9 and LLVM dev TOT, some of the short-vec* tests fail with an LLVM assertion:

    Assertion failed: (ShuffleVectorInst::isValidOperands(V1, V2, Mask) && "Invalid shuffle vector constant expr operands!"), function getShuffleVector, file Constants.cpp, line 1752.

Both of these seem to be bugs in LLVM; these bugs aren't present in the current LLVM development tree, so it's advisable to run with that if at all possible.

Test System Design

Each test is in a self-contained ispc source file; it must define three functions:

• width(), which returns the number of result values that the test computes, this is almost always programCount.
• result(uniform float[]), which returns the result that the test function should return
• A test function, named one of f_v, f_f, f_fu, f_di, f_du, or f_duf. These various names encode the signature of the test function.

The test script compiles each test function to LLVM bitcode and runs ispc_test with the name of the bitcode file. ispc_test uses LLVM JIT compiler to compile executable code for the test and then runs the three functions above, passing the test function particular values of particular types based on its signature. It then checks to make sure that the values returned from the call to result() match the values returned from the call to the test function; if they don't the differing values are printed along with an error message.

To make this concrete, here is a example of a test (a cleaned-up version of tests/bool-float-typeconv.ipsc). It does a quick sanity check of bool to float type conversion.

    export uniform int width() { return programCount; }
    
    export void f_f(uniform float RET[], uniform float aUniform[]) {
        float a = aUniform[programIndex];
        RET[programIndex] = a < 3.;
    }
    
    export void result(uniform float RET[]) { 
        RET[programIndex] = 0; RET[0] = RET[1] = 1; 
    }

First, notice that the test function defined here is f_f. In addition to the array in which to store the result values computed by the function, the RET parameter, functions with the name f_f are also passed an array of floats, with values {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}. The test function converts this to a varying value a by directly indexing into this array with programIndex, giving a the value one in the first program instance and so forth. By inspection, we can see that the boolean test in the last line of f_f should evaluate to true for the first two program instances running, and false for all of the rest, and that the conversion of those bools to floats should put 1 in the first two program instances result values and zero in the rest. These, in turn, are the values that result() reports are expected.

Here are the types and values passed as parameters by ispc_test for functions with the various signatures listed above:

    export void f_v(uniform float RET[]);  // i.e. no parameters passed other than the output array

    // a[] = { 1, 2, 3, ... };
    export void f_f(uniform float RET[], uniform float a[]);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    export void f_fu(uniform float RET[], uniform float a[], float b);

    // a[] = { 1, 2, 3, ... };
    // b[] = { 2, 4, 6, ... };
    export void f_fi(uniform float RET[], uniform float a[], int b[]);

    // a[] = { 1, 2, 3, ... };
    // b[] = { 5, 6, 7, ... };
    export void f_di(uniform float RET[], uniform double a[], int b[]);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    export void f_du(uniform float RET[], uniform double a[], double b);

    // a[] = { 1, 2, 3, ... };
    // b = 5;
    export void f_duf(uniform float RET[], uniform double a[], float b);

Writing New Tests

New functionality should have targeted tests that exercise the set of features that the functionality introduces. If the functionality is in any way dependent on the mask, it's important to exercise a few cases like 'mask all on', 'mask all off', and 'mixed mask'.