Writing a JQF test

JQF tests are JUnit tests that take one or more arguments. JQF generates the values for these arguments many times, in what is called a fuzzing loop.

JQF builds on junit-quickcheck, so do checkout the junit-quickcheck documentation for tips on writing generators for complex types. Let's summarize some of the differences between junit-quickcheck and JQF:

• Test methods must be annotated with @Fuzz instead of @Property. The @Property annotation allowed setting several configuration parameters such as the number of trials or the random seed to use; in JQF, we defer the running of the fuzzing loop to a separately configurable front-end. For example, when using AFL as the front-end, the fuzzing loop runs infinitely unless the AFL process is killed by an external user.
• Test classes must be annotated with @RunWith(JQF.class) instead of @RunWith(JunitQuickcheck.class). However, class JQF extends JunitQuickcheck, so a JQF test class can have a mix of JQF @Fuzz targets, Quickcheck @Property tests and classic Junit @Test methods without any arguments. All of these can be part of your standard project test suite. By default, JQF uses NoGuidance for 100 trials, which is basically the same as vanilla quickcheck.

Adding Dependencies to JQF

You'll of course need to add JQF as a dependency to your test classes. You can either add that manually or using Maven.

Command-line

Use the handy script classpath.sh that expands to the JQF classpath needed to compile your tests.

# Assume you have a test class in `MyTest.java`
javac -cp $(jqf/scripts/classpath.sh) MyTest.java

Maven

If you mvn install JQF from the JQF directory then you'll have the snapshot in your local repository which you can now import in your custom project as follows:

    <dependencies>
        <dependency>
            <groupId>edu.berkeley.cs</groupId>
            <artifactId>jqf-fuzz</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>edu.berkeley.cs</groupId>
            <artifactId>jqf-instrument</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
    </dependencies>

We are still working on making a release to Maven central so that you won't have to install locally.

Sample tests

@RunWith(JQF.class)
public class SortTest {
    @Fuzz
    public void checkSorted(Integer[] items) {
        // Sort using TimSort
        Arrays.sort(items);

        // Assert sorted
        for (int i = 1; i < items.length; i++) {
            Assert.assertTrue(items[i-1] <= items[i]);
        }
    }
}

@RunWith(JQF.class)
public class NumberParsingTest {
    @Fuzz
    public void fuzzParseFloat(String input) throws NumberFormatException {
         Float.parseFloat(input);
    }

    @Fuzz
    public void fuzzParseInt(String input, 
                             @InRange(minInt=Character.MIN_RADIX, 
                                      maxInt=Character.MAX_RADIX) int radix) 
            throws NumberFormatException {
        Integer.parseInt(input, radix);
    }
}

Note: These examples show tests that stress JDK classes but instrumentation of java.lang and java.util classes is disabled by default to improve performance.

Assume, Assert and throws

The throws clause can be used to specify which exceptions the test method expects to throw under normal circumstances. Examples include checked exceptions when parsing malformed or otherwise invalid input data. If an expected exception is thrown, then JQF will mark the test run as a SUCCESS, similar to if the test method returned normally without any exceptions.

@Fuzz
public void fuzzParseInt(String input) throws NumberFormatException {
    Integer.parseInt(input);
}

The Assume class from Junit can be used to make assumptions about input values generated when using JQF. If a generated input leads to an assumption violation, it signals to JQF that this trial is invalid and that another input should be generated.

@Fuzz
public void fuzzParseInt(String input) {
    assumeThat(input.length(), greaterThan(0));
    assumeTrue(Character.isDigit(input.charAt(0));
    Integer.parseInt(input); // Should not throw NumberFormatException
}

The Assert class from Junit can be used to make strong assertions about the results of applying application-specific operations on generated input data. If an assertion is violated, then JQF will mark the test run as a FAILURE.

@Fuzz
public void fuzzParseInt(String input) {
    assumeThat(input.length(), greaterThan(0));
    assumeTrue(Character.isDigit(input.charAt(0));
    // If first char is a digit, then number cannot be negative
    int x = Integer.parseInt(input); 
    assertThat(x, greaterThanOrEqual(0));
}

If the test method throws a run-time exception that is not listed in the throws clause, then JQF will mark such a test execution as a FAILURE. If your test involves invoking methods that throw checked exceptions but if you do not expect such an exception to be thrown, then it is better to wrap it and throw an unchecked runtime exception instead, so that JQF will catch it:

@Fuzz
public void canReadFromByteArray(String str) {
    assumeThat(str.length(), greaterThan(0));
    InputStream in = new ByteArrayInputStream(str.getBytes());
    try {
        in.read();
    } catch (IOException e) {
        throw new RuntimeException("Should not happen", e);
    }
}