Testing

Testing in ProfessorJ	PLT \| DrScheme \| ProfessorJ

ProfessorJ implements linguistic support for writing and executing unit tests in each of its language levels.

The pedagogic levels implement a more restricted form of these extensions, as they implement a more restricted form of the overall language.

Pedagogic testing

All three pedagogic language levels contain a new expression form that compares two values and returns a boolean

check EXPRESSION expect EXPRESSION

check EXPRESSION expect EXPRESSION within EXPRESSION

The shortened variant of the check expression may be used to compare any non-floating point values. The longer form may be used to compare any two values, using the third argument to set the acceptable margin of error between any floating point values.

The expression -- check "Hello World" expect "Hello World" -- compares the two strings and produces true.
The expression -- check 5 expect 4 -- compares the two numbers and produces false, however using the within variant -- check 5 expect 4 within 1 -- produces true.

In the following examples, a student tests their implementation of a truck:

class Truck {
int year;
int modelNo;
Truck( int year, int modelNo) {
     this.year = year;
     this.modelNo = modelNo;
}
double fuel() {
     return 4.5;//gallons
}
}

Given this definition, each of the following check expressions produces true, without the student writing an equality method within their Truck class

check new Truck(2002,123456) expect new Truck(2002,123456)
check new Truck(2002,123456).fuel() expect 4.0 within 0.6
check new Truck(2002,1) expect new Truck(2006,1) within 5

And these expressions produce false

check 10/0 expect 0
check new Truck(2002,1) expect new Truck(2006,1)

Even though the expression 10/0 causes a runtime error, the check expression does not cause a runtime error. Instead the check expression successfully produces false, and the error will be reported by the overall test report.

This form simplifies the specification of particular tests, but does not provide unit testing. For that, students must be able to appropriately group and identify their tests. This is done using particular classes and methods.

Classes with Example in their name are considered to be test classes. Methods within such a class beginning with the word test are considered to be individual tests.

A test of the Truck class follows

class TruckExamples {
Truck t = new Truck( 2002, 123456);

boolean testFuel() {
     return (check t.fuel() expect 4.5 within 0.1) &&
               (check t.drive().fuel() expect 4.0 within 0.1);
}
}

Test classes such as TruckExamples are used to automatically run tests for the program by the running the program, with a report as described below.

Full testing extension

In the Java and Java+dynamci language levels, three expression forms can be enabled (see enabling options below), all producing booleans. The first form corresponds to the check form provided in the pedagogic language levels.

check EXPRESSION expect EXPRESSION

check EXPRESSION expect EXPRESSION within EXPRESSION

check EXPRESSION throws NAME

EXPRESSION -> EXPRESSION

The check ... throws expression anticipates that evaluating the tested expression will result in exceptional behavior (i.e. that the expression will throw an instance of Throwable). The provided name should refer to the Throwable class that is expected.

The expression -- check 10/0 throws RuntimeException -- produces true. If the test expression uses a method or constructor with a declared throws, the check expression operates like a try statement to control the exception.

If the tested expression either produces a value, such as check 4 throws Throwable, or throws an instance of different class, such as check 10/0 throws Error, the result of the check expression is false.

The expression within the check ... throws form does not need to produce a value, i.e. the expression can be a call to a method returning void.

The -> expression does not perform additional calculations, but sequences the two expressions and record information for the test report. The produced value of this expression is the result of the second expression, which must return a boolean.

In the expression, file.write("Hello") -> check file.peek(0) expect 'H', the call to write modifies the state of the file and the check ensures that the correct action is performed. The presence of the arrow indicates to a reader of the expression the connection between the two expressions as well as providing context for the test report, should the check expression fail.

These expressions can be used within specific testing forms, designed to support unit division of tests. The non-pedagogic language levels add two additional forms to the language, test and testcase.

test FuelMilage {
testcase TruckFuel() { return check new Truck(...).fuel() expect 4.5 within .01; }
}

The test form resembles a class, and the testcase form resembles a method (with return type boolean and no arguments).

One problem in specifying unit tests through a naming convention lies in silent failure. A simple typo in the name can cause a test case to not execute at the proper time, and with sufficient numbers of tests, the programmer will not notice the omission of the test. By exetending the language with these forms, the compiler reports an error when incorrectly specifying a test, ensuring no silent failure.

A test can extend another test, including inheriting fields, methods and testcases, but cannot extend classes or implement interfaces. An instance of a test can be created manually, as though FuelMilage were a class. A test must always contain a non-private constructor that requires no arguments. An instance of a test contains all of the members provided to Object, and can be cast to Object.

Test Reports

In all ProfessorJ language levels, executing the program using the Run button invokes the test reporting system.

Each test (or Example class) is instantiated, and its name provided to the test report. A display of the instantiated tests appears within the Interactions window.

Each testcase (or test method) is executed, and deemed to succeed when returning true and failed otherwise. A summary of the performance of the testcases is provided to the test report.

The test report is contained within a third DrScheme window, which can be undocked from the main DrScheme window.

The summary report includes the number of attempted tests, the number of failed tests, and the same information for individual check expressions.

Test in Beginner: docked window

When coverage analysis is enabled, expression-level coverage information is available for each instantiated test (or Example class) and each testcase (or test method).

The coverage information is displayed by changing the program's colors in the Definition window. Under the default settings, covered expressions become a reddish color while uncovered expressions become black.

Coverage information can be displayed for each individual test, each individual testcase (failed or successful), as well as for the entire set of tests executed.

Test in Beginner with coverage

Highlighting of a failed check

When a check expression returns false, the test reporting system is provided with information including the source of the check, the expected and received values, as well as other information discussed in a moment.

The source location is used to provide links to highlight the location of the check within the source program as well as to provide context describing the operation undertaken (such as a call to method fuel).

When the check expression fails due to exceptional behavior during the evaluation of the tested expression, the error report indicates that no value was received and reports the encountered error.

For a check ... throws expression, either the unexpected value received is displayed or the type of the exception encountered, as well as the expected type.

For a check to the right of the -> operator, the report will include information regarding the expression to the left, such as a call to the method write on variable file.

Enabling/disabline Options

Support for test execution is not enabled by default in ProfessorJ Full or Java + dynamic. It can be enabled by changing the language preferences, found under the language menu. Removing the check/test extensions can also be done from this menu.

For all language levels, coverage information and test execution can be enabled/disabled within the language options menu.


Language menu for Advanced	Language menu for Java+dynamic

Colors for the coverage can be modified in the preferences for Java editing colors.

Color preference panel, under the main DrScheme preferences