2.2 Testing

2.2.1 check: and where: blocks

Tests in Pyret are written in special testing blocks. These blocks can contain any Pyret code that isn’t toplevel-only (like data definitions and import or provide statements), and are the only places where Testing Operators can be used.

2.2.1.1 check: blocks

The simplest testing blocks are check: blocks. They can be written at the top-level or inside other testing blocks. Check blocks are a unit of reporting test results, so all the test operators that evaluate inside a check block will be reported as part of that block. For example, these two check blocks:

check "a first block":
  5 is 5
  4 is 5
end

check "a second block":
  6 is 7
end

will report:

Check block: a first block

test (5 is 5): ok

test (4 is 5): failed, reason:

Values not equal:

1/2 tests passed in check block: a first block

Check block: a second block

test (6 is 7): failed, reason:

Values not equal:

The test failed.

1/3 tests passed in all check blocks

Testing blocks are also a unit of failure: most of the time an error stops the whole program, but inside a check block (and also inside raises, mentioned later), the error is stopped and reported, and Pyret goes on to evaluating the next check block:

check "error-block":
  raise("an error here doesn't stop the next check block from running")
  string-length("this test doesn't run") is 21
end

check "a later block":
  string-length("these tests still run") is 21
end

Keep an eye out for the message "Check block <some-block> ended in an error (all tests may not have run):", because it means that later tests in the same block may not have run, so the output doesn’t reflect all the tests that were written.

2.2.1.2 where: blocks

Sometimes a function has tests that are explicitly associated with it. For these cases, the function can end in a where: block rather than immediately with end. where: blocks run the same way that check: blocks do, and their name is taken from the function they are attached to.

Examples:

fun double(n):
  n + n
where:
  double(10) is 20
  double(15) is 30
end

2.2.2 Testing Operators

Testing operators should be written on their own line inside a check: or where: block. They can check for a number of properties and come in several forms.

2.2.2.1 Binary Test Operators

Many useful tests compare two values, whether for a specific type of equality or a more sophisticated predicate.

expr1 is expr2

Evaluates expr1 and expr2 to values, and checks if two values are equal via equal-always, reporting success if they are equal, and failure if they are not.

expr1 is-not expr2

Like is, but failure and success are reversed.

expr1 is-roughly expr2

Like is, but tolerant of roughnum values: specifically, this is a shorthand for is%(within(0.000001)).

expr1 is%(pred) expr2

Evaluates expr1 and expr2 to values, and pred to a value that must be a function (an error is reported if pred is not a function). It then applies pred to the two values from expr1 and expr2. If the result of that call is true, reports success, otherwise reports failure.

expr1 is-not%(pred) expr2

Like is%, but failure and success are reversed.

Examples:

check:
  fun less-than(n1, n2): n1 < n2 end

  1 is%(less-than) 2
  2 is-not%(less-than) 1
end

check:
  fun longer-than(s1, s2):
    string-length(s1) > string-length(s2)
  end

  "abc" is%(longer-than) "ab"
  "" is-not%(longer-than) ""
end

check:
  fun equal-any-order<a>(l1 :: List<a>, l2 :: List<a>):
    same-length = (l1.length() == l2.length())
    all-present = for lists.all(elt from l1):
      lists.member(l2, elt)
    end
    same-length and all-present
  end

  [list: 1, 2, 3] is%(equal-any-order) [list: 3, 2, 1]
  [list: 1, 2, 3] is%(equal-any-order) [list: 2, 1, 3]
  [list: 1, 2, 3, 3] is-not%(equal-any-order) [list: 2, 1, 3]
end

check:
  fun one-of(ans, elts):
    lists.member(elts, ans)
  end

  some-strings = [list: "123", "132", "213", "231", "312", "321"]
  "321" is%(one-of) some-strings
  "123" is%(one-of) some-strings
end

check:
  fun around(delta):
    lam(actual, target):
      num-abs(target - actual) <= delta
    end
  end

  5.05 is%(around(0.1)) 5
  5.00002 is-not%(around(0.00001)) 5
end

expr1 is== expr2

Shorthand for expr1 is%(equal-always) expr1. Same as is.

expr1 is-not== expr2

Like is==, but failure and success are reversed. Same as is-not.

expr1 is=~ expr2

Shorthand for expr1 is%(equal-now) expr1

expr1 is-not=~ expr2

Like is=~, but failure and success are reversed.

expr1 is<=> expr2

Shorthand for expr1 is%(identical) expr1

expr1 is-not<=> expr2

Like is<=>, but failure and success are reversed.

2.2.2.2 Unary Test Operators

expr satisfies pred

Evaluates expr to a value and pred to a value expected to be a function (if not a function, an error is thrown). Then, pred(val) is evaluated, and if the result is true, the test succeeds, and if false, the test fails.

expr violates pred

Like satisfies, but failure and success are reversed.

Examples:

check:
  [list:] satisfies is-empty
  [list:] satisfies lam(l): l.length() == 0 end

  is-odd = lam(n :: Number): num-modulo(n, 2) == 1 end
  5 satisfies is-odd
  6 violates is-odd
end

2.2.2.3 Exception Test Operators

expr raises exn-string

Evaluates expr and expects an error to be raised. If no error is raised, the test fails.

If an error is the result, the torepr function is called on the exception value, and raises checks that exn-string is contained within that string. If so, the test passes, otherwise, it fails.

For simple errors (like those in many programming assignments), it works to use raise on a string value and check that that string is raised. For larger programs, it can be useful to construct more sophisticated error values and use raises-satisfies to test them.

Examples:

check:
  raise("the roof!") raises "the roof"

  string-length("too", "many", "strings") raises "arity-mismatch"

  {}.x raises "field-not-found"
end

Warning! These two tests are not equivalent:

check "actually catches the error":
  raise("error!") raises "error!"
end

check "error happens before raises":
  value = raise("error!")
  value raises "error!"
end

This is because the left-hand-side of raises is a special position that can detect and catch errors, which normal expressions do not do. So the second check block fails before even getting to the raises line; try it out and see what happens.

expr raises-other-than exn-string

Like raises, but the result must not contain exn-string.

expr does-not-raise

Evaluates expr and checks that no error is raised while evaluating it. The expression can evaluate to any value.

expr raises-satisfies pred

As the name suggests, this combines the idea of raises with satisfies and calls pred on the exception that expr raises (if any). Still fails if no exception is raised.

Examples:

import is-field-not-found from error
check:
  o = {}
  o.x raises-satisfies is-field-not-found
end

expr raises-violates pred

Like raises-satisfies, but the predicate must return false. Still fails if no exception is raised.

2.2.3 Reasons for tests: because clauses

When writing a test case, we may have several goals in mind: we might want to demonstrate whether a particular function works properly, or we might want to explore why a particular function works the way it does. Consider the following two test cases: when reading them, what meaning do they convey?

Examples:

check:
  distance-to-origin(3, 4) is 5
  distance-to-origin(3, 4) is num-sqrt(num-sqr(3) + num-sqr(4))
end

Reading the first test case is concise and clear: the expected distance is simply 5. But why? Nothing in the expected output gives any insight into how the function works. By contrast, the second test case gives far more insight, and “shows our work”...but it is also much lengthier. Writing many such test cases would get very tedious, very quickly. Additionally, there’s nothing tying the two test cases together: we have to notice that the two tests are adjacent in our program and their left sides are identical, to notice that both tests are about the same input scenario.

Pyret allows us to write test cases in a slightly different way, that addresses both of these concerns:

Examples:

check:
  distance-to-origin(3, 4) is-roughly 5
    because num-sqrt(num-sqr(3) + num-sqr(4))
end

Read this aloud as “The distance to origin of (3, 4) is roughly 5, because the square-root of three squared plus four squared is roughly 5.” The because clause lets us show work, while also connecting the explanation to the original test case.

Now that there are potentially three components to writing a single test case, there are multiple ways a test case can fail:

The explanation could be wrong, and so the expected value and the explanation do not match.
The expected value could be wrong, and so the explanation does not match the expected value even if the explanation is correct.
The function itself could be wrong, and so the left-hand side produces the wrong value and does not match the expected value.
The expected value could be wrong, and so the left-hand side does not match the right-hand side even if the function behaves properly.

As an example of the first case, suppose we had a typo in our explanation (we used num-sqrt instead of num-sqr):

check:
  distance-to-origin(3, 4) is-roughly 5
    because num-sqrt(num-sqrt(3) + num-sqr(4))
end

Pyret will show us

Here, even if the function is defined properly, the explanation and the expected result are inconsistent. Pyret will show this inconsistency as a test failure, even if the left-hand side and the expected value do match — after all, we might simply have gotten lucky, and the explanation is more accurate! A test case using a because clause will pass only if the explanation matches the expected value and the left-hand side matches the expected value.

Using a because clause is optional, and is most helpful to illustrate a few select examples to demonstrate how we think a function should be working. Once we have a few test cases are passing, we can easily add several more and leave out the because clauses for them...but if any of them unexpectedly fail, we can easily add a because clause to them to help debug the failure.

2.2.3.1 Using because with other testing operators

For an arbitrary test case expr1 <test-op> expr2 because expr3, read this aloud as “expr1 <test-op> expr2 because expr3 <test-op> expr2.” So for example:

times-two(3) is 6 because 2 * 3 reads as “times-two of 3 is 6, because two times 3 is 6.” Note that this reads analogously for is-roughly, is==, is=~, is<=>, and is%(...).
times-two(3) satisfies is-even because 2 * 3 reads as “times-two of 3 should be even, because two times 3 should be even.”
reciprocal(0) raises "division by zero" because 1 / 0 reads as “The reciprocal of zero raises a division-by-zero error, because 1 / 0 raises a division-by-zero error.”

The other testing operators also work with because in the same way, though it is a bit harder to read them aloud.

contents ← prev up next →

1	Getting Started
2	Language Concepts
3	Builtins and Libraries
4	Pyret Style Guide
5	Internals
6	Glossary

2.1	Language Constructs
2.2	Testing
2.3	equality
2.4	Combining Multiple Operators
2.5	Modules
2.6	Use
2.7	Brands
2.8	Type Checking
2.9	Spies

2.2.1	check: and where: blocks
2.2.2	Testing Operators
2.2.3	Reasons for tests: because clauses