Introduction
This is yet another framework for defining unit tests for the Scheme programming language (in particular, the Chicken implementation). Most testing frameworks require tests to be defined in their own files, tipically one file with tests per module; this one was created on the idea that the best location for test definitions is preceding the procedures they test.
Tests serve as documentation of the expectations of procedures, specially in unusual cases. The most natural location for documentation about a procedure is immediately before its definition, which is where most programmers will embbed comments describing it. By placing the tests in that location, they serve as examples of usage in common and unusual cases, which increases their value. Furthermore, placing tests adjacent to the procedures they test makes it less likely for a programmer to forget to update them when he changes the procedure tested.
Another design decision of this framework is to make tests part of the compiled program or library. Tests will typically require very little additional space in a binary, which we consider worth paying for the convenience of not having to build a separate binary and libraries for the purpose of testing a program. Of course, this doesn't mean the tests will always run —to run them, one will set the TESTS environment variable—, but they will be present in binaries and libraries. In other words, while it would be trivial to extend this framework to allow compilation without tests, we've purposefully decided not to do so.
A third important design decision we've taken is performing the evaluation of the tests only after all code has been loaded. Most testing frameworks will run the tests as soon as their definition is found; this one delays the evaluation until the run-tests procedure is called (which programs should do as part of their initialization). This allows testing of mutually recursive libraries.
Usage
To use this testing framework in your program or library, you should use the test macro once per test. It takes as arguments:
- The test expresion that you want to evaluate. During the evaluation of this expresion, the symbol “test-fail” will be bound to a procedure that will abort the test and cause it to fail; you should pass to this procedure arguments describing the reason for the abortion of the test.
- The expected result. This is optional: if you don't specify, the test is treated as an assertion, which means that any value other than #f will cause the test to pass.
- A comparison procedure. The default is equal?.
Here are a few examples:
(test (list? (list 1 2 3)))
(test (string-split "a:b:c" ":")
(list "a" "b" "c"))
(test (stream-filter even? (stream 1 2 3 4 5))
(stream 2 4)
(cut stream= char=? <...>))
Related tests should be grouped using the test-group macro. The macro expects a name for the group of tests to be followed by the actual tests. An example is provided below. Typically the name will be the name of the procedure tested. The use of test-group is optional.
Defining tests doesn't do anything by itself: at some point you have to call the run-tests procedure. This will cause the tests to be executed if the TESTS environment variable is set. If the TESTS_VERBOSE variable is set, information about each test will be printed. You should call the run-tests procedure when all your modules have been loaded and your environment is ready for the execution of your program.
You can control which tests to run by setting the TESTS_GROUPS variable to a list of the groups of tests that you want to run.
Example
Here is an example of a full program, with some tests:
(use embedded-test)
(test-group square
(test (square 0) 0)
(test (square 1) 1)
(test (square 5) 25))
(define (square x) (* x x))
(test-group fact
(test (fact 0) 1)
(test (fact 1) 1)
(test (fact 5) (* 5 4 3 2 1)))
(define (fact x)
(if (= x 0)
x
(* x (fact (- x 1)))))
(run-tests)
(display (fact (square (read))))
(newline)
Running the above program with the TESTS environment variable set will produce the following output and abort the execution, showing that some of the tests failed due to the bug in the fact procedure:
Test from group fact failed: (fact 0) Expected: 1 Received: 0 Test from group fact failed: (fact 1) Expected: 1 Received: 0 Test from group fact failed: (fact 5) Expected: 120 Received: 0 Error: Unit tests failed
Implementation
Chicken Headers
Scheme Library: embedded-test
- Author
- Alejandro Forero Cuervo <azul@freaks-unidos.net>
- Category
- testing
- License
- GPL-3
- Exports
- register-test run-tests test test-group test-error *test-group-name*
Runtime code
Defining tests
We use a list to keep track of all tests defined so far:
(define *tests* '())
Unit tests are represented as a record:
(define-record test group-name name proc expect equal?)
The fields serve the following purpose:
- group-name
- If this test belongs to a group of tests, the name of the group. #f otherwise.
- name
- The name of the test, used for reporting failures.
- proc
- A procedure of one argument that evaluates the test expression. The argument is a procedure that, if called, will cause the test to fail.
- expect
- A procedure of no arguments that evaluates the expected result.
- equal?
- The comparison procedure, receiving the results of calling proc and expect and returning a boolean to indicate if they are the same.
Now a procedure for registering new tests:
(define (register-test . args) (set! *tests* (cons (apply make-test args) *tests*)))
We provide a global binding with the name of the current test group. This is used during the definition of tests (see the test-group macro bellow).
(define *test-group-name* #f)
Running a tests
We create an internal procedure to run one test and return success or failure.
(define test-error #f)
(define (errors-from-test test)
(when (getenv "TESTS_VERBOSE")
(format (current-error-port) "Test~A: ~S..."
(if (test-group-name test)
(format #f " from group ~S" (test-group-name test))
"")
(test-name test)))
(let ((errors #f))
(call-with-current-continuation
(lambda (return)
(set! test-error (lambda args (set! errors args) (return #f)))
(let ((result ((test-proc test))) (expect ((test-expect test))))
(unless ((test-equal? test) result expect)
(set! errors
(list (format #f "Expected: ~S~%Received: ~A" expect result)))))))
(set! test-error #f)
(when (getenv "TESTS_VERBOSE")
(format (current-error-port) " ~A~%" (if errors "FAIL" "PASS")))
(when errors
(format (current-error-port) "Test~A failed: ~S~%~{~A~%~}~%"
(if (test-group-name test)
(format #f " from group ~A" (test-group-name test))
"")
(test-name test)
errors))
errors))
Based on that, we define the public run-tests procedure that evaluates all tests in the order in which they were defined and, if one or more fail, raises an error.
(define (run-tests)
(when (getenv "TESTS_SHOW_GROUPS")
(format (current-error-port) "Groups:~{ ~A~}~%"
(delete-duplicates (map test-group-name (reverse *tests*)))))
(when (getenv "TESTS")
(let* ((tests (reverse
(filter
(let ((groups (map string->symbol
(string-split
(or (getenv "TESTS_GROUPS") "")
" "))))
(if (null? groups)
identity
(compose (cut member <> groups)
test-group-name)))
*tests*)))
(failures (count errors-from-test tests)))
(cond
((positive? failures)
(format (current-error-port) "Tests failed.~%")
(format (current-error-port) "Tests defined: ~A~%" (length *tests*))
(format (current-error-port) "Tests executed: ~A (~A%)~%"
(length tests)
(* 100 (/ (length tests) (length *tests*))))
(format (current-error-port) "Tests failed: ~A (executed: ~A%, defined: ~A%)~%"
failures
(* 100 (/ failures (length tests)))
(* 100 (/ failures (length *tests*))))
(error "Unit tests failed"))
((getenv "TESTS_VERBOSE")
(format (current-error-port) "Tests passed.~%")
(format (current-error-port) "Tests defined: ~A~%" (length *tests*))
(format (current-error-port) "Tests executed: ~A (~A%)~%"
(length tests)
(* 100 (/ (length tests) (length *tests*)))))))))
Macros
All that remains is creating the test and test-group macros. The former simply fills in the blanks and calls register-test. Note that the users should never pass the name or group parameters but let it be filled for them.
(define-syntax test
(syntax-rules ()
((test expr) (test expr #t))
((test expr expect) (test expr expect equal?))
((test expr expect cmp?) (test expr expect cmp? 'expr))
((test expr expect cmp? name) (test expr expect cmp? name *test-group-name*))
((test expr expect cmp? name group)
(register-test group name (lambda () expr) (lambda () expect) cmp?))))
The test-group macro simply modifies the internal (test ...) forms, filling the group, and returns them.
(define-syntax test-group
(syntax-rules ()
((test-group name test ...)
(begin
(set! *test-group-name* 'name)
test ...
(set! *test-group-name* #f)))))
) ; close module def
Last update: 2010-07-09 (Rev 17016)