Related Books
Books / Introduction to C Programming Language / Chapter 46
Unit testing C programs
It is a truth universally acknowledged that test code should be written early in the development process. Unfortunately, most programmers (including me) tend to assume that a program will work on the first attempt and there’s not much point in testing it anyway, so writing and running test code often gets deferred indefinitely. The solution is to write the test code first, and run it directly from your Makefile every time you save and compile your program. Not only will this guarantee that your program actually works when you are done (or at least passes the tests you thought of), it allows you to see how the program is improving with each positive change, and prevents you from accidentally making new negative changes that break things that used to work.
Going one step further, we can often write our interface and test code first, build a non-working stub implementation, and then slowly flesh out the missing pieces until the implementation passes all the tests. This way there is always some obvious step to do next, and we don’t find ourselves stuck staring at an empty file.
A straightforward approach to testing is to include test code with every unit in your program, where a unit is any part of the program that can be sensibly run by itself. Typically, this will be a single function or a group of functions that together implement some data structure.
In C, these will often make up the contents of a single source file. Though this is probably not the best approach if you are building a production-quality testing framework, a simple way to include unit tests in a program is to append to each source file a test main function that can be enabled by defining a macro (I like TEST_MAIN). You can then build this file by itself with the macro defined to get a stand-alone test program for just this code.
What to put in the test code
Ideally, you want to use enough different inputs that every line of code in your program is reached by some test, a goal called code coverage. For complex programs, this may be hard to achieve, and there are programs, such as the gcov program that comes with gcc, that will analyze how much code coverage you get out of your tests. For simple programs, we can just try to come up with a set of inputs that covers all our bases.
Testing can be done as black-box testing, where the test code assumes no knowledge of the implementation, or white-box testing, where the test code has direct access to the implementation and can observe the effects of its actions. Black-box testing is handy if your implementation may change, and it is generally a good idea to write black-box tests first. White-box testing can be useful if some states of the data structure are hard to reach otherwise, or if black-box testing is not very informative about why a particular operation is failing. The example given below uses both.
Example
Here is an example of a simple data structure with some built-in test code conditionally compiled by defining TEST_MAIN. The data structure implements a counter with built-in overflow protection. The counter interface does not provide the ability to read the counter value; instead, the user can only tell if it is zero or not.
Because the counter is implemented internally as a uint64_t, black-box testing of what happens with too many increments would take centuries. So we include some white-box tests that directly access the counter value to set up this (arguably unnecessary) test case.
The code is given below. We include both the interface file and the implementation, as well as a Makefile showing how to build and run the test program. The Makefile includes some extra arguments to gcc to turn on the TEST_MAIN macro and supply the extra information needed to run gcov. If you type make test, it will make and run testCounter, and then run gcov to verify that we did in fact hit all lines of code in the program.
/*
* Abstract counter type.
*
* You can increment it, decrement it, and test for zero.
*
* Increment and decrement operations return 1 if successful,
* 0 if the operation would cause underflow or overflow.
*/
typedef struct counter Counter;
/* make a new counter starting at 0 */
Counter *counterCreate(void);
/* destroy a counter */
void counterDestroy(Counter *);
/* return 1 if counter is 0, 0 otherwise */
int counterIsZero(const Counter *);
/* increment a counter, returns 1 if successful, 0 if increment would cause overflow */
int counterIncrement(Counter *);
/* decrement a counter, returns 1 if successful, 0 if decrement would cause underflow */
int counterDecrement(Counter *);
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include "counter.h"
#include <stdint.h>
#define COUNTER_MAX (UINT64_MAX)
struct counter {
uint64_t value;
};
/* make a new counter starting at 0 */
Counter *
counterCreate(void)
{
Counter *c;
c = malloc(sizeof(Counter));
assert(c);
c->value = 0;
return c;
}
/* destroy a counter */
void
counterDestroy(Counter *c)
{
free(c);
}
/* return 1 if counter is 0, 0 otherwise */
int
counterIsZero(const Counter *c)
{
return c->value == 0;
}
/* increment a counter, returns 1 if successful, 0 if increment would cause overflow */
int
counterIncrement(Counter *c)
{
if(c->value == COUNTER_MAX) {
return 0;
} else {
c->value++;
return 1;
}
}
/* decrement a counter, returns 1 if successful, 0 if decrement would cause underflow */
int
counterDecrement(Counter *c)
{
if(c->value == 0) {
return 0;
} else {
c->value--;
return 1;
}
}
#ifdef TEST_MAIN
int
main(int argc, char **argv)
{
Counter *c;
/* black box testing */
c = counterCreate(); /* 0 */
assert(counterIsZero(c));
assert(counterIncrement(c) == 1); /* 1 */
assert(!counterIsZero(c));
assert(counterIncrement(c) == 1); /* 2 */
assert(!counterIsZero(c));
assert(counterDecrement(c) == 1); /* 1 */
assert(!counterIsZero(c));
assert(counterDecrement(c) == 1); /* 0 */
assert(counterIsZero(c));
assert(counterDecrement(c) == 0); /* 0 */
assert(counterIsZero(c));
assert(counterIncrement(c) == 1); /* 1 */
assert(!counterIsZero(c));
counterDestroy(c);
/* white box testing */
c = counterCreate(); /* 0 */
assert(c->value == 0);
assert(counterIncrement(c) == 1); /* 1 */
assert(c->value == 1);
assert(counterIncrement(c) == 1); /* 2 */
assert(c->value == 2);
assert(counterDecrement(c) == 1); /* 1 */
assert(c->value == 1);
assert(counterDecrement(c) == 1); /* 0 */
assert(c->value == 0);
assert(counterDecrement(c) == 0); /* 0 */
assert(c->value == 0);
assert(counterIncrement(c) == 1); /* 1 */
assert(c->value == 1);
/* force counter value to COUNTER_MAX to test for overflow protection */
c->value = COUNTER_MAX; /* COUNTER_MAX */
assert(counterIncrement(c) == 0); /* COUNTER_MAX */
assert(c->value == COUNTER_MAX);
assert(counterDecrement(c) == 1); /* COUNTER_MAX-1 */
assert(c->value == COUNTER_MAX-1);
assert(counterIncrement(c) == 1); /* COUNTER_MAX */
assert(c->value == COUNTER_MAX);
counterDestroy(c);
return 0;
}
#endif
CC=c99
CFLAGS=-g3 -Wall
all: seqprinter
seqprinter: main.o sequence.o
$(CC) $(CFLAGS) -o $@ $^
test: seqprinter
./seqprinter
# these rules say to rebuild main.o and sequence.o if sequence.h changes
main.o: main.c sequence.h
sequence.o: sequence.c sequence.h
clean:
$(RM) -f seqprinter *.o
examples/ADT/sequence/Makefile
Test harnesses
Here are some older notes on testing using a test harness that does some basic tricks like catching segmentation faults so that a program can keep going even if one test fails.
Module interface
The module will be a stack for storing integers.
Let’s start with the interface, which we’ll put in a file called stack.h:
stack.h
/*
* This is an "opaque struct"; it discourages people from looking at
* the inside of our structure. The actual definiton of struct stack
* is contained in stack.c.
*/
typedef struct stack *Stack;
/* constructor and destructor */
Stack stack_create(void); /* returns 0 on allocation error */
void stack_destroy(Stack);
/* push a new element onto the stack */
void stack_push(Stack , int new_element);
/* return 1 if the stack is empty, 0 otherwise */
int stack_isempty(Stack);
/* remove and return top element of stack */
/* returns STACK_EMPTY if stack is empty */
#define STACK_EMPTY (-1)
int stack_pop(Stack);
Our intent is that an Stack acts like a stack— we push things onto it using stack_push, and then pull them off again in reverse order using stack_pop. Ideally, we don’t ever pop the stack when it’s empty (which we can detect using stack_isempty), but if we do, we have stack_pop return something well-defined.
Test code
Let’s write some test code to try this out. Because our initial stack implementation may be exceptionally bug-ridden, we’ll use a test harness that provides macros for detecting and intercepting segmentation faults and similar disasters. The various testing wrappers are defined in the files tester.h and tester.c from earlier in this chapter; you should feel free to use it for your own purposes. I’ve added line numbers in comments to all the TEST lines so we can find them again later.
test-stack.c
#include <stdio.h>
#include <setjmp.h>
#include <signal.h>
#include <unistd.h>
#include <stdlib.h>
#include "stack.h"
#include "tester.h"
#define STRESS_TEST_ITERATIONS (1000000)
int
main(int argc, char **argv)
{
Stack s;
int i;
tester_init();
/* first we need to build one */
TRY { s = stack_create(); } ENDTRY;
/* 25 */ TEST_ASSERT(s != 0);
/* now we'll try pushing and popping a bit */
TRY { stack_push(s, 1); } ENDTRY;
TRY { stack_push(s, 2); } ENDTRY;
TRY { stack_push(s, 3); } ENDTRY;
/* 32 */ TEST(stack_isempty(s), 0);
/* 33 */ TEST(stack_pop(s), 3);
/* 34 */ TEST(stack_isempty(s), 0);
/* 35 */ TEST(stack_pop(s), 2);
/* 36 */ TEST(stack_isempty(s), 0);
/* 37 */ TEST(stack_pop(s), 1);
/* 38 */ TEST(stack_isempty(s), 1);
/* 39 */ TEST(stack_pop(s), STACK_EMPTY);
/* 40 */ TEST(stack_isempty(s), 1);
/* 41 */ TEST(stack_pop(s), STACK_EMPTY);
/* can we still push after popping too much? */
TRY { stack_push(s, 4); } ENDTRY;
/* 45 */ TEST(stack_isempty(s), 0);
/* 46 */ TEST(stack_pop(s), 4);
/* 47 */ TEST(stack_isempty(s), 1);
/* 48 */ TEST(stack_pop(s), STACK_EMPTY);
/* 49 */ TEST(stack_isempty(s), 1);
/* let's do some stress testing */
/* we won't use TEST for this because we might get too much output */
TRY {
for(i = 0; i < STRESS_TEST_ITERATIONS; i++) {
stack_push(s, i);
}
for(i = 0; i < STRESS_TEST_ITERATIONS; i++) {
stack_push(s, 957);
if(stack_pop(s) != 957) {
/* 60 */ FAIL("wanted 957 but didn't get it");
abort();
}
}
for(i = STRESS_TEST_ITERATIONS - 1; i >= 0; i--) {
if(stack_isempty(s)) {
/* 66 */ FAIL("stack empty too early");
abort();
}
if(stack_pop(s) != i) {
/* 70 */ FAIL("got wrong value!");
abort();
}
}
} ENDTRY; /* 74 */
/* 76 */ TEST(stack_isempty(s), 1);
TRY { stack_destroy(s); } ENDTRY;
tester_report(stdout, argv[0]);
return tester_result();
}
There is a lot of test code here. In practice, we might write just a few tests to start off with, and, to be honest, I didn’t write all of this at once. But you can never have too many tests— if nothing else, they give an immediate sense of gratification as the number of failed tests drops.
Makefile
- Finally, we’ll write a
Makefile:
Makefile
CC=gcc
CFLAGS=-g3 -Wall
all:
test: test-stack
./test-stack
@echo OK!
test-stack: test-stack.o tester.o stack.o
$(CC) $(CFLAGS) -o $@ $^
test-stack.o: stack.h tester.h
stack.o: stack.h
Note that we don’t provide a convenient shortcut for building test-stack without running it. That’s because we want to run the test code every single time.
Stub implementation
Of course, we still can’t compile anything, because we don’t have any implementation. Let’s fix that. To make it easy to write, we will try to add as little as possible to what we already have in stack.h:
stack.c
#include <stdlib.h>
#include "stack.h"
struct stack { int placeholder; };
Stack stack_create(void) { return malloc(sizeof(struct stack)); }
void stack_destroy(Stack s) { free(s); }
void stack_push(Stack s, int elem) { ; }
int stack_pop(Stack s) { return STACK_EMPTY; }
int stack_isempty(Stack s) { return 1; }
Will this work? Of course not. There’s hardly any code! But maybe it will compile if we run make test:
$ make test
gcc -g3 -Wall -c -o test-stack.o test-stack.c
gcc -g3 -Wall -c -o tester.o tester.c
gcc -g3 -Wall -c -o stack.o stack.c
gcc -g3 -Wall -o test-stack test-stack.o tester.o stack.o
./test-stack
test-stack.c:32: TEST FAILED: stack_isempty(s) -> 1 but expected 0
test-stack.c:33: TEST FAILED: stack_pop(s) -> -1 but expected 3
test-stack.c:34: TEST FAILED: stack_isempty(s) -> 1 but expected 0
test-stack.c:35: TEST FAILED: stack_pop(s) -> -1 but expected 2
test-stack.c:36: TEST FAILED: stack_isempty(s) -> 1 but expected 0
test-stack.c:37: TEST FAILED: stack_pop(s) -> -1 but expected 1
test-stack.c:45: TEST FAILED: stack_isempty(s) -> 1 but expected 0
test-stack.c:46: TEST FAILED: stack_pop(s) -> -1 but expected 4
test-stack.c:60: wanted 957 but didn't get it
test-stack.c:74: Aborted (signal 6)
./test-stack: errors 8/17, signals 1, FAILs 1
make[1]: *** [test] Error 8
Hooray! It compiles on the first try! (Well, not really, but let’s pretend it did.) Unfortunately, it only passes any tests at all by pure dumb luck. But now we just need to get the code to pass a few more tests.
Bounded-space implementation
Here’s a first attempt at a stack that suffers from some artificial limits. We retain the structure of the original broken implementation, we just put a few more lines of code in and format it more expansively.
stack.c
#include <stdlib.h>
#include "stack.h"
#define MAX_STACK_SIZE (100)
struct stack {
int top;
int data[MAX_STACK_SIZE];
};
Stack
stack_create(void)
{
struct stack *s;
s = malloc(sizeof(*s));
s->top = 0;
return s;
}
void
stack_destroy(Stack s)
{
free(s);
}
void
stack_push(Stack s, int elem)
{
s->data[(s->top)++] = elem;
}
int
stack_pop(Stack s)
{
return s->data[--(s->top)];
}
int
stack_isempty(Stack s)
{
return s->top == 0;
}
Let’s see what happens now:
$ make test
gcc -g3 -Wall -c -o test-stack.o test-stack.c
gcc -g3 -Wall -c -o tester.o tester.c
gcc -g3 -Wall -c -o stack.o stack.c
gcc -g3 -Wall -o test-stack test-stack.o tester.o stack.o
./test-stack
test-stack.c:40: TEST FAILED: stack_isempty(s) -> 0 but expected 1
test-stack.c:41: TEST FAILED: stack_pop(s) -> 409 but expected -1
test-stack.c:47: TEST FAILED: stack_isempty(s) -> 0 but expected 1
test-stack.c:48: TEST FAILED: stack_pop(s) -> 0 but expected -1
test-stack.c:49: TEST FAILED: stack_isempty(s) -> 0 but expected 1
test-stack.c:74: Segmentation fault (signal 11)
test-stack.c:76: TEST FAILED: stack_isempty(s) -> 0 but expected 1
free(): invalid pointer 0x804b830!
./test-stack: errors 6/17, signals 1, FAILs 0
make[1]: *** [test] Error 6
There are still errors, but we get past several initial tests before things blow up. Looking back at the line numbers in test-stack.c, we see that the first failed test is the one that checks if the stack is empty after we pop from an empty stack. The code for stack_isempty looks pretty clean, so what happened? Somewhere s->top got set to a nonzero value, and the only place this can happen is inside stack_pop. Aha! There’s no check in stack_pop for an empty stack, so it’s decrementing s->top past 0. (Exercise: why didn’t the test of stack_pop fail?)
First fix
If we’re lucky, fixing this problem will make the later tests happier. Let’s try a new version of stack_pop. We’ll leave everything else the same.
int
stack_pop(Stack s)
{
if(stack_isempty(s)) {
return STACK_EMPTY;
} else {
return s->data[--(s->top)];
}
}
And now we get:
$ make test
gcc -g3 -Wall -c -o test-stack.o test-stack.c
gcc -g3 -Wall -c -o tester.o tester.c
gcc -g3 -Wall -c -o stack.o stack.c
gcc -g3 -Wall -o test-stack test-stack.o tester.o stack.o
./test-stack
test-stack.c:74: Segmentation fault (signal 11)
test-stack.c:76: TEST FAILED: stack_isempty(s) -> 0 but expected 1
./test-stack: errors 1/17, signals 1, FAILs 0
make[1]: *** [test] Error 1
Which is much nicer. We are still failing the stress test, but that’s not terribly surprising.
Final version
After some more tinkering, this is what I ended up with. This version uses a malloc’d data field, and realloc’s it when the stack gets too big.
stack.c
#include <stdlib.h>
#include "stack.h"
struct stack {
int top; /* first unused slot in data */
int size; /* number of slots in data */
int *data; /* stack contents */
};
#define INITIAL_STACK_SIZE (1)
#define STACK_SIZE_MULTIPLIER (2)
Stack
stack_create(void)
{
struct stack *s;
s = malloc(sizeof(*s));
if(s == 0) return 0;
s->top = 0;
s->size = INITIAL_STACK_SIZE;
s->data = malloc(s->size * sizeof(*(s->data)));
if(s->data == 0) return 0;
/* else everything is ok */
return s;
}
void
stack_destroy(Stack s)
{
free(s->data);
free(s);
}
void
stack_push(Stack s, int elem)
{
if(s->top == s->size) {
/* need more space */
s->size *= STACK_SIZE_MULTIPLIER;
s->data = realloc(s->data, s->size * sizeof(*(s->data)));
if(s->data == 0) {
abort(); /* we have no other way to signal failure :-( */
}
}
/* now there is enough room */
s->data[s->top++] = elem;
}
int
stack_pop(Stack s)
{
if(stack_isempty(s)) {
return STACK_EMPTY;
} else {
return s->data[--(s->top)];
}
}
int
stack_isempty(Stack s)
{
return s->top == 0;
}
At last we have a version that passes all tests:
$ make test
gcc -g3 -Wall -c -o test-stack.o test-stack.c
gcc -g3 -Wall -c -o tester.o tester.c
gcc -g3 -Wall -c -o stack.o stack.c
gcc -g3 -Wall -o test-stack test-stack.o tester.o stack.o
./test-stack
OK!
Moral
Writing a big program all at once is hard. If you can break the problem down into little problems, it becomes easier. “Test first” is a strategy not just for getting a well-tested program, but for giving you something easy to do at each step— it’s usually not too hard to write one more test, and it’s usually not too hard to get just one test working. If you can keep taking those small, easy steps, eventually you will run out of failed tests and have a working program.
Appendix: Test macros
/*
* Test macros.
*
* Usage:
*
* #include <setjmp.h>
* #include <stdio.h>
* #include <signal.h>
* #include <unistd.h>
*
* testerInit(); -- Initialize internal data structures.
* testerReport(FILE *, "name"); -- Print report.
* testerResult(); -- Returns # of failed tests.
*
* TRY { code } ENDTRY;
*
* Wraps code to catch seg faults, illegal instructions, etc. May not be
* nested.
* Prints a warning if a signal is caught.
* To enforce a maximum time, set alarm before entering.
*
* TEST(expr, expected_value);
*
* Evaluates expr (which should yield an integer value) inside a TRY.
* Prints a warning if evaluating expr causes a fault or returns a value
* not equal to expected_value.
*
* TEST_ASSERT(expr)
*
* Equivalent to TEST(!(expr), 0)
*
* You can also cause your own failures with FAIL:
*
* TRY {
* x = 1;
* if(x == 2) FAIL("why is x 2?");
* } ENDTRY;
*
* To limit the time taken by a test, call tester_set_time_limit with
* a new limit in seconds, e.g.
*
* tester_set_time_limit(1);
* TRY { while(1); } ENDTRY;
*
* There is an initial default limit of 10 seconds.
* If you don't want any limit, set the limit to 0.
*
*/
/* global data used by macros */
/* nothing in here should be modified directly */
extern struct tester_global_data {
jmp_buf escape_hatch; /* jump here on surprise signals */
int escape_hatch_active; /* true if escape hatch is usable */
int tests; /* number of tests performed */
int errors; /* number of tests failed */
int signals; /* number of signals caught */
int expr_value; /* expression value */
int setjmp_return; /* return value from setjmp */
int try_failed; /* true if last try failed */
int user_fails; /* number of calls to FAIL */
int time_limit; /* time limit for TRY */
} TesterData;
/* set up system; call this before using macros */
void testerInit(void);
/* prints a summary report of all errors to f, prefixed with preamble */
/* If there were no errors, nothing is printed */
void testerReport(FILE *f, const char *preamble);
/* returns number of errors so far. */
int testerResult(void);
/* set a time limit t for TRY, TEST, TEST_ASSERT etc. */
/* After t seconds, an ALARM signal will interrupt the test. */
/* Set t = 0 to have no time limit. */
/* Default time limit is 10 seconds. */
void tester_set_time_limit(int t);
const char *testerStrsignal(int); /* internal hack; don't use this */
/* gruesome non-syntactic macros */
#define TRY \
TesterData.try_failed = 0; \
alarm(TesterData.time_limit); \
if(((TesterData.setjmp_return = setjmp(TesterData.escape_hatch)) == 0) \
&& (TesterData.escape_hatch_active = 1) /* one = is correct*/)
#define ENDTRY else { \
fprintf(stderr, "%s:%d: %s (signal %d)\n", \
__FILE__, __LINE__, \
testerStrsignal(TesterData.setjmp_return), \
TesterData.setjmp_return); \
TesterData.signals++; \
TesterData.try_failed = 1; \
} \
alarm(0); \
TesterData.escape_hatch_active = 0
/* another atrocity */
#define TEST(expr, expected_value) \
TesterData.tests++; \
TesterData.errors++; /* guilty until proven innocent */ \
TRY { TesterData.expr_value = (expr); \
if(TesterData.expr_value != expected_value) { \
fprintf(stderr, "%s:%d: TEST FAILED: %s -> %d but expected %d\n", \
__FILE__, __LINE__, __STRING(expr), \
TesterData.expr_value, expected_value); \
} else { \
TesterData.errors--; \
} \
} \
ENDTRY; \
if(TesterData.try_failed) \
fprintf(stderr, "%s:%d: TEST FAILED: %s caught signal\n", \
__FILE__, __LINE__, __STRING(expr))
#define TEST_ASSERT(expr) TEST((expr) != 0, 1)
#define FAIL(msg) \
(fprintf(stderr, "%s:%d: %s\n", __FILE__, __LINE__, (msg)), \
TesterData.user_fails++, \
TesterData.try_failed = 1)
#define _GNU_SOURCE /* get strsignal def */
#include <stdio.h>
#include <signal.h>
#include <string.h>
#include <setjmp.h>
#include "tester.h"
struct tester_global_data TesterData;
const char *
testerStrsignal(int sig)
{
return strsignal(sig);
}
static void
tester_sighandler(int signal)
{
if(TesterData.escape_hatch_active) {
TesterData.escape_hatch_active = 0;
longjmp(TesterData.escape_hatch, signal);
}
}
void
testerInit(void)
{
TesterData.escape_hatch_active = 0;
TesterData.tests = 0;
TesterData.errors = 0;
TesterData.signals = 0;
TesterData.user_fails = 0;
signal(SIGSEGV, tester_sighandler);
signal(SIGILL, tester_sighandler);
signal(SIGFPE, tester_sighandler);
signal(SIGALRM, tester_sighandler);
signal(SIGBUS, tester_sighandler);
signal(SIGABRT, tester_sighandler);
}
void
testerReport(FILE *f, const char *preamble)
{
if(TesterData.errors != 0 || TesterData.signals != 0) {
fprintf(f, "%s: errors %d/%d, signals %d, FAILs %d\n",
preamble,
TesterData.errors,
TesterData.tests,
TesterData.signals,
TesterData.user_fails);
}
}
int
testerResult(void)
{
return TesterData.errors;
}
void
tester_set_time_limit(int t)
{
TesterData.time_limit = t;
}
#/media/File:Tallrik_-_Ystad-2018.jpg){kind=link}