Back to TOC Features

Coverage Testing with a C/C++ Profiler

Paolo Argenton

There's lots of neat stuff packed in with gcc. It even helps you do a pretty respectable job of test coverage analysis.

The Need for Coverage Testing

I've always been intrigued by code coverage testing, perhaps due to a form of imprinting: the very first time I tried this technique — on the most frequently used program in my company — I immediately found a first-class bug. I quickly learned a lesson from this experience and started applying what I had learned.

Coverage testing belongs to the so-called white-box, or structural testing methods. They are so called because they use knowledge about the internals of the program under scrutiny.

The rationale behind coverage testing is rather simple. You check the percentage of code being exercised during tests, to know how much is actually tested, and especially, to discover untested code. The code that needs the most rigorous testing is not always obvious. For instance, error-handling code has a higher percentage of bugs than mainstream code, because during normal program usage it is not executed. Therefore, it is more likely to contain undetected errors.

The simplest form of coverage testing is that of function-level coverage. It determines which functions in your code have been called and which haven't. While you may be confident from looking at your code that every function is being processed, program flow is sometimes counter-intuitive. Or, you may simply forget to test something using non-automated tests.

Consider the code fragment below, which is similar to the program of my first test:

...
     
/* argument processing:
   lot of options here */
while((ch = getopt(argc,argv,
  "u:t:r:m:SCB:ifn:p:RT:"   ))
   != EOF )
{
  switch( ch )
  {
     case 'u':   
     /* username */
...     
     case 'C':    
/* very rarely used option */
     enable_canc = TRUE;
     break;
...
   }
}
...
if ( enable_canc )
    do_something();
    /* very seldom executed */
...

Here it seems obvious from the context that do_something was insufficiently tested. Even so, I can attest that its bugs went unobserved for a long time.

Another Use for the Profiler

Now let's see what can be done with available tools to improve the testing process. While I know of no C/C++ compiler that comes integrated with coverage testing tools, most C/C++ compilers probably come with a profiler. Instead of using the profiler to find hot spots — that is, the functions that use most of the CPU time — you can use it the other way round, to find the functions never called.

As an example I demonstrate the use of GNU gprof, since GNU programs are widely available and free, but other compilers have similar tools. Although I did all the sample programs and tests on a PC running Linux, there should be very few operating system dependencies.

Suppose you want to test the simple program given in Listing 1. First you compile it with some special options:

$ gcc -pg -o test01 test01.c

Then you run it:

$ test01

And then you can analyze the profile data. The following command creates a report file named gmon.out:

$ gprof -b -z test01 gmon.out

The output should be similar to that given in Table 1. You can safely ignore strange looking functions, such as __do_global_ctors_aux, and all the data after the form feed character. To weed out this extraneous data I wrote the awk filter shown in Listing 2. (You'll have to adapt the string filtering if you use this technique with different systems.)

The command becomes:

$ gprof -b -z test01 gmon.out | awk -f ignoref.awk

Now, looking at the results in Table 2, you can quickly see that f001 was never called.

Basic-Block Testing

At this point the goal is to achieve 100% function level coverage. In other words, you want to make sure every function is called at least once. You can accomplish this by adding other tests and using gprof's cumulative option. But knowing which functions are called guarantees nothing about what happens inside the functions. You can easily imagine a function like the one below:

               
int buggy()
{  if ( low_probability )
  {
    /* buggy code here */
  }
  else
    /* correct code here */
}

To adequately test functions such as this you must go beyond 100% function level coverage. You must check that every possible path within every function is being executed.

There are several opinions on the best way to proceed. For instance, you could test that every Boolean condition in the if statements is tested at least once for true and false values. This technique is called branch, or decision testing.

One idea I find pretty natural and easy to develop is to check the execution of every block of code. This enables me to test every possible execution path. I say this idea is "easy to develop" because, from this level of detail on, it is very difficult to find or adapt existing tools. Luckily I discovered, almost by chance, a feature in the gcc compiler named basic-block profiling.

Let's say, for the sake of simplicity, that a basic block is a sequence of statements without internal branching constructs. gcc's -a option allows you to profile your program in a manner similar to the -pg option, but at basic-block level instead of function level. When you run a program compiled this way you will find, after execution, a bb.out file. gcc always writes to this file by appending, a feature that comes in handy for executing multiple tests.

gcc creates this file via its runtime library. If you want to dig in the code, the function name is __bb_exit_func in file libgcc2.c A sample bb.out is shown in Table 3. The more interesting fields are the line number, which marks where the basic block begins; a count of how many times the block was executed; the file name; and the function name.

Given all of the above, here's a typical way to use basic-block programming manually:

Compile the program. (The -g option is important; it enables output of file name and line number information.)

$ gcc -g -a -o test01 test01.c

Delete bb.out, if it exists:

$ rm bb.out

Run the program:

$ test01

Now you can see the results by looking in the bb.out file. You can find out by inspection which blocks were never executed. But of course, since the file could reflect the results of multiple tests, this method is pretty impractical. So I developed a little awk program that prints out only the blocks that were never executed throughout the whole set of tests.

The awk program in Listing 3 is straightforward; the only thing to note is an interesting feature of arrays in awk: you can use a string as the index of the array. The index I use is a string made by concatenating the file name and line number. For every line of data read in from bb.out the awk program increments a counter. After the program reaches the end of the file it prints the elements of the array whose values are still zero, that is, the blocks that were never executed in the whole set of tests. The program also prints some statistics.

Now, from the beginning: you want to test the program test01. Compile it with:

$ gcc -g -a -o test01 test01

Then run it:

$ rm bb.out
$ test01

Now you condition the test results with:

$ gawk -f pblock.awk bb.out

Sample output appears in Table 4

The last number appearing in Table 4 is the percentage of code coverage. If you are unsatisfied with this number you must look at the former lines, which are the line numbers of blocks never executed, and find a way to exercise them. In this trivial example you should add a test like the following:

$ test01 other-argument-supplied

The sample output is:

00008 blocks out of 00008 executed; 100% coverage

Now if you look at the results you will be happy to see 100% code coverage.

Conclusion

Of course this home-built tool can not compete with commercial packages; however, it has some nice properties:

References

Roger S. Pressman. Software Engineering: A Practitioner's Approach (McGraw-Hill, 1996) ISBN 0-07-052182-4.

Paolo Argenton is a freelance consultant and programmer. He has been programming in C and C++ since 1983. His interests include software quality assurance, reliability, and safety. Paolo can be reached at paolo.argenton@iol.it.