Class 6: Testing and Debugging

Class 6: Testing and Debugging#

Today’s assignment#

Sign up for the assignment. We start with the example C++ code in the repository’s top-level directory.

Build integration with the IDE#

So far, we have called cmake by hand (on the command line), and that’s how we ran our compiled executables, too. This is definitely a useful skill to become familiar with – but it’s also worth noting that the “I” in IDE is for “Integrated”, and IDEs usually integrate features to compile, run and debug your code as well.

In how far you want to use that in everyday work is up to you. For VS Code, I would like to note that it does have integrated cmake support (and also various extensions, including “CMake Tools” from Microsoft, which can be useful). This means I can actually build, compile, and even debug the code by just clicking some buttons in the status bar. If you build code through VS Code, rather than manually on the command line, that’ll also allow VS Code to see and interpret warning and error messages.

More on VS Code and cmake is here.

Introduction to testing / debugging#

Testing and debugging are actually closely related. That is, if you test your code as you develop it, you may have a lot less debugging to do later :smirk:.

In general, it is almost never a good idea to write large pieces of code without testing them as you go.

“Test early, test often” is a popular mantra in the Linux community, and for good reason. Another one: “Premature optimization is the root of all evil”.

Testing and debugging can and should happen at many different levels:

  • Your editor / IDE

  • The compiler (errors and warnings)

  • Unit tests

  • Integration Tests

  • Also: End-to-end tests, performance tests, …

  • [more on debugging to follow later…]

The editor / IDE#

Your editor: Your editor, through syntax highlighting, automatic indentation, etc. can show that something is wrong before you’re even done typing it. Modern IDEs are even better, as they actually analyze code as you type and alert you if you’re calling a function that doesn’t exist, or calling it with the wrong kind of arguments, etc.

[Note: In the assignment repo, I added a .clang-format file. This is a config file for the clang-format tool, which auto-formats C, C++ and various other languages. I find it rather useful, and you’re of course welcome to install/use it, too. VS Code has support for it.]

The compiler#

If you mess something up badly enough, your compiler will throw an error (or dozens of them), trying to tell you what the problem is. In general, you want to start at the top of the list trying to understand what’s wrong, since subsequent errors can be misleading as they are caused by the compiler being confused from that first mistake.

In addition to real errors, the compiler can also warn you that while some construct is legal, you probably meant to write something else. These days, compilers tend to give useful warnings by default already, but it may be worthwhile to add more useful warnings, enabled by -Wall. As usual, the man page for the compiler (man gcc) will tell you a lot more.

Example of Warnings#

On my system (using clang), I get a bunch of warnings from today’s sample code:

vscode ➜ /workspaces/class-6/build (main) $ make
[ 20%] Building CXX object CMakeFiles/stuff.dir/greeting.cxx.o
[ 40%] Building CXX object CMakeFiles/stuff.dir/factorial.cxx.o
/workspaces/class-6/factorial.cxx:6:9: warning: using the result of an assignment as a condition without parentheses [-Wparentheses]
    6 |   if (n = 0) {
      |       ~~^~~
/workspaces/class-6/factorial.cxx:6:9: note: place parentheses around the assignment to silence this warning
    6 |   if (n = 0) {
      |         ^  
      |       (    )
/workspaces/class-6/factorial.cxx:6:9: note: use '==' to turn this assignment into an equality comparison
    6 |   if (n = 0) {
      |         ^
      |         ==
/workspaces/class-6/factorial.cxx:11:1: warning: non-void function does not return a value [-Wreturn-type]
   11 | }
      | ^
2 warnings generated.
[ 60%] Linking CXX static library libstuff.a
[ 60%] Built target stuff
[ 80%] Building CXX object CMakeFiles/hello.dir/hello.cxx.o
[100%] Linking CXX executable hello
[100%] Built target hello

The code also doesn’t exactly work all that great…

Adding compiler flags with cmake#

The following is a quick and dirty way to manipulate compiler flags with cmake. It requires a specific user intervention, rather than happening automatically which is not all that great, but it’ll do for now, and it’ll show you how to manually experiment with flags. Instead of simply calling cmake -S .., you can set all kinds of options when calling cmake, in particular

vscode ➜ /workspaces/class-6/build (main) $ cmake -DCMAKE_CXX_FLAGS="-Wall" -S ..
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /workspaces/class-6/build

Depending on your compiler, you might already get all the relevant warnings without adding -Wall. But it doesn’t hurt to learn how to set custom flags.

If you want to learn more and do things less ad-hoc, here’s an interesting article.

In-class exercise#

  • What warning messages are you getting?

  • What do they mean?

  • How do you fix this code?

Fix the code and commit the changes.

Note: This is an area where AI tools can be useful, including the integrated “Copilot” support in VS Code. As a general note, if you use AI tools (or just google), that’s perfectly okay, but please state so in your homework notes. Also, I strongly recommend not going overboard – AI can be very useful for helping understand what’s going on and how to fix it. It can do the fixing for you, too, but that is really something you should be able to do by yourself.

Unit tests#

Unit tests are tests designed to test one particular piece of your code, a “unit”. This may well be one function, or maybe one class. It’s often important to cover posssible corner cases as well.

A unit test for the factorial function might be as simple as

  printf("4! = %d\n", factorial(4));
  printf("1! = %d\n", factorial(1));
  printf("0! = %d\n", factorial(0));

[or you might use C++’s std::cout] This isn’t really all that great, though, since it not only requires you to run the test by hand, but you also need to look at the output and check that it’s correct (and remember for the factorial function is defined for a zero argument…)

So a relatively simple way to improve on this is to do this:

  assert(factorial(4) == 24);
  assert(factorial(1) == 1);
  assert(factorial(0) == 1);
  // we don't have a good way to handle negative arguments, so we don't.
  // Hence it doesn't make sense to test this case, either. FIXME

Fortran doesn’t have anything quite like this built in (and really, I’m abusing assertions above, anyway). But you could do something like

  if (factorial(4) .ne. 24) stop 'error: factorial(4) does not return 24'

In-class exercise#

  • Create a test_factorial executable that tests correctness of the factorial function as shown above.

What would be even better is for those tests to run automatically - clearly, it’s not a big deal if you just have a single test, but once you have a bunch, it’d be nice to have a single command run them all. cmake has its own testing framework, called ctest. This is how it’s done:

This is how to do automated tests in cmake:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index c9dd95b..bb11b5a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -2,7 +2,13 @@ cmake_minimum_required(VERSION 3.16)
 
 project(hello LANGUAGES C CXX)
 
+enable_testing()
+
 add_library(stuff greeting.cxx factorial.cxx hello.h)
 
 add_executable(hello hello.cxx)
 target_link_libraries(hello PRIVATE stuff)
+
+add_executable(test_factorial test_factorial.cxx)
+target_link_libraries(test_factorial stuff)
+add_test(FactorialTest test_factorial)

[The actual testing code test_factorial.cxx also needs to be added – see above.]

To run the test(s), use the ctest command in your build directory:

➜  build git:(class7s) ✗ ctest .
Test project /Users/kai/class/iam851/iam851/build
    Start 1: FactorialTest
1/1 Test #1: FactorialTest ....................***Exception: SegFault  0.06 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.07 sec

The following tests FAILED:
          1 - FactorialTest (SEGFAULT)
Errors while running CTest
Output from these tests are in: /Users/kai/class/iam851/iam851/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

In-class exercise#

Try out ctest .. If you’ve fixed the factorial() bugs previously, it should succeed, otherwise you’ll at least get confirmation that it’s still broken.

More complex (integration, end-to-end) testing#

Sometimes, you want to not just test units, but make sure those units work together properly, and of course you can create tests for that, too. This is highly problem-specific, and maybe overkill for most cases in scientific computing – Test your application as it gets completed instead. Keeping track of test cases you can run with your application as you go is very useful, even if it involves a manual process.

Testing unfortunately does not avoid debugging completely, but it may help narrowing down where things go wrong. We’ll talk about debugging more next time.

Homework#

  • Finish the in-class exercises for the factorial function testing and fixing. As usual, commit as you go and keep notes to be added to the Feedback PR. Since this is a group assignment, you probably should coordinate who does what and when, unless you also want to practice make branches, separate pull requests, merging, etc.

  • Here’s a catch on (ab)using assert for testing. Build your factorial unit test with cmake -DCMAKE_BUILD_TYPE=Debug [...] and verify that the testing works as intended (ie., tests that are supposed to pass, pass; tests that are supposed to fail, fail). Now do so again with -DCMAKE_BUILD_TYPE=Release. What do you observe? Challenge: Can you figure out why?

    If you use VS Code for building your code you can instead use “CMake: Variant” from the command palette.

  • We didn’t actually get there today, but we will next week, so: If you don’t already know them, figure out the scoring rules for the game of ten-pin bowling (spares, strikes, etc).

  • Read Barely Sufficient Software Engineering Take note if you see something in there that relates to your prior experiences or that you find particularly interesting.

  • Read the Googletest primer through “Simple Tests”.

  • Sign up for a presentation here.