Why testing?
Tests are part of every software development. The question is when and where they are executed.
In the worst case, the software is tested by the customer using it. This is a rather expensive and impractical tests, since the cost is the chance of losing customers and reputation. Also, their feedback is not standardized and partially difficult to interpret, that’s why we strive to test the software before it reaches the customer.
For us as developers, tests also simplify the debugging process, since they contain implicit information about the state and functionality of the code, thereby allowing us to localize undesired behaviors faster.
What to test?
Before we discuss how we test, we should define what we desire to test. In general, our program acts on input and delivers some output or controls some variable. So, we can imagine our program as a mathematical function that is applied to an input vector and returns an output vector. I haven’t encountered a problem defining this framing yet, but I had to include physical objects like robotic arms and screws into it.
We can now split our program in different sub functions taking each other’s output and external parameters as input, forming a possible cyclic graph.
How to test?
After having visualized our program and its parts as functions, we can derive our tests directly from that. Our tests simply secure that our software implementation maps the desired input to the desired output. In consequence, we start by defining these mappings. In practice, we will quickly discover, that we cannot test for all values, so we have to use sample values. Depending on the problem, the software tests might be randomly generated or have to be crafted by hand.
To randomly generate test cases, we have to either find a simple inverse function or an easy to test subset. Both of these approaches are dangerous, since our function may fail outside the subset, or we may fail to correctly calculate the inverse function. A perfect example would be a square-root finder function. To test it we can simply square any number, hand it to the function and check if the result is within (truncation)-error-range of the function’s output.
Levels of testing
According to our abstraction, we will have functions contained within functions. This directly leads us to a test hierarchy or a test tree. It is useful to abstract these software tests and sort them into groups, for example:
-
small unit tests: testing every function,
-
more comprehensive unit tests: testing entire components,
-
integration tests: testing the interaction between different components,
-
comprehensive system and deployment tests: testing the entire project.
Some of these tests will be small and easy to perform and should therefore be automated. While others require more complex input or a physical test setup and have to be done by hand. The aim of the automated tests should be to catch errors before they escalate into larger undesired behavior. The manual tests should catch whatever slipped before.
The three-party problem
Unfortunately, we will often find ourselves unable to test the entire project, since the code is controlled by a third party instead of us. Thankfully, this does not change the way we test, just the scale of the test. We may now be forced to test a much more complex component like a robot, a neural network or a camera.
This can be complex and challenging, but we can use the fact, that these objects are outside our control to a certain extent, by assuming they are static. Meaning we have to test their behavior only once before we ship the product, after this test is concluded we may assume they keep behaving that way.
Dos and Don’ts
Here are some general tips for your tests:
-
Make sure your tests are repeatable and predictable. So, if they fail, you can reproduce the failure. This is mostly relevant for randomized tests.
-
Design your tests before your function, module or program to avoid biasing your test towards your implementation.
-
Design your tests independently, so the result of one test does not contaminate the result of others.
-
Attempt to test broad.
-
If a bug occurs for the first time, write a test to automatically find this type of bug in the future.
If you feel inspired by this guide or wish to learn more, I advise to read “Growing Object-Oriented Software, Guided by Tests” by Steve Freeman and Nat Pryce.
