BBT Strategies: Test Early and Often

Test Early and Often:

As was said in the beginning of the chapter, executing your test cases as soon as possible is an excellent way of getting concrete feedback about your program. In order to run test cases early, programmers need to integrate the pieces of their code into the code base often. Programmers could be tempted to work on their own computer until the finish implementing a “whole” requirement. In industry, this could quite feasibly mean they keep their code to themselves for several months. However, this is a dangerous practice – and can lead to what is known in industry as integration hell. Just because a component works on a programmer’s own computer, this doesn’t mean it will work when it is assembled with the code other programmers are working on. The earlier it is known that there are some interface problems or some data that’s not getting passed properly the better. This knowledge can only be gained by integrating code and testing early and often. Then, integration problems can be more easily localized in the work that was just integrated. By localizing the code that contains a new defect, the programmer can efficiently identify and remove defects.

BBT Strategies: Failure (“Dirty”) Test Cases

Donald Knuth is many times referred to as one of the fathers of computer science. He is also known as a stickler when it comes to bugs in his code (and in his books. He sends checks to readers who find errors in his books!). Anticipating the unexpected is one of his techniques. Think the way Knuth does when you write your test cases. Be mean and nasty!
My test programs are intended to break the system, to push it to its extreme limits, to pile complication on complication, in ways that the system programmer never consciously anticipated. To prepare such test data, I get into the meanest, nastiest frame of mind that I can manage, and I write the cruelest code I can think of; then I turn around and embed that in even nastier constructions that are almost obscene.

Think diabolically! Think of every possible thing a user could possibly do with your system to demolish the software. You need to make sure your program is robust – in that it can properly respond in the face of erroneous user input. This type of testing is called robustness testing, whereby test cases are chosen outside the domain to test robustness to unexpected, erroneous input, and is included in defensive testing which includes tests under both normal and abnormal conditions. Look at every input. Does the program respond “gracefully” to these error conditions?

• Can any form of input to the program cause division by zero? Get creative!
• What if the input type is wrong? (You’re expecting an integer, they input a float. You’re expecting a character, you get an integer.)
• What if the customer takes an illogical path through your functionality?
• What if mandatory fields are not entered?
• What if the program is aborted abruptly or input or output devices are unplugged?

BBT Strategies: Decision Table Testing

Decision Table Testing:

Decision tables are used to record complex business rules that must be implemented in the program, and therefore tested. A sample decision table is found in Table 4. In the table, the conditions represent possible input conditions. The actions are the events that should trigger, depending upon the makeup of the input conditions. Each column in the table is a unique combination of input conditions (and is called a rule) that result in triggering the action(s) associated with the rule. Each rule (or column) should become a test case.

If a Player (A) lands on property owned by another player (B), A must pay rent to B. If A does not have enough money to pay B, A is out of the game.



Decision table

BBT Strategies: Boundary Value Analysis

Boundary Value Analysis

Boris Beizer, well-known author of testing book advises, “Bugs lurk in corners and congregate at boundaries.” Programmers often make mistakes on the boundaries of the equivalence classes/input domain. As a result, we need to focus testing at these boundaries. This type of testing is called Boundary Value Analysis (BVA) and guides you to create test cases at the “edge” of the equivalence classes. Boundary value is defined as a data value that corresponds to a minimum or maximum input, internal, or output value specified for a system or component. In our above example, the boundary of the class is at 50, as shown in Figure 3. We should create test cases for the Player 1 having $49, $50, and $51. These test cases will help to find common off-by-one errors, caused by errors like using >= when you mean to use >.


Boundary Value Analysis. Test cases should be created for the boundaries (arrows) between equivalence classes

When creating BVA test cases, consider the following:

• If input conditions have a range from a to b (such as a=100 to b=300), create test cases:
  

1. immediately below a (99)
2. at a (100)
3. immediately above a (101)
4. immediately below b (299)
5. at b (300)
6. immediately above b (301)

• If input conditions specify a number of values that are allowed, test these limits. For example, input conditions specify that only one train is allowed to start in each direction on each station. In testing, try to add a second train to the same station/same direction. If (somehow) three trains could start on one station/direction, try to add two trains (pass), three trains (pass), and four trains (fail).

BBT Strategies: Equivalence Partitioning

Equivalence Partitioning:

To keep down our testing costs, we don’t want to write several test cases that test the same aspect of our program. A good test case uncovers a different class of errors (e.g., incorrect processing of all character data) than has been uncovered by prior test cases.

Equivalence partitioning is a strategy that can be used to reduce the number of test cases that need to be developed. Equivalence partitioning divides the input domain of a program into classes. For each of these equivalence classes, the set of data should be treated the same by the module under test and should produce the same answer. Test cases should be designed so the inputs lie within these equivalence classes. For example, for tests of “Go to Jail” the most important thing is whether the player has enough money to pay the $50 fine. Therefore, the two equivalence classes can be partitioned, as shown in Figure.


Equivalence Classes for Player Money

Once you have identified these partitions, you choose test cases from each partition. To start, choose a typical value somewhere in the middle of (or well into) each of these two ranges. See Table 6 for test cases written to test the equivalent classes of money. However, you will note that Test Cases 6 (Player 1 has $1200) and 7 (Player 1 has $100) are both in the same equivalence class. Therefore, Test Case 7 is unlikely to discover any defect not found in Test Case 6.


Test Plan #2 for the Jail Requirement

For each equivalent class, the test cases can be defined using the following guidelines:

• If input conditions specify a range of values, create one valid and one or two invalid equivalence classes. In the above example, this is (1) less than 50/invalid; (2) 50 or more/valid.
• If input conditions require a certain value (for example R and L for the side in our train example), create an equivalence class of the valid values (R and L) and one of invalid values (all other letters other than R and L). In this case, you need to test all valid values individually and several invalid values.
• If input conditions specify a member of a set, create one valid and one invalid equivalence class.
• If an input condition is a Boolean, define one valid and one invalid class.

Equivalence class partitioning is just the start, though. An important partner to this partitioning is boundary value analysis.

Strategies for Black Box Testing

Ideally, we’d like to test every possible thing that can be done with our program. But, as we said, writing and executing test cases is expensive. We want to make sure that we definitely write test cases for the kinds of things that the customer will do most often or even fairly often. Our objective is to find as many defects as possible in as few test cases as possible. To accomplish this objective, we use some strategies that will be discussed in this subsection. We want to avoid writing redundant test cases that won’t tell us anything new (because they have similar conditions to other test cases we already wrote). Each test case should probe a different mode of failure. We also want to design the simplest test cases that could possibly reveal this mode of failure – test cases themselves can be error-prone if we don’t keep this in mind.

Tests of Customer Requirements

Black box test cases are based on customer requirements. We begin by looking at each customer requirement. To start, we want to make sure that every single customer requirement has been tested at least once. As a result, we can trace every requirement to its test case(s) and every test case back to its stated customer requirement. The first test case we’d write for any given requirement is the most-used success path for that requirement. By success path, we mean that we want to execute some desirable functionality (something the customer wants to work) without any error conditions. We proceed by planning more success path test cases, based on other ways the customer wants to use the functionality and some test cases that execute failure paths. Intuitively, failure paths intentionally have some kind of errors in them, such as errors that users can accidentally input. We must make sure that the program behaves predictably and gracefully in the face of these errors. Finally, we should plan the execution of our tests out so that the most troublesome, risky requirements are tested first. This would allow more time for fixing problems before delivering the product to the customer. It would be devastating to find a critical flaw right before the product is due to be delivered. We’ll start with one basic requirement. We can write many test cases based on this one requirement, which follows below. As we’ve said before, it is impossible to test every single possible combination of input. We’ll outline an incomplete sampling of test cases and reason about them in this section.

Requirement: When a user lands on the “Go to Jail” cell, the player goes directly to jail, does not pass go, does not collect $200. On the next turn, the player must pay $50 to get out of jail and does not roll the dice or advance. If the player does not have enough money, he or she is out of the game.

There are many things to test in this short requirement above, including:

• Does the player get sent to jail after landing on “Go to Jail”?
• Does the player receive $200 if “Go” is between the current space and jail?
• Is $50 correctly decremented if the player has more than $50?
• Is the player out of the game if he or she has less than $50?

At first it is good to start out by testing some input that you know should definitely pass or definitely fail. If these kinds of tests don’t work properly, you know you should just quit testing and put the code back into development. We can start with a two obvious passing test case, as shown in Table

Test Plan #1 for the Jail Requirement

You will also note that we should test the simplest possible means to force the condition we are trying to achieve. For example, in Test Case 5, we only have one player so we temporarily didn’t have to spend our time with Player 2. We add Player 2 in Test Case 6 so we can observe that the loss of $50 and dice roll occurs on the next turn (after Player 2 goes). We could go on and test many more aspects of the above requirement. We will now discuss some strategies to consider in creating more test cases.

Testing Techniques: Black Box Technique

Black box testing, also called functional testing and behavioral testing, focuses on determining whether or not a program does what it is supposed to do based on its functional requirements. Black box testing attempts to find errors in the external behavior of the code in the following categories:

• Incorrect or missing functionality;
• Interface errors;
• Errors in data structures used by interfaces;
• Behavior or performance errors; and
• Initialization and termination errors.

Through this testing, we can determine if the functions appear to work according to specifications. However, it is important to note that no amount of testing can unequivocally demonstrate the absence of errors and defects in your code. It is best if the person who plans and executes black box tests is not the programmer of the code and does not know anything about the structure of the code. The programmers of the code are innately biased and are likely to test that the program does what they programmed it to do. What are needed are tests to make sure that the program does what the customer wants it to do. As a result, most organizations have independent testing groups to perform black box testing. These testers are not the developers and are often referred to as third-party testers. Testers should just be able to understand and specify what the desired output should be for a given input into the program, as shown in Figure.



A black-box test takes into account only the input and output of the software without regard to the internal code of the program