Test Case
Figure 1.1 portrays a life cycle model for testing. Notice that, in the development phases, there are three opportunities for errors to be made, resulting in faults that propagate through the remainder of the development. One prominent tester summarizes this life cycle as follows: the first three phases are “Putting Bugs IN”, the testing phase is Finding Bugs, and the last three phases are “Getting Bugs OUT” The Fault Resolution step is another opportunity for errors (and new faults). When a “fix” causes formerly correct software to misbehave, the fix is deficient. We’ll revisit this when we discuss regression testing. From this sequence of terms, we see that test cases occupy a central position in testing. The process of testing can be subdivided into separate steps: test planning, test case development, running test cases, and evaluating test results. The focus of this book is how to identify useful sets of test cases.
A test case has an identity, and is associated with a program behavior. A test case also has a set of
inputs, a list of expected outputs.
Figure 1.1 portrays a life cycle model for testing. Notice that, in the development phases, there are three opportunities for errors to be made, resulting in faults that propagate through the remainder of the development. One prominent tester summarizes this life cycle as follows: the first three phases are “Putting Bugs IN”, the testing phase is Finding Bugs, and the last three phases are “Getting Bugs OUT” The Fault Resolution step is another opportunity for errors (and new faults). When a “fix” causes formerly correct software to misbehave, the fix is deficient. We’ll revisit this when we discuss regression testing. From this sequence of terms, we see that test cases occupy a central position in testing. The process of testing can be subdivided into separate steps: test planning, test case development, running test cases, and evaluating test results. The focus of this book is how to identify useful sets of test cases.
Test Cases
The essence of software testing is to determine a set of test cases for the item being tested. Before
going on, we need to clarify what information should be in a test case. The most obvious
information is inputs; inputs are really of two types: pre-conditions (circumstances that hold prior to
test case execution) and the actual inputs that were identified by some testing method. The next most
obvious part of a test case is the expected outputs; again, there are two types: post conditions and
actual outputs. The output portion of a test case is frequently overlooked. Unfortunate, because this is
often the hard part. Suppose, for example, you were testing software that determined an optimal route
for an aircraft, given certain FAA air corridor constraints and the weather data for a
flight day.
How would you know what the optimal route really is? There have been various responses to this
problem. The academic response is to postulate the existence of an oracle, who “knows all the
answers”. One industrial response to this problem is known as Reference Testing, where the system is
tested in the presence of expert users, and these experts make judgments as to whether or not outputs
of an executed set of test case inputs are acceptable. The act of testing entails establishing the
necessary pre-conditions, providing the test case inputs, observing the outputs, and then comparing
these with the expected outputs to determine whether or not the test passed. The
remaining information in a well-developed test case primarily supports testing management. Test
cases should have an identity, and a reason for being (requirements tracing is a fine reason). It is also
useful to record the execution history of a test case, including when and by whom it was run, the
pass/fail result of each execution, and the version (of software) on which it was run. From all of this,
it should be clear that test cases are valuable — at least as valuable as source code. Test cases need to
be developed, reviewed, used, managed, and saved.
Insights from a Venn diagram
Testing is fundamentally concerned with behavior; and behavior is orthogonal to the structural view
common to software (and system) developers. A quick differentiation is that the structural view
focuses on “what it is” and the behavioral view considers “what it does”. One of the continuing
sources of difficulty for testers is that the base documents are usually written by and for developers,
and therefore the emphasis is on structural, rather than behavioral, information. In this section, we
develop a simple Venn diagram which clarifies several nagging questions about testing.
Consider a Universe of program behaviors. (Notice that we are forcing attention on the essence of
testing.) Given a program and its specification, consider the set S of specified behaviors, and the set P
of programmed behaviors. Figure 1.3 shows the relationship between our universe of discourse and
the specified and programmed behaviors. Of all the possible program behaviors, the specified
ones are in the circle labeled S; and all those behaviors actually programmed
note the slight difference between P and U, the Universe) are in P. With this diagram, we can see
more clearly the problems that confront a tester. What if there are specified behaviors that have
programmed? In our earlier terminology, these are faults of omission. Similarly, what if there are
programmed (implemented) behaviors that have not been specified? These correspond to faults of
commission, and to errors which occurred after the specification was complete. The intersection of S
and P (the football shaped region) is the “correct” portion, that is behaviors that are both specified
and implemented. A very good view of testing is that it is the determination of the extent of
program behavior that is both specified and implemented. (As a sidelight, note that “correctness” only
has meaning with respect to a specification and an implementation. It is a relative term, not an
absolute.)
The new circle in Fig. 1.3 is for Test Cases. Notice there is a slight discrepancy with our Universe
of Discourse, the set of program behaviors. Since a test case causes a program behavior, the
mathematicians might forgive us. Now, consider the relationships among the sets S, P, and T. There
may be specified behaviors that are not tested (regions 2 and 5), specified behaviors that are tested
(regions 1 and 4), and test cases that correspond to unspecified behaviors (regions 3 and 7).
Similarly, there may be programmed behaviors that are not tested (regions 2 and 6), programmed
behaviors that are tested (regions 1 and 3), and test cases that correspond to unprogrammed behaviors
(regions 4 and 7). Each of these regions is important. If there are specified behaviors for which there
are no test cases, the testing is necessarily incomplete. If there are test cases that correspond to
unspecified behaviors, two possibilities arise: either such a test case is unwarranted, or the
specification is deficient. (In my experience, good testers often postulate test cases of this latter type.
This is a fine reason to have good testers participate in specification and design reviews.) We are
already at a point where we can see some possibilities for testing as a craft: what can a tester do to
make the region where these sets all intersect (region 1) be as large as possible? Another way to get at
this is to ask how the test cases in the set T are identified. The short answer is that test cases are
identified a testing method. This framework gives us a way to compare the effectiveness of diverse
testing methods.
Identifying Test Cases
There are two fundamental approaches to identifying test cases; these are known as functional and
structural testing. Each of these approaches has several distinct test case identification methods, more
commonly called testing methods.
Functional Testing
Functional testing is based on the view that any program can be considered to be a function that maps
values from its input domain to values in its output range. (Function, domain, and range are defined in
Chapter 3.) This notion is commonly used in engineering, when systems are considered to be “black
boxes”. This leads to the term Black Box Testing, in which the content (implementation) of a black
box is not known, and the function of the black box is understood completely in terms of its inputs
and outputs. In Zen and The Art of Motorcycle Maintenance , Pirsig refers to this as “romantic”
comprehension Many times, we operate very effectively with black box knowledge; in fact this is
central to object orientation. As an example, most people successfully operate automobiles with only
black box knowledge.
With the functional approach to test case identification, the only information that is used is the
specification of the software. There are two distinct advantages to functional test cases: they are
independent of how the software is implemented, so if the implementation changes, the test cases are
still useful, and test case development can occur in parallel with the implementation, thereby reducing
overall project development interval. On the negative side, functional test cases frequently suffer from
two problems: there can be significant redundancies among test cases, and this is compounded by the
possibility of gaps of untested software. Figure 1.6 shows the results of test cases identified by two
functional methods. Method A identifies a larger set of test cases than does Method B. Notice that, for
both methods, the set of test cases is completely contained within the set of specified behavior. Since
functional methods are based on the specified behavior, it is hard to imagine these methods
identifying behaviors that are not specified.
Structural Testing
Structural testing is the other fundamental approach to test case identification. To contrast it with
Functional Testing, it is sometimes called White Box (or even Clear Box) Testing. The clear box
metaphor is probably more appropriate, because the essential difference is that the implementation (of
the Black Box) is known and used to identify test cases. Being able to “see inside” the black
boxallows het tester to identify test cases based on how the function is actually implemented.
Structural Testing has been the subject of some fairly strong theory. To really understand structural
testing, the concepts of linear graph theory are essential. With these concepts, the tester can rigorously describe exactly what is being tested. Because of its strong theoretical basis, structural
testing lends itself to the definition and use of test coverage metrics. Test coverage metrics provide a
way to explicitly state the extent to which a software item has been tested, and this in turn, makes
testing management more meaningful.
This is dummy text. It is not meant to be read. Accordingly, it is difficult to figure out when to end it. But then, this is dummy text. It is not meant to be read. Period.
ConversionConversion EmoticonEmoticon