ಬಿ.ಎಂ.ಎಸ್.

ತಂತ್ರಿ ಕ ಮತ್ತು ವ್ಯ ವ್ಸ್ಥಾ ಪನಾ ಮಹಾವಿದ್ಯಯ ಲಯ

BMS Institute of Technology and Management
(An Autonomous Institution Affiliated to VTU, Belagavi)
Avalahalli, Doddaballapur Main Road, Bengaluru – 560064

Department of Information Science and Engineering

Course Name: Software Engineering and Project Semester: V
Management
Course Code: BCS501 Faculty: Dr. Harish Kumar N
Module 4
Software Testing: Development testing, Test-driven development, Release testing, User testing

Testing is intended to show that a program does what it is intended to do and to discover program
defects before it is put into use. When you test software, you execute a program using artificial data.
You check the results of the test run for errors, anomalies, or information about the program’s non-
functional attributes.

When you test software, you are trying to do two things:

1. Demonstrate to the developer and the customer that the software meets its requirements. For
custom software, this means that there should be at least one test for every requirement in the
requirements document. For generic software products, it means that there should be tests for all of
the system features that will be included in the product release.

2. Find inputs or input sequences where the behavior of the software is incorrect, undesirable, or
does not conform to its specification. These are caused by defects (bugs) in the software. When you
test software to find defects, you are trying to root out undesirable system behavior such as system
crashes, unwanted interactions with other systems, incorrect computations, and data corruption

The first of these is validation testing, where you expect the system to perform correctly using a set
of test cases that reflect the system’s expected use. The second is defect testing, where the test cases
are designed to expose defects. The test cases in defect testing can be deliberately obscure and need
not reflect how the system is normally used.

Testing is part of a broader process of software verification and validation (V & V).
Verification and validation are not the same thing, although they are often confused.

■ Validation: Are we building the right product?

■ Verification: Are we building the product right?

Verification and validation processes are concerned with checking that software being developed
meets its specification and delivers the functionality expected by the people paying for the
software. These checking processes start as soon as requirements become available and continue
through all stages of the development process.
Software verification is the process of checking that the software meets its stated functional and
non-functional requirements. Validation is a more general process. The aim of software validation
is to ensure that the software meets the customer’s expectations. It goes beyond checking
conformance with the specification to demonstrating that the software does what the customer expects
it to do.
The goal of verification and validation processes is to establish confidence that the software system is
“fit for purpose.”

Figure 8.1 shows the differences between validation testing and defect testing. Think of the system
being tested as a black box. The system accepts inputs from some input set I and generates outputs in
an output set O. Some of the outputs will be erroneous. These are the outputs in set Oe that are
generated by the system in response to inputs in the set Ie. The priority in defect testing is to find
those inputs in the set Ie because these reveal problems with the system. Validation testing
involves testing with correct inputs that are outside Ie.

Software purpose The more critical the software, the more important it is that it is reliable. For
example, the level of confidence required for software used to control a safety-critical system is much
higher than that required for a demonstrator system that prototypes new product ideas.

User expectations Because of their previous experiences with buggy, unreliable software, users
sometimes have low expectations of software quality. They are not surprised when their software fails.
When a new system is installed, users may tolerate failures because the benefits of use outweigh the
costs of failure recovery.

Marketing environment When a software company brings a system to market, it must take into
account competing products, the price that customers are willing to pay for a system, and the required
schedule for delivering that system

As well as software testing, the verification and validation process may involve software inspections
and reviews. Inspections and reviews analyze and check the system requirements, design models, the
program source code, and even proposed system tests.

Figure 8.2 shows that software inspections and testing support V & V at different stages in the software
process. The arrows indicate the stages in the process where the techniques may be used.
Inspections mostly focus on the source code of a system, but any readable representation of the
software, such as its requirements or a design model, can be inspected. When you inspect a system,
you use knowledge of the system, its application domain, and the programming or modeling
language to discover errors.
Software inspection has three advantages over testing:
1. During testing, errors can mask (hide) other errors. When an error leads to unexpected outputs, you
can never be sure if later output anomalies are due to a new error or are side effects of the original
error.
2. Incomplete versions of a system can be inspected without additional costs. If a program is
incomplete, then you need to develop specialized test harnesses to test the parts that are available.
3. As well as searching for program defects, an inspection can also consider broader quality
attributes of a program, such as compliance with standards, portability, and maintainability.

Inspections cannot replace software testing. Inspections are not good for discovering defects that
arise because of unexpected interactions between different parts of a program, timing problems, or
problems with system performance

A model of the software testing process

Figure 8.3 is an abstract model of the traditional testing process, as used in plan driven development.
Test cases are specifications of the inputs to the test and the expected output from the system (the
test results), plus a statement of what is being tested. Test data are the inputs that have been devised
to test a system. Test data can sometimes be generated automatically, but automatic test case
generation is impossible.
People who understand what the system is supposed to do must be involved to specify the expected
test results. However, test execution can be automated. The test results are automatically compared
with the predicted results, so there is no need for a person to look for errors and anomalies in the test
run.

Typically, a commercial software system has to go through three stages of testing:

1. Development testing, where the system is tested during development to discover bugs and
defects. System designers and programmers are likely to be involved in the testing process.
2. Release testing, where a separate testing team tests a complete version of the system before it is
released to users. The aim of release testing is to check that the system meets the requirements of
the system stakeholders.
3. User testing, where users or potential users of a system test the system in their own environment.
For software products, the “user” may be an internal marketing group that decides if the software
can be marketed, released and sold.
Acceptance testing is one type of user testing where the customer formally tests a system to decide
if it should be accepted from the system supplier or if further development is required.

The testing process usually involves a mixture of manual and automated testing. In manual testing,
a tester runs the program with some test data and compares the results to their expectations.
They note and report discrepancies to the program developers. In automated testing, the tests are
encoded in a program that is run each time the system under development is to be tested. This is
faster than manual testing, especially when it involves regression testing.

Development testing

 Development testing includes all testing activities that are carried out by the team developing
the system.
 The tester of the software is usually the programmer who developed that software.
 Some development processes use programmer/tester pairs where each programmer has an
associated tester who develops tests and assists with the testing process.
 There are three stages of development testing:
 Unit testing, where individual program units or object classes are tested. Unit testing
should focus on testing the functionality of objects or methods.
 Component testing, where several individual units are integrated to create
composite components. Component testing should focus on testing the component
interfaces that provide access to the component functions.
 System testing, where some or all of the components in a system are integrated and
the system is tested as a whole. System testing should focus on testing component
interactions.
 Development testing is primarily a defect testing process, where the aim of testing is to discover
bugs in the software

Unit testing
 Unit testing is the process of testing program components, such as methods or object classes.
 Individual functions or methods are the simplest type of component.
When you are testing object classes, you should design your tests to provide coverage of all of the
features of the object.
This means that you should test all operations associated with the object; set and check the value of
all attributes associated with the object; and put the object into all possible states

Consider, for example, the weather station object.

The attributes and operations of this object are shown in Figure 8.4. It has a single attribute, which is
its identifier. This is a constant that is set when the weather station is installed. You therefore only need
a test that checks if it has been properly set up. You need to define test cases for all of the methods
associated with the object such as reportWeather and reportStatus.

To test the states of the weather station, you can use a state model. Using this model, you identify
sequences of state transitions that have to be tested and define event sequences to force these
transitions. In principle, you should test every possible state transition sequence.
Examples of state sequences that should be tested in the weather station include:
Shutdown → Running → Shutdown
Configuring → Running → Testing → Transmitting → Running
Running → Collecting → Running → Summarizing → Transmitting → Running

Whenever possible, you should automate unit testing. In automated unit testing, you make use of a test
automation framework, such as JUnit to write and run your program tests. Unit testing frameworks
provide generic test classes that you extend to create specific test cases.

An automated test has three parts:

1. A setup part, where you initialize the system with the test case, namely, the inputs and expected
outputs.
2. A call part, where you call the object or method to be tested.
3. An assertion part, where you compare the result of the call with the expected result. If the
assertion evaluates to true, the test has been successful; if false, then it has failed.

Choosing unit test cases

Testing is expensive and time consuming, so it is important that you choose effective unit test cases.
Effectiveness, in this case, means two things:
1. The test cases should show that, when used as expected, the component that you are testing does
what it is supposed to do.
2. If there are defects in the component, these should be revealed by test cases.

Two strategies that can be effective in helping you choose test cases are:

1. Partition testing, where you identify groups of inputs that have common characteristics and
should be processed in the same way. You should choose tests from within each of these groups.
2. Guideline-based testing, where you use testing guidelines to choose test cases. These guidelines
reflect previous experience of the kinds of errors that programmers often make when developing
components.
  In Figure 8.5, the large shaded ellipse on the left represents the set of all possible inputs to the
program that is being tested.
 The smaller unshaded ellipses represent equivalence partitions. A program being tested
should process all of the members of an input equivalence partition in the same way.
 Output equivalence partitions are partitions within which all of the outputs have something
in common. Once you have identified a set of partitions, you choose test cases from each of
these partitions.
 A good rule of thumb for test-case selection is to choose test cases on the boundaries of the
partitions, plus cases close to the midpoint of the partition.
 The reason for this is that designers and programmers tend to consider typical values of inputs
when developing a system.
 Equivalence partitioning is an effective approach to testing because it helps account for errors
that programmers often make when processing inputs at the edges of partitions.
 You can also use testing guidelines to help choose test cases.

Component testing
Software components are often made up of several interacting objects. For example, in the weather
station system, the reconfiguration component includes objects that deal with each aspect of the
reconfiguration. You access the functionality of these objects through component interfaces.

Figure 8.7 illustrates the idea of component interface testing. Assume that components A, B, and C
have been integrated to create a larger component or subsystem.
The test cases are not applied to the individual components but rather to the interface of the
composite component created by combining these components. Interface errors in the composite
component may not be detectable by testing the individual objects because these errors result from
interactions between the objects in the component

There are different types of interface between program components and, consequently, different
types of interface error that can occur:
1. Parameter interfaces These are interfaces in which data or sometimes function references are
passed from one component to another. Methods in an object have a parameter interface.
2. Shared memory interfaces These are interfaces in which a block of memory is shared between
components. Data is placed in the memory by one subsystem and retrieved from there by other
subsystems. This type of interface is used in embedded systems, where sensors create data that is
retrieved and processed by other system components.
3. Procedural interfaces These are interfaces in which one component encapsulates a set of
procedures that can be called by other components. Objects and reusable components have this
form of interface.
4. Message passing interfaces These are interfaces in which one component requests a service from
another component by passing a message to it. A return message includes the results of executing
the service. Some object-oriented systems have this form of interface, as do client–server systems.

Interface errors are one of the most common forms of error in complex systems. These errors fall into
three classes:
■ Interface misuse A calling component calls some other component and makes an error in the
use of its interface. This type of error is common in parameter interfaces, where parameters may be of
the wrong type or be passed in the wrong order, or the wrong number of parameters may be passed.

■ Interface misunderstanding A calling component misunderstands the specification of the

interface of the called component and makes assumptions about its behavior. The called component
does not behave as expected, which then causes unexpected behavior in the calling component. For
example, a binary search method may be called with a parameter that is an unordered array. The search
would then fail.
■ Timing errors These occur in real-time systems that use a shared memory or a message-passing
interface. The producer of data and the consumer of data may operate at different speeds. Unless
particular care is taken in the interface design, the consumer can access out-of-date information
because the producer of the information has not updated the shared interface information.

Some general guidelines for interface testing are:

1. Examine the code to be tested and identify each call to an external component. Design a set of tests
in which the values of the parameters to the external components are at the extreme ends of their ranges.
These extreme values are most likely to reveal interface inconsistencies.
2. Where pointers are passed across an interface, always test the interface with null pointer parameters.
3. Where a component is called through a procedural interface, design tests that deliberately cause the
component to fail. Differing failure assumptions are one of the most common specification
misunderstandings.
4. Use stress testing in message passing systems. This means that you should design tests that generate
many more messages than are likely to occur in practice. This is an effective way of revealing timing
problems.
5. Where several components interact through shared memory, design tests that vary the order in which
these components are activated. These tests may reveal implicit assumptions made by the programmer
about the order in which the shared data is produced and consumed.
System testing
System testing during development involves integrating components to create a version of the
system and then testing the integrated system. System testing checks that components are
compatible, interact correctly, and transfer the right data at the right time across their interfaces. It
obviously overlaps with component testing, but there are two important differences:

1. During system testing, reusable components that have been separately developed and off-the-
shelf systems may be integrated with newly developed components. The complete system is then
tested.

2. Components developed by different team members or sub teams may be integrated at this stage.
System testing is a collective rather than an individual process. In some companies, system testing may
involve a separate testing team with no involvement from designers and programmers.

System testing should focus on testing the interactions between the components and objects that
make up a system. You may also test reusable components or systems to check that they work as
expected when they are integrated with new components. This interaction testing should discover those
component bugs that are only revealed when a component is used by other components in the system.

In the wilderness weather station example, the system software reports summarized weather data
to a remote computers. The sequence of operations in the weather station when it responds to a
request to collect data for the mapping system. You can use this diagram to identify operations that
will be tested and to help design the test cases to execute the tests. Therefore issuing a request for a
report will result in the execution of the following thread of methods:
SatComms:request → WeatherStation:reportWeather → Commslink:Get(summary) →
WeatherData:summarize

The sequence diagram helps you design the specific test cases that you need, as it shows what inputs
are required and what outputs are created:
1. An input of a request for a report should have an associated acknowledgment. A report should
ultimately be returned from the request. During testing, you should create summarized data that can be
used to check that the report is correctly organized.
2. An input request for a report to WeatherStation results in a summarized report being generated. You
can test this in isolation by creating raw data corresponding to the summary that you have prepared for
the test of SatComms and checking that the WeatherStation object correctly produces this summary.
This raw data is also used to test the WeatherData object.

For most systems, it is difficult to know how much system testing is essential and when you should
stop testing. Exhaustive testing, where every possible program execution sequence is tested, is
impossible. Testing, therefore, has to be based on a subset of possible test cases. Ideally, software
companies should have policies for choosing this subset.

1. All system functions that are accessed through menus should be tested.
2. Combinations of functions (e.g., text formatting) that are accessed through the same menu must
be tested.
3. Where user input is provided, all functions must be tested with both correct and incorrect input.

Automated system testing is usually more difficult than automated unit or component testing.
Automated unit testing relies on predicting the outputs and then encoding these predictions in a
program. The prediction is then compared with the result.

Release testing

Release testing is the process of testing a particular release of a system that is intended for use outside
of the development team. Normally, the system release is for customers and users. In a complex project,
however, the release could be for other teams that are developing related systems.

There are two important distinctions between release testing and system testing during the development
process:
1. The system development, team should not be responsible for release testing.
2. Release testing is a process of validation checking to ensure that a system meets its requirements
and is good enough for use by system customers.
System testing by the development team should focus on discovering bugs in the system (defect
testing).
The primary goal of the release testing process is to convince the supplier of the system that it is good
enough for use. If so, it can be released as a product or delivered to the customer. Release testing,
therefore, has to show that the system delivers its specified functionality, performance, and
dependability, and that it does not fail during normal use.
Release testing is usually a black-box testing process whereby tests are derived from the system
specification.
The system is treated as a black box whose behaviour can only be determined by studying its inputs
and the related outputs. Another name for this is functional testing, so-called because the tester is only
concerned with functionality and not the implementation of the software.

Requirements-based testing
A general principle of good requirements engineering practice is that requirements should be testable.
That is, the requirement should be written so that a test can be designed for that requirement. A tester
can then check that the requirement has been satisfied. Requirements-based testing, therefore, is a
systematic approach to test-case design where you consider each requirement and derive a set of tests
for it. Requirements-based testing is validation rather than defect testing—you are trying to
demonstrate that the system has properly implemented its requirements.

For example, consider the following Mentcare system requirements that are concerned with checking
for drug allergies:
If a patient is known to be allergic to any particular medication, then prescription of that medication
shall result in a warning message being issued to the system user. If a prescriber chooses to ignore an
allergy warning, he or she shall provide a reason why this has been ignored.
To check if these requirements have been satisfied, you may need to develop several related tests:
1. Set up a patient record with no known allergies. Prescribe medication for allergies that are known
to exist. Check that a warning message is not issued by the system.
2. Set up a patient record with a known allergy. Prescribe the medication that the patient is allergic to
and check that the warning is issued by the system.
3. Set up a patient record in which allergies to two or more drugs are recorded. Prescribe both of these
drugs separately and check that the correct warning for each drug is issued.
4. Prescribe two drugs that the patient is allergic to. Check that two warnings are correctly issued.
5. Prescribe a drug that issues a warning and overrule that warning. Check that the system requires the
user to provide information explaining why the warning was overruled.

Scenario testing
Scenario testing is an approach to release testing whereby you devise typical scenarios of use and use
these scenarios to develop test cases for the system.
A scenario is a story that describes one way in which the system might be used. Scenarios should be
realistic, and real system users should be able to relate to them. If you have used scenarios or user
stories as part of the requirements engineering process then you may be able to reuse them as testing
scenarios.

This scenario tests a number of features of the Mentcare system:

1. Authentication by logging on to the system.
2. Downloading and uploading of specified patient records to a laptop.
3. Home visit scheduling.
4. Encryption and decryption of patient records on a mobile device.
5. Record retrieval and modification.
6. Links with the drugs database that maintains side-effect information.
7. The system for call prompting.

When you use a scenario-based approach, you are normally testing several requirements within the
same scenario. Therefore, as well as checking individual requirements, you are also checking that
combinations of requirements do not cause problems.

Performance testing

 Once a system has been completely integrated, it is possible to test for emergent properties,
such as performance and reliability.
 Performance tests have to be designed to ensure that the system can process its intended load.
 This usually involves running a series of tests where you increase the load until the system
performance becomes unacceptable.
 Performance testing is concerned both with demonstrating that the system meets its
requirements and discovering problems and defects in the system.
 To test whether performance requirements are being achieved, you may have to construct an
operational profile.

In performance testing, this means stressing the system by making demands that are outside the design
limits of the software. This is known as stress testing.

Say you are testing a transaction processing system that is designed to process up to 300 transactions
per second. You start by testing this system with fewer than 300 transactions per second. You then
gradually increase the load on the system beyond 300 transactions per second until it is well beyond
the maximum design load of the system and the system fails.

Stress testing helps you do two things:

1. Test the failure behavior of the system. Circumstances may arise through an unexpected combination
of events where the load placed on the system exceeds the maximum anticipated load. Stress testing
checks that overloading the system causes it to “fail-soft” rather than collapse under its load. Reveal
defects that only show up when the system is fully loaded. Although it can be argued that these defects
are unlikely to cause system failures in normal use, there may be unusual combinations of
circumstances that the stress testing replicates.

Stress testing is particularly relevant to distributed systems based on a network of processors. These
systems often exhibit severe degradation when they are heavily loaded. The network becomes
swamped with coordination data that the different processes must exchange. The processes become
slower and slower as they wait for the required data from other processes. Stress testing helps you
discover when the degradation begins so that you can add checks to the system to reject transactions
beyond this point.

8.4 User testing

User or customer testing is a stage in the testing process in which users or customers provide input and
advice on system testing. This may involve formally testing a system that has been commissioned from
an external supplier User testing is essential, even when comprehensive system and release testing
have been carried out. Influences from the user’s working environment can have a major effect on the
reliability, performance, usability, and robustness of a system.
There are three different types of user testing:
1. Alpha testing, where a selected group of software users work closely with the development team to
test early releases of the software.
2. Beta testing, where a release of the software is made available to a larger group of users to allow
them to experiment and to raise problems that they discover with the system developers.
3. Acceptance testing, where customers test a system to decide whether or not it is ready to be accepted
from the system developers and deployed in the customer environment.

In alpha testing, users and developers work together to test a system as it is being developed. This
means that the users can identify problems and issues that are not readily apparent to the development
testing team. Developers can only really work from the requirements, but these often do not reflect
other factors that affect the practical use of the software.
Alpha testing is often used when developing software products or apps. Experienced users of these
products may be willing to get involved in the alpha testing process because this gives them early
information about new system features that they can exploit. It also reduces the risk that unanticipated
changes to the software will have disruptive effects on their business. alpha testing may also be used
when custom software is being developed.

Beta testing takes place when an early, sometimes unfinished, release of a software system is made
available to a larger group of customers and users for evaluation. Beta testers may be a selected group
of customers who are early adopters of the system. Alternatively, the software may be made publicly
available for use by anyone who is interested in experimenting with it.

Beta testing is mostly used for software products that are used in many different settings. This is
important as, unlike custom product developers, there is no way for the product developer to limit the
software’s operating environment. It is impossible for product developers to know and replicate all the
settings in which the software product will be used. Beta testing is therefore used to discover interaction
problems between the software and features of its operational environment. Beta testing is also a form
of marketing. Customers learn about their system and what it can do for them.

Acceptance testing is an inherent part of custom systems development. Customers test a system, using
their own data, and decide if it should be accepted from the system developer. Acceptance implies that
final payment should be made for the software.

There are six stages in the acceptance testing process

1. Define acceptance criteria This stage should ideally take place early in the process before the
contract for the system is signed. The acceptance criteria should be part of the system contract and be
approved by the customer and the developer. Detailed requirements may not be available, and the
requirements will almost certainly change during the development process
2. Plan acceptance testing This stage involves deciding on the resources, time, and budget for
acceptance testing and establishing a testing schedule. The acceptance test plan should also discuss the
required coverage of the requirements and the order in which system features are tested. It should
define risks to the testing process such as system crashes and inadequate performance, and discuss how
these risks can be mitigated.
3. Derive acceptance tests Once acceptance criteria have been established, tests have to be designed
to check whether or not a system is acceptable. Acceptance tests should aim to test both the functional
and non-functional characteristics (e.g., performance) of the system. They should ideally provide
complete coverage of the system requirements. In practice, it is difficult to establish completely
objective acceptance criteria. There is often scope for argument about whether or not a test shows that
a criterion has definitely been met.
4. Run acceptance tests The agreed acceptance tests are executed on the system. Ideally, this step
should take place in the actual environment where the system will be used, but this may be disruptive
and impractical. Therefore, a user testing environment may have to be set up to run these tests. It is
difficult to automate this process as part of the acceptance tests may involve testing the interactions
between end-users and the system.
5. Negotiate test results It is very unlikely that all of the defined acceptance tests will pass and that
there will be no problems with the system. If this is the case, then acceptance testing is complete and
the system can be handed over. More commonly, some problems will be discovered. In such cases, the
developer and the customer have to negotiate to decide if the system is good enough to be used. They
must also agree on how the developer will fix the identified problems.
6. Reject/accept system This stage involves a meeting between the developers and the customer to
decide on whether or not the system should be accepted. If the system is not good enough for use, then
further development is required to fix the identified problems. Once complete, the acceptance testing
phase is repeated

If a system does not pass its acceptance tests, then it should not be accepted and payment should not
be made. However, the reality is more complex. Customers want to use the software as soon as they
can because of the benefits of its immediate deployment. They may have bought new hardware, trained
staff, and changed their processes.