Copyright Notice

This document may be copied in its entirety, or extracts made, if the source is acknowledged.
Copyright Notice International Software Testing Qualifications Board (hereinafter called ISTQB) ISTQB is a registered trademark
of the International Software Testing Qualifications Board,
Copyright 2011 the authors for the update 2011 (Thomas Mller (chair), Debra Friedenberg, and the ISTQB WG Foundation Level)
Copyright 2010 the authors for the update 2010 (Thomas Mller (chair), Armin Beer, Martin Klonk, Rahul Verma)
Copyright 2007 the authors for the update 2007 (Thomas Mller (chair), Dorothy Graham, Debra Friedenberg and Erik van
Veenendaal)
Copyright 2005, the authors (Thomas Mller (chair), Rex Black, Sigrid Eldh, Dorothy Graham, Klaus Olsen, Maaret Pyhjrvi, Geoff
Thompson and Erik van Veenendaal).
All rights reserved.
The authors hereby transfer the copyright to the International Software Testing Qualifications Board (ISTQB). The authors (as current
copyright holders) and ISTQB (as the future copyright holder) have agreed to the following conditions of use:
1) Any individual or training company may use this syllabus as the basis for a training course if the authors and the ISTQB are
acknowledged as the source and copyright owners of the syllabus and provided that any advertisement of such a training course may
mention the syllabus only after submission for official accreditation of the training materials to an ISTQB recognized National Board.
2) Any individual or group of individuals may use this syllabus as the basis for articles, books, or other derivative writings if the authors
and the ISTQB are acknowledged as the source and copyright owners of the syllabus.
3) Any ISTQB-recognized National Board may translate this syllabus and license the syllabus (or its translation) to other parties.
Digital book(s) (epub and mobi) produced by Booknook.biz under the direction of Rex Black and Lois Kostroski of the ASTQB.

Revision History
Version

Date

Remarks

Effective 1-Apr-2011

Certified Tester Foundation Level Syllabus

Maintenance Release see Appendix E
Release Notes

ISTQB 2010

Effective 30-Mar-2010

Certified Tester Foundation Level Syllabus

Maintenance Release see Appendix E
Release Notes

ISTQB 2007

01-May-2007

Certified Tester Foundation Level Syllabus

Maintenance Release

ISTQB 2005

01-July-2005

Certified Tester Foundation Level Syllabus

ASQF V2.2

July-2003

ASQF Syllabus Foundation Level Version 2.2

Lehrplan Grundlagen des Software-testens

ISEB V2.0

25-Feb-1999

ISEB Software Testing Foundation Syllabus V2.0

25 February 1999

ISTQB 2011

Table of Contents
Acknowledgements
E-book Search Instructions
Introduction to this Syllabus
Purpose of this Document
The Certified Tester Foundation Level in Software Testing
Learning Objectives/Cognitive Level of Knowledge
The Examination
Accreditation
Level of Detail
How this Syllabus is Organized
1. Fundamentals of Testing (K2)
1.1 Why is Testing Necessary (K2)
1.1.1 Software Systems Context (K1)
1.1.2 Causes of Software Defects (K2)
1.1.3 Role of Testing in Software Development, Maintenance and Operations (K2)
1.1.4 Testing and Quality (K2)
1.1.5 How Much Testing is Enough? (K2)
1.2 What is Testing? (K2)
1.3 Seven Testing Principles (K2)
1.4 Fundamental Test Process (K1)
1.4.1 Test Planning and Control (K1)
1.4.2 Test Analysis and Design (K1)
1.4.3 Test Implementation and Execution (K1)
1.4.4 Evaluating Exit Criteria and Reporting (K1)
1.4.5 Test Closure Activities (K1)
1.5 The Psychology of Testing (K2)
1.6 Code of Ethics
2. Testing Throughout the Software Life Cycle (K2)
2.1 Software Development Models (K2)
2.1.1 V-model (Sequential Development Model) (K2)
2.1.2 Iterative-incremental Development Models (K2)
2.1.3 Testing within a Life Cycle Model (K2)
2.2 Test Levels (K2)
2.2.1 Component Testing (K2)

2.2.2 Integration Testing (K2)

2.2.3 System Testing (K2)
2.2.4 Acceptance Testing (K2)
2.3 Test Types (K2)
2.3.1 Testing of Function (Functional Testing) (K2)
2.3.2 Testing of Non-functional Software Characteristics (Non-functional Testing) (K2)
2.3.3 Testing of Software Structure/Architecture (Structural Testing) (K2)
2.3.4 Testing Related to Changes: Re-testing and Regression Testing (K2)
2.4 Maintenance Testing (K2)
3. Static Techniques (K2)
3.1 Static Techniques and the Test Process (K2)
3.2 Review Process (K2)
3.2.1 Activities of a Formal Review (K1)
3.2.2 Roles and Responsibilities (K1)
3.2.3 Types of Reviews (K2)
3.2.4 Success Factors for Reviews (K2)
3.3 Static Analysis by Tools (K2)
4. Test Design Techniques (K4)
4.1 The Test Development Process (K3)
4.2 Categories of Test Design Techniques (K2)
4.3 Specification-based or Black-box Techniques (K3)
4.3.1 Equivalence Partitioning (K3)
4.3.2 Boundary Value Analysis (K3)
4.3.3 Decision Table Testing (K3)
4.3.4 State Transition Testing (K3)
4.3.5 Use Case Testing (K2)
4.4 Structure-based or White-box Techniques (K4)
4.4.1 Statement Testing and Coverage (K4)
4.4.2 Decision Testing and Coverage (K4)
4.4.3 Other Structure-based Techniques (K1)
4.5 Experience-based Techniques (K2)
4.6 Choosing Test Techniques (K2)
5. Test Management (K3)
5.1 Test Organization (K2)
5.1.1 Test Organization and Independence (K2)
5.1.2 Tasks of the Test Leader and Tester (K1)
5.2 Test Planning and Estimation (K3)
5.2.1 Test Planning (K2)

5.2.2 Test Planning Activities (K3)

5.2.3 Entry Criteria (K2)
5.2.4 Exit Criteria (K2)
5.2.5 Test Estimation (K2)
5.2.6 Test Strategy, Test Approach (K2)
5.3 Test Progress Monitoring and Control (K2)
5.3.1 Test Progress Monitoring (K1)
5.3.2 Test Reporting (K2)
5.3.3 Test Control (K2)
5.4 Configuration Management (K2)
5.5 Risk and Testing (K2)
5.5.1 Project Risks (K2)
5.5.2 Product Risks (K2)
5.6 Incident Management (K3)
6. Tool Support for Testing (K2)
6.1 Types of Test Tools (K2)
6.1.1 Tool Support for Testing (K2)
6.1.2 Test Tool Classification (K2)
6.1.3 Tool Support for Management of Testing and Tests (K1)
6.1.4 Tool Support for Static Testing (K1)
6.1.5 Tool Support for Test Specification (K1)
6.1.6 Tool Support for Test Execution and Logging (K1)
6.1.7 Tool Support for Performance and Monitoring (K1)
6.1.8 Tool Support for Specific Testing Needs (K1)
6.2 Effective Use of Tools: Potential Benefits and Risks (K2)
6.2.1 Potential Benefits and Risks of Tool Support for Testing (for all tools) (K2)
6.2.2 Special Considerations for Some Types of Tools (K1)
6.3 Introducing a Tool into an Organization (K1)
7. References
Standards
Books
8. Appendix A Syllabus Background
History of this Document
Objectives of the Foundation Certificate Qualification
Objectives of the International Qualification (adapted from ISTQB meeting at Sollentuna,
November 2001)
Entry Requirements for this Qualification
Background and History of the Foundation Certificate in Software Testing

9. Appendix B Learning Objectives/Cognitive Level of Knowledge

Level 1: Remember (K1)
Level 2: Understand (K2)
Level 3: Apply (K3)
Level 4: Analyze (K4)
10. Appendix C Rules Applied to the ISTQB
Foundation Syllabus
10.1.1 General Rules
10.1.2 Current Content
10.1.3 Learning Objectives
10.1.4 Overall Structure
11. Appendix D Notice to Training Providers
12. Appendix E Release Notes
Release 2010
Release 2011
13. Index

Acknowledgements
International Software Testing Qualifications Board Working Group Foundation Level (Edition
2011): Thomas Mller (chair), Debra Friedenberg. The core team thanks the review team (Dan
Almog, Armin Beer, Rex Black, Julie Gardiner, Judy McKay, Tuula Pkknen, Eric Riou du
Cosquier Hans Schaefer, Stephanie Ulrich, Erik van Veenendaal) and all National Boards for the
suggestions for the current version of the syllabus.
International Software Testing Qualifications Board Working Group Foundation Level (Edition
2010): Thomas Mller (chair), Rahul Verma, Martin Klonk and Armin Beer. The core team thanks the
review team (Rex Black, Mette Bruhn-Pederson, Debra Friedenberg, Klaus Olsen, Judy McKay,
Tuula Pkknen, Meile Posthuma, Hans Schaefer, Stephanie Ulrich, Pete Williams, Erik van
Veenendaal) and all National Boards for their suggestions.
International Software Testing Qualifications Board Working Group Foundation Level (Edition
2007): Thomas Mller (chair), Dorothy Graham, Debra Friedenberg, and Erik van Veenendaal. The
core team thanks the review team (Hans Schaefer, Stephanie Ulrich, Meile Posthuma, Anders
Pettersson, and Wonil Kwon) and all the National Boards for their suggestions.
International Software Testing Qualifications Board Working Group Foundation Level (Edition
2005): Thomas Mller (chair), Rex Black, Sigrid Eldh, Dorothy Graham, Klaus Olsen, Maaret
Pyhjrvi, Geoff Thompson and Erik van Veenendaal and the review team and all National Boards
for their suggestions.

E-book Search Instructions

Search Instructions
For the purposes of facilitating reference and review of the important terms used throughout, this page
will instruct you in how to use the various search functions on the leading e-reading devices so that
you may find Glossary definitions with ease.

How To Search in Kindle e-ink Devices

On a Kindle e-ink device, to look up a word in the dictionary, use the 5-way key to navigate to the
specific word you wish to search.
1. Simply highlight the word, and the dictionary definition will appear at the bottom of the Kindle
screen.
2. For a more wide-ranging search, start typing the word you wish to look up while the book is
open on the screen.
3. When youve finished typing the word in the rounded box that will pop up, use the 5-way
controller to move to the right. When the find button turns black, simply click it and it will
search within the current book.
4. For more options, move to the right one more time. Youll then see 5 options: my items,
store, google, wikipedia and dictionary. Press the desired option to execute the search.
Please note that Kindle is fairly picky, so try to type accurately and competely, e.g., fired instead of
fire, if fired is the word youre seeking. A list of the search results will be returned, and you can
navigate to the desired result. Simply then press the back toggle, once or twice, (beneath the 5-way
button) to return to where you were reading.

How To Search in Kindle Fire Tablet Devices

1. On the Kindle Fire tablet, with the book open, simply highlight the word you wish to search.
2. A dialogue box will pop up, with a dictionary definition, a Note option, a Highlight option and a
Search option.
3. Press Search and youll be asked Search in Book, Search in Wikipedia or Search the
Web.
4. Pick the search you wantin this case, for the glossary definitions, youd choose Search in
Book, and a chapter-by-chapter list of search hits will be returned.
5. Select the item you want, (in many cases, simply the search result will give you the answer you
seek), and youll be taken to the result, with your searched term highlighted in yellow.
When done reading, tap the page lightly, and when the bottom menu bar pops up, tap the BACK arrow
to return to where you were reading.

How To Search on the Nook and NookColor Tablet Devices

On the Nook and NookColor tablets, with the book open, simply highlight the word you wish to
search (using touch).
1. A dialogue box will pop up, with a Highlight option, a Notes option, a Share option (if enabled),
a Look Up option and a find option.
2. Press Lookup for a dictionary search (you can also search Google and Wikipedia from this
interface.)
3. For an in-book search, simply press Find.
4. A list of every occurrence of the term, by chapter, will pop-up in the lower-left-hand corner.
5. Select the choice to which you want to navigate with a thumb-press.
After reviewing the search result, press the double button (occasionally you may need to press it
twice) to return to the location at which you initiated the search.

How To Search on the iPad iBooks application

In the iBooks application, with the book open, simply highlight the word you wish to search (using
touch).
1. A dialogue box will pop up, with a Define, Highlight, Note and Search option, with the word
you identified highighted in blue.
2. Press Search for an in-book search.
3. A list of every occurence of the term, by page number, will pop-up in the upper-right-hand
corner.
4. Select the choice to which you want to navigate with a thumb-press.
After reviewing the search result, look down at the lower-left-hand corner of the page, where youll
see, in very small text, beneath the progress bar, text that says Back to page X. Simply press that
text to return to the location at which you initiated the search.

How To Search on the Kobo device

Back Button
There is no Back button as such on the Kobo Touch. There is one hardware button that will take you
from anywhere to the Kobo Home Page which is a collection of thumbnails of the last five books
youve been reading. At this page, if you touch the cover image for a particular book, you will be
positioned at the last page you read to.

How to Search on the Kobo Touch

To search for a word or phrase in the book:
1. Touch the bottom of the screen and bring up the menu bar that has icons including the open book.
2. Touch the open book to pop up a menu with five items, the third of which is Search in Book.
3. Touch Search in Book and an input box appears at the top of the screen. Underneath is a
touchable list of the last six words or phrases that you searched for. On the bottom half of the
screen is a keyboard.
4. Type the word or phrase you want to search for with the keyboard and touch Go in the lower
right corner.

5. You will be presented with a list of phrases from the book text where the word or phrase occurs,
organized by the section of the book where they occur.
To find a definition for a word:
1. Touch the bottom of the screen and bring up the menu bar that has icons including the open book.
2. Touch the open book to pop up a menu with five items, the fourth of which is Definition.
3. Touch Definition and an input box appears at the top of the screen. On the bottom half of the
screen is a keyboard.
4. Type the word for which you want a definition with the keyboard and touch Go in the lower
right corner.
5. You will be presented with a list of definitions.

Introduction to this Syllabus

Purpose of this Document
This syllabus forms the basis for the International Software Testing Qualification at the Foundation
Level. The International Software Testing Qualifications Board (ISTQB) provides it to the National
Boards for them to accredit the training providers and to derive examination questions in their local
language. Training providers will determine appropriate teaching methods and produce courseware
for accreditation. The syllabus will help candidates in their preparation for the examination.
Information on the history and background of the syllabus can be found in Appendix A.

The Certified Tester Foundation Level in Software Testing

The Foundation Level qualification is aimed at anyone involved in software testing. This includes
people in roles such as testers, test analysts, test engineers, test consultants, test managers, user
acceptance testers and software developers. This Foundation Level qualification is also appropriate
for anyone who wants a basic understanding of software testing, such as project managers, quality
managers, software development managers, business analysts, IT directors and management
consultants. Holders of the Foundation Certificate will be able to go on to a higher-level software
testing qualification.

Learning Objectives/Cognitive Level of Knowledge

Learning objectives are indicated for each section in this syllabus and classified as follows:
K1: remember
K2: understand
K3: apply
K4: analyze
Further details and examples of learning objectives are given in Appendix B.
All terms listed under Terms just below chapter headings shall be remembered (K1), even if not
explicitly mentioned in the learning objectives.

The Examination
The Foundation Level Certificate examination will be based on this syllabus. Answers to examination
questions may require the use of material based on more than one section of this syllabus. All sections
of the syllabus are examinable.
The format of the examination is multiple choice.
Exams may be taken as part of an accredited training course or taken independently (e.g., at an
examination center or in a public exam). Completion of an accredited training course is not a
prerequisite for the exam.

Accreditation
An ISTQB National Board may accredit training providers whose course material follows this
syllabus. Training providers should obtain accreditation guidelines from the board or body that
performs the accreditation. An accredited course is recognized as conforming to this syllabus, and is
allowed to have an ISTQB examination as part of the course.
Further guidance for training providers is given in Appendix D.

Level of Detail
The level of detail in this syllabus allows internationally consistent teaching and examination. In
order to achieve this goal, the syllabus consists of:
General instructional objectives describing the intention of the Foundation Level
A list of information to teach, including a description, and references to additional sources if
required
Learning objectives for each knowledge area, describing the cognitive learning outcome and
mindset to be achieved
A list of terms that students must be able to recall and understand
A description of the key concepts to teach, including sources such as accepted literature or
standards
The syllabus content is not a description of the entire knowledge area of software testing; it reflects
the level of detail to be covered in Foundation Level training courses.

How this Syllabus is Organized

There are six major chapters. The top-level heading for each chapter shows the highest level of
learning objectives that is covered within the chapter and specifies the time for the chapter. For
example:

2. Testing Throughout the Software Life Cycle (K2)

115 minutes

This heading shows that Chapter 2 has learning objectives of K1 (assumed when a higher level is
shown) and K2 (but not K3), and it is intended to take 115 minutes to teach the material in the chapter.
Within each chapter there are a number of sections. Each section also has the learning objectives and
the amount of time required. Subsections that do not have a time given are included within the time for
the section.

1.

Fundamentals of Testing (K2)

155 minutes

Learning Objectives for Fundamentals of Testing

The objectives identify what you will be able to do following the completion of each module.

1.1 Why is Testing Necessary? (K2)

LO-1.1.1 Describe, with examples, the way in which a defect in software can cause harm to a
person, to the environment or to a company (K2)
LO-1.1.2 Distinguish between the root cause of a defect and its effects (K2)
LO-1.1.3 Give reasons why testing is necessary by giving examples (K2)
LO-1.1.4 Describe why testing is part of quality assurance and give examples of how testing
contributes to higher quality (K2)
LO-1.1.5 Explain and compare the terms error, defect, fault, failure, and the corresponding terms
mistake and bug, using examples (K2)

1.2 What is Testing? (K2)

LO-1.2.1 Recall the common objectives of testing (K1)
LO-1.2.2 Provide examples for the objectives of testing in different phases of the software life
cycle (K2)
LO-1.2.3 Differentiate testing from debugging (K2)

1.3 Seven Testing Principles (K2)

LO-1.3.1 Explain the seven principles in testing (K2)

1.4 Fundamental Test Process (K1)

LO-1.4.1 Recall the five fundamental test activities and respective tasks from planning to closure
(K1)

1.5 The Psychology of Testing (K2)

LO-1.5.1 Recall the psychological factors that influence the success of testing (K1)
LO-1.5.2 Contrast the mindset of a tester and of a developer (K2)

1.1

Why is Testing Necessary (K2)

20 minutes

Terms
Bug, defect, error, failure, fault, mistake, quality, risk

1.1.1 Software Systems Context (K1)

Software systems are an integral part of life, from business applications (e.g., banking) to consumer
products (e.g., cars). Most people have had an experience with software that did not work as
expected. Software that does not work correctly can lead to many problems, including loss of money,
time or business reputation, and could even cause injury or death.

1.1.2 Causes of Software Defects (K2)

A human being can make an error (mistake), which produces a defect (fault, bug) in the program code,
or in a document. If a defect in code is executed, the system may fail to do what it should do (or do
something it shouldnt), causing a failure. Defects in software, systems or documents may result in
failures, but not all defects do so.
Defects occur because human beings are fallible and because there is time pressure, complex code,
complexity of infrastructure, changing technologies, and/or many system interactions.
Failures can be caused by environmental conditions as well. For example, radiation, magnetism,
electronic fields, and pollution can cause faults in firmware or influence the execution of software by
changing the hardware conditions.

1.1.3 Role of Testing in Software Development, Maintenance and Operations

(K2)
Rigorous testing of systems and documentation can help to reduce the risk of problems occurring
during operation and contribute to the quality of the software system, if the defects found are
corrected before the system is released for operational use.
Software testing may also be required to meet contractual or legal requirements, or industry-specific
standards.

1.1.4 Testing and Quality (K2)

With the help of testing, it is possible to measure the quality of software in terms of defects found, for
both functional and non-functional software requirements and characteristics (e.g., reliability,
usability, efficiency, maintainability and portability). For more information on non-functional testing
see Chapter 2; for more information on software characteristics see Software Engineering
Software Product Quality (ISO 9126).

Testing can give confidence in the quality of the software if it finds few or no defects. A properly
designed test that passes reduces the overall level of risk in a system. When testing does find defects,
the quality of the software system increases when those defects are fixed.
Lessons should be learned from previous projects. By understanding the root causes of defects found
in other projects, processes can be improved, which in turn should prevent those defects from
reoccurring and, as a consequence, improve the quality of future systems. This is an aspect of quality
assurance.
Testing should be integrated as one of the quality assurance activities (i.e., alongside development
standards, training and defect analysis).

1.1.5 How Much Testing is Enough? (K2)

Deciding how much testing is enough should take account of the level of risk, including technical,
safety, and business risks, and project constraints such as time and budget. Risk is discussed further in
Chapter 5.
Testing should provide sufficient information to stakeholders to make informed decisions about the
release of the software or system being tested, for the next development step or handover to
customers.

1.2

What is Testing? (K2)

30 minutes

Terms
Debugging, requirement, review, test case, testing, test objective

Background
A common perception of testing is that it only consists of running tests, i.e., executing the software.
This is part of testing, but not all of the testing activities.
Test activities exist before and after test execution. These activities include planning and control,
choosing test conditions, designing and executing test cases, checking results, evaluating exit criteria,
reporting on the testing process and system under test, and finalizing or completing closure activities
after a test phase has been completed. Testing also includes reviewing documents (including source
code) and conducting static analysis.
Both dynamic testing and static testing can be used as a means for achieving similar objectives, and
will provide information that can be used to improve both the system being tested and the
development and testing processes.
Testing can have the following objectives:
Finding defects
Gaining confidence about the level of quality
Providing information for decision-making
Preventing defects
The thought process and activities involved in designing tests early in the life cycle (verifying the test
basis via test design) can help to prevent defects from being introduced into code. Reviews of
documents (e.g., requirements) and the identification and resolution of issues also help to prevent
defects appearing in the code.
Different viewpoints in testing take different objectives into account. For example, in development
testing (e.g., component, integration and system testing), the main objective may be to cause as many
failures as possible so that defects in the software are identified and can be fixed. In acceptance
testing, the main objective may be to confirm that the system works as expected, to gain confidence
that it has met the requirements. In some cases the main objective of testing may be to assess the
quality of the software (with no intention of fixing defects), to give information to stakeholders of the
risk of releasing the system at a given time. Maintenance testing often includes testing that no new
defects have been introduced during development of the changes. During operational testing, the main
objective may be to assess system characteristics such as reliability or availability.

Debugging and testing are different. Dynamic testing can show failures that are caused by defects.
Debugging is the development activity that finds, analyzes and removes the cause of the failure.
Subsequent re-testing by a tester ensures that the fix does indeed resolve the failure. The
responsibility for these activities is usually testers test and developers debug.
The process of testing and the testing activities are explained in Section 1.4.

1.3

Seven Testing Principles (K2)

35 minutes

Terms
Exhaustive testing

Principles
A number of testing principles have been suggested over the past 40 years and offer general
guidelines common for all testing.
Principle 1 Testing shows presence of defects
Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces
the probability of undiscovered defects remaining in the software but, even if no defects are found, it
is not a proof of correctness.
Principle 2 Exhaustive testing is impossible
Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial
cases. Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts.
Principle 3 Early testing
To find defects early, testing activities shall be started as early as possible in the software or system
development life cycle, and shall be focused on defined objectives.
Principle 4 Defect clustering
Testing effort shall be focused proportionally to the expected and later observed defect density of
modules. A small number of modules usually contains most of the defects discovered during prerelease testing, or is responsible for most of the operational failures.
Principle 5 Pesticide paradox
If the same tests are repeated over and over again, eventually the same set of test cases will no longer
find any new defects. To overcome this pesticide paradox, test cases need to be regularly reviewed
and revised, and new and different tests need to be written to exercise different parts of the software
or system to find potentially more defects.
Principle 6 Testing is context dependent
Testing is done differently in different contexts. For example, safety-critical software is tested
differently from an e-commerce site.
Principle 7 Absence-of-errors fallacy
Finding and fixing defects does not help if the system built is unusable and does not fulfill the users
needs and expectations.

1.4

Fundamental Test Process (K1)

35 minutes

Terms
Confirmation testing, re-testing, exit criteria, incident, regression testing, test basis, test condition, test
coverage, test data, test execution, test log, test plan, test procedure, test policy, test suite, test
summary report, testware

Background
The most visible part of testing is test execution. But to be effective and efficient, test plans should
also include time to be spent on planning the tests, designing test cases, preparing for execution and
evaluating results.
The fundamental test process consists of the following main activities:
Test planning and control
Test analysis and design
Test implementation and execution
Evaluating exit criteria and reporting
Test closure activities
Although logically sequential, the activities in the process may overlap or take place concurrently.
Tailoring these main activities within the context of the system and the project is usually required.

1.4.1 Test Planning and Control (K1)

Test planning is the activity of defining the objectives of testing and the specification of test activities
in order to meet the objectives and mission.
Test control is the ongoing activity of comparing actual progress against the plan, and reporting the
status, including deviations from the plan. It involves taking actions necessary to meet the mission and
objectives of the project. In order to control testing, the testing activities should be monitored
throughout the project. Test planning takes into account the feedback from monitoring and control
activities.
Test planning and control tasks are defined in Chapter 5 of this syllabus.

1.4.2 Test Analysis and Design (K1)

Test analysis and design is the activity during which general testing objectives are transformed into
tangible test conditions and test cases.
The test analysis and design activity has the following major tasks:

Reviewing the test basis (such as requirements, software integrity level1 (risk level), risk
analysis reports, architecture, design, interface specifications)
Evaluating testability of the test basis and test objects
Identifying and prioritizing test conditions based on analysis of test items, the specification,
behavior and structure of the software
Designing and prioritizing high level test cases
Identifying necessary test data to support the test conditions and test cases
Designing the test environment setup and identifying any required infrastructure and tools
Creating bi-directional traceability between test basis and test cases

1.4.3 Test Implementation and Execution (K1)

Test implementation and execution is the activity where test procedures or scripts are specified by
combining the test cases in a particular order and including any other information needed for test
execution, the environment is set up and the tests are run.
Test implementation and execution has the following major tasks:
Finalizing, implementing and prioritizing test cases (including the identification of test data)
Developing and prioritizing test procedures, creating test data and, optionally, preparing test
harnesses and writing automated test scripts
Creating test suites from the test procedures for efficient test execution
Verifying that the test environment has been set up correctly
Verifying and updating bi-directional traceability between the test basis and test cases
Executing test procedures either manually or by using test execution tools, according to the
planned sequence
Logging the outcome of test execution and recording the identities and versions of the software
under test, test tools and testware
Comparing actual results with expected results
Reporting discrepancies as incidents and analyzing them in order to establish their cause (e.g., a
defect in the code, in specified test data, in the test document, or a mistake in the way the test
was executed)
Repeating test activities as a result of action taken for each discrepancy, for example, reexecution of a test that previously failed in order to confirm a fix (confirmation testing),
execution of a corrected test and/or execution of tests in order to ensure that defects have not
been introduced in unchanged areas of the software or that defect fixing did not uncover other
defects (regression testing)

1.4.4 Evaluating Exit Criteria and Reporting (K1)

Evaluating exit criteria is the activity where test execution is assessed against the defined objectives.
This should be done for each test level (see Section 2.2).
Evaluating exit criteria has the following major tasks:

Checking test logs against the exit criteria specified in test planning
Assessing if more tests are needed or if the exit criteria specified should be changed
Writing a test summary report for stakeholders

1.4.5 Test Closure Activities (K1)

Test closure activities collect data from completed test activities to consolidate experience, testware,
facts and numbers. Test closure activities occur at project milestones such as when a software system
is released, a test project is completed (or cancelled), a milestone has been achieved, or a
maintenance release has been completed.
Test closure activities include the following major tasks:
Checking which planned deliverables have been delivered
Closing incident reports or raising change records for any that remain open
Documenting the acceptance of the system
Finalizing and archiving testware, the test environment and the test infrastructure for later reuse
Handing over the testware to the maintenance organization
Analyzing lessons learned to determine changes needed for future releases and projects
Using the information gathered to improve test maturity

1.5

The Psychology of Testing (K2)

25 minutes

Terms
Error guessing, independence

Background
The mindset to be used while testing and reviewing is different from that used while developing
software. With the right mindset developers are able to test their own code, but separation of this
responsibility to a tester is typically done to help focus effort and provide additional benefits, such as
an independent view by trained and professional testing resources. Independent testing may be
carried out at any level of testing.
A certain degree of independence (avoiding the author bias) often makes the tester more effective at
finding defects and failures. Independence is not, however, a replacement for familiarity, and
developers can efficiently find many defects in their own code. Several levels of independence can
be defined as shown here from low to high:
Tests designed by the person(s) who wrote the software under test (low level of independence)
Tests designed by another person(s) (e.g., from the development team)
Tests designed by a person(s) from a different organizational group (e.g., an independent test
team) or test specialists (e.g., usability or performance test specialists)
Tests designed by a person(s) from a different organization or company (i.e., outsourcing or
certification by an external body)
People and projects are driven by objectives. People tend to align their plans with the objectives set
by management and other stakeholders, for example, to find defects or to confirm that software meets
its objectives. Therefore, it is important to clearly state the objectives of testing.
Identifying failures during testing may be perceived as criticism against the product and against the
author. As a result, testing is often seen as a destructive activity, even though it is very constructive in
the management of product risks. Looking for failures in a system requires curiosity, professional
pessimism, a critical eye, attention to detail, good communication with development peers, and
experience on which to base error guessing.
If errors, defects or failures are communicated in a constructive way, bad feelings between the testers
and the analysts, designers and developers can be avoided. This applies to defects found during
reviews as well as in testing.
The tester and test leader need good interpersonal skills to communicate factual information about
defects, progress and risks in a constructive way. For the author of the software or document, defect
information can help them improve their skills. Defects found and fixed during testing will save time
and money later, and reduce risks.

Communication problems may occur, particularly if testers are seen only as messengers of unwanted
news about defects. However, there are several ways to improve communication and relationships
between testers and others:
Start with collaboration rather than battles remind everyone of the common goal of better
quality systems
Communicate findings on the product in a neutral, fact-focused way without criticizing the person
who created it, for example, write objective and factual incident reports and review findings
Try to understand how the other person feels and why they react as they do
Confirm that the other person has understood what you have said and vice versa

1.6

Code of Ethics

10 minutes

Involvement in software testing enables individuals to learn confidential and privileged information.
A code of ethics is necessary, among other reasons to ensure that the information is not put to
inappropriate use. Recognizing the ACM and IEEE code of ethics for engineers, the ISTQB states the
following code of ethics:
PUBLIC - Certified software testers shall act consistently with the public interest
CLIENT AND EMPLOYER - Certified software testers shall act in a manner that is in the best
interests of their client and employer, consistent with the public interest
PRODUCT - Certified software testers shall ensure that the deliverables they provide (on the
products and systems they test) meet the highest professional standards possible
JUDGMENT- Certified software testers shall maintain integrity and independence in their
professional judgment
MANAGEMENT - Certified software test managers and leaders shall subscribe to and promote an
ethical approach to the management of software testing
PROFESSION - Certified software testers shall advance the integrity and reputation of the profession
consistent with the public interest
COLLEAGUES - Certified software testers shall be fair to and supportive of their colleagues, and
promote cooperation with software developers
SELF - Certified software testers shall participate in lifelong learning regarding the practice of their
profession and shall promote an ethical approach to the practice of the profession

References
1.1.5 Black, 2001, Kaner, 2002
1.2 Beizer, 1990, Black, 2001, Myers, 1979
1.3 Beizer, 1990, Hetzel, 1988, Myers, 1979
1.4 Hetzel, 1988
1.4.5 Black, 2001, Craig, 2002
1.5 Black, 2001, Hetzel, 1988
_____________________________
1

The degree to which software complies or must comply with a set of stakeholder-selected software and/or software-based system
characteristics (e.g., software complexity, risk assessment, safety level, security level, desired performance, reliability, or cost) which are
defined to reflect the importance of the software to its stakeholders.

2. Testing Throughout the Software Life Cycle

(K2)

115
minutes

Learning Objectives for Testing Throughout the Software Life Cycle

The objectives identify what you will be able to do following the completion of each module.

2.1 Software Development Models (K2)

LO-2.1.1 Explain the relationship between development, test activities and work products in the
development life cycle, by giving examples using project and product types (K2)
LO-2.1.2 Recognize the fact that software development models must be adapted to the context of
project and product characteristics (K1)
LO-2.1.3 Recall characteristics of good testing that are applicable to any life cycle model (K1)

2.2 Test Levels (K2)

LO-2.2.1 Compare the different levels of testing: major objectives, typical objects of testing,
typical targets of testing (e.g., functional or structural) and related work products, people
who test, types of defects and failures to be identified (K2)

2.3 Test Types (K2)

LO-2.3.1 Compare four software test types (functional, non-functional, structural and changerelated) by example (K2)
LO-2.3.2 Recognize that functional and structural tests occur at any test level (K1)
LO-2.3.3 Identify and describe non-functional test types based on non-functional requirements (K2)
LO-2.3.4 Identify and describe test types based on the analysis of a software systems structure or
architecture (K2)
LO-2.3.5 Describe the purpose of confirmation testing and regression testing (K2)

2.4 Maintenance Testing (K2)

LO-2.4.1 Compare maintenance testing (testing an existing system) to testing a new application with
respect to test types, triggers for testing and amount of testing (K2)
LO-2.4.2 Recognize indicators for maintenance testing (modification, migration and retirement)
(K1)
LO-2.4.3 Describe the role of regression testing and impact analysis in maintenance (K2)

2.1

Software Development Models (K2)

20 minutes

Terms
Commercial Off-The-Shelf (COTS), iterative-incremental development model, validation,
verification, V-model

Background
Testing does not exist in isolation; test activities are related to software development activities.
Different development life cycle models need different approaches to testing.

2.1.1 V-model (Sequential Development Model) (K2)

Although variants of the V-model exist, a common type of V-model uses four test levels,
corresponding to the four development levels.
The four levels used in this syllabus are:
Component (unit) testing
Integration testing
System testing
Acceptance testing
In practice, a V-model may have more, fewer or different levels of development and testing,
depending on the project and the software product. For example, there may be component integration
testing after component testing, and system integration testing after system testing.
Software work products (such as business scenarios or use cases, requirements specifications, design
documents and code) produced during development are often the basis of testing in one or more test
levels. References for generic work products include Capability Maturity Model Integration (CMMI)
or Software life cycle processes (IEEE/IEC 12207). Verification and validation (and early test
design) can be carried out during the development of the software work products.

2.1.2 Iterative-incremental Development Models (K2)

Iterative-incremental development is the process of establishing requirements, designing, building
and testing a system in a series of short development cycles. Examples are: prototyping, Rapid
Application Development (RAD), Rational Unified Process (RUP) and agile development models. A
system that is produced using these models may be tested at several test levels during each iteration.
An increment, added to others developed previously, forms a growing partial system, which should
also be tested. Regression testing is increasingly important on all iterations after the first one.
Verification and validation can be carried out on each increment.

2.1.3 Testing within a Life Cycle Model (K2)

In any life cycle model, there are several characteristics of good testing:
For every development activity there is a corresponding testing activity
Each test level has test objectives specific to that level
The analysis and design of tests for a given test level should begin during the corresponding
development activity
Testers should be involved in reviewing documents as soon as drafts are available in the
development life cycle
Test levels can be combined or reorganized depending on the nature of the project or the system
architecture. For example, for the integration of a Commercial Off-The-Shelf (COTS) software
product into a system, the purchaser may perform integration testing at the system level (e.g.,
integration to the infrastructure and other systems, or system deployment) and acceptance testing
(functional and/or non-functional, and user and/or operational testing).

2.2

Test Levels (K2)

40 minutes

Terms
Alpha testing, beta testing, component testing, driver, field testing, functional requirement, integration,
integration testing, non-functional requirement, robustness testing, stub, system testing, test
environment, test level, test-driven development, user acceptance testing

Background
For each of the test levels, the following can be identified: the generic objectives, the work
product(s) being referenced for deriving test cases (i.e., the test basis), the test object (i.e., what is
being tested), typical defects and failures to be found, test harness requirements and tool support, and
specific approaches and responsibilities.
Testing a systems configuration data shall be considered during test planning,

2.2.1 Component Testing (K2)

Test basis:
Component requirements
Detailed design
Code
Typical test objects:
Components
Programs
Data conversion / migration programs
Database modules
Component testing (also known as unit, module or program testing) searches for defects in, and
verifies the functioning of, software modules, programs, objects, classes, etc., that are separately
testable. It may be done in isolation from the rest of the system, depending on the context of the
development life cycle and the system. Stubs, drivers and simulators may be used.
Component testing may include testing of functionality and specific non-functional characteristics,
such as resource-behavior (e.g., searching for memory leaks) or robustness testing, as well as
structural testing (e.g., decision coverage). Test cases are derived from work products such as a
specification of the component, the software design or the data model.
Typically, component testing occurs with access to the code being tested and with the support of a
development environment, such as a unit test framework or debugging tool. In practice, component

testing usually involves the programmer who wrote the code. Defects are typically fixed as soon as
they are found, without formally managing these defects.
One approach to component testing is to prepare and automate test cases before coding. This is called
a test-first approach or test-driven development. This approach is highly iterative and is based on
cycles of developing test cases, then building and integrating small pieces of code, and executing the
component tests correcting any issues and iterating until they pass.

2.2.2 Integration Testing (K2)

Test basis:
Software and system design
Architecture
Workflows
Use cases
Typical test objects:
Subsystems
Database implementation
Infrastructure
Interfaces
System configuration and configuration data
Integration testing tests interfaces between components, interactions with different parts of a system,
such as the operating system, file system and hardware, and interfaces between systems.
There may be more than one level of integration testing and it may be carried out on test objects of
varying size as follows:
1. Component integration testing tests the interactions between software components and is done
after component testing
2. System integration testing tests the interactions between different systems or between hardware
and software and may be done after system testing. In this case, the developing organization may
control only one side of the interface. This might be considered as a risk. Business processes
implemented as workflows may involve a series of systems. Cross-platform issues may be
significant.
The greater the scope of integration, the more difficult it becomes to isolate defects to a specific
component or system, which may lead to increased risk and additional time for troubleshooting.
Systematic integration strategies may be based on the system architecture (such as top-down and
bottom-up), functional tasks, transaction processing sequences, or some other aspect of the system or
components. In order to ease fault isolation and detect defects early, integration should normally be
incremental rather than big bang.

Testing of specific non-functional characteristics (e.g., performance) may be included in integration

testing as well as functional testing.
At each stage of integration, testers concentrate solely on the integration itself. For example, if they
are integrating module A with module B they are interested in testing the communication between the
modules, not the functionality of the individual module as that was done during component testing.
Both functional and structural approaches may be used.
Ideally, testers should understand the architecture and influence integration planning. If integration
tests are planned before components or systems are built, those components can be built in the order
required for most efficient testing.

2.2.3 System Testing (K2)

Test basis:
System and software requirement specification
Use cases
Functional specification
Risk analysis reports
Typical test objects:
System, user and operation manuals
System configuration and configuration data
System testing is concerned with the behavior of a whole system/product. The testing scope shall be
clearly addressed in the Master and/or Level Test Plan for that test level.
In system testing, the test environment should correspond to the final target or production environment
as much as possible in order to minimize the risk of environment-specific failures not being found in
testing.
System testing may include tests based on risks and/or on requirements specifications, business
processes, use cases, or other high level text descriptions or models of system behavior, interactions
with the operating system, and system resources.
System testing should investigate functional and non-functional requirements of the system, and data
quality characteristics. Testers also need to deal with incomplete or undocumented requirements.
System testing of functional requirements starts by using the most appropriate specification-based
(black-box) techniques for the aspect of the system to be tested. For example, a decision table may be
created for combinations of effects described in business rules. Structure-based techniques (whitebox) may then be used to assess the thoroughness of the testing with respect to a structural element,
such as menu structure or web page navigation (see Chapter 4).
An independent test team often carries out system testing.

2.2.4 Acceptance Testing (K2)

Test basis:
User requirements
System requirements
Use cases
Business processes
Risk analysis reports
Typical test objects:
Business processes on fully integrated system
Operational and maintenance processes
User procedures
Forms
Reports
Configuration data
Acceptance testing is often the responsibility of the customers or users of a system; other stakeholders
may be involved as well.
The goal in acceptance testing is to establish confidence in the system, parts of the system or specific
non-functional characteristics of the system. Finding defects is not the main focus in acceptance
testing. Acceptance testing may assess the systems readiness for deployment and use, although it is
not necessarily the final level of testing. For example, a large-scale system integration test may come
after the acceptance test for a system.
Acceptance testing may occur at various times in the life cycle, for example:
A COTS software product may be acceptance tested when it is installed or integrated
Acceptance testing of the usability of a component may be done during component testing
Acceptance testing of a new functional enhancement may come before system testing
Typical forms of acceptance testing include the following:
User acceptance testing
Typically verifies the fitness for use of the system by business users.
Operational (acceptance) testing
The acceptance of the system by the system administrators, including:
Testing of backup/restore
Disaster recovery
User management
Maintenance tasks
Data load and migration tasks

Periodic checks of security vulnerabilities

Contract and regulation acceptance testing

Contract acceptance testing is performed against a contracts acceptance criteria for producing
custom-developed software. Acceptance criteria should be defined when the parties agree to the
contract. Regulation acceptance testing is performed against any regulations that must be adhered to,
such as government, legal or safety regulations.
Alpha and beta (or field) testing
Developers of market, or COTS, software often want to get feedback from potential or existing
customers in their market before the software product is put up for sale commercially. Alpha testing is
performed at the developing organizations site but not by the developing team. Beta testing, or fieldtesting, is performed by customers or potential customers at their own locations.
Organizations may use other terms as well, such as factory acceptance testing and site acceptance
testing for systems that are tested before and after being moved to a customers site.

2.3

Test Types (K2)

40 minutes

Terms
Black-box testing, code coverage, functional testing, interoperability testing, load testing,
maintainability testing, performance testing, portability testing, reliability testing, security testing,
stress testing, structural testing, usability testing, white-box testing

Background
A group of test activities can be aimed at verifying the software system (or a part of a system) based
on a specific reason or target for testing.
A test type is focused on a particular test objective, which could be any of the following:
A function to be performed by the software
A non-functional quality characteristic, such as reliability or usability
The structure or architecture of the software or system
Change related, i.e., confirming that defects have been fixed (confirmation testing) and looking
for unintended changes (regression testing)
A model of the software may be developed and/or used in structural testing (e.g., a control flow
model or menu structure model), non-functional testing (e.g., performance model, usability model
security threat modeling), and functional testing (e.g., a process flow model, a state transition model
or a plain language specification).

2.3.1 Testing of Function (Functional Testing) (K2)

The functions that a system, subsystem or component are to perform may be described in work
products such as a requirements specification, use cases, or a functional specification, or they may be
undocumented. The functions are what the system does.
Functional tests are based on functions and features (described in documents or understood by the
testers) and their interoperability with specific systems, and may be performed at all test levels (e.g.,
tests for components may be based on a component specification).
Specification-based techniques may be used to derive test conditions and test cases from the
functionality of the software or system (see Chapter 4). Functional testing considers the external
behavior of the software (black-box testing).
A type of functional testing, security testing, investigates the functions (e.g., a firewall) relating to
detection of threats, such as viruses, from malicious outsiders. Another type of functional testing,
interoperability testing, evaluates the capability of the software product to interact with one or more
specified components or systems.

2.3.2 Testing of Non-functional Software Characteristics (Non-functional

Testing) (K2)
Non-functional testing includes, but is not limited to, performance testing, load testing, stress testing,
usability testing, maintainability testing, reliability testing and portability testing. It is the testing of
how the system works.
Non-functional testing may be performed at all test levels. The term non-functional testing describes
the tests required to measure characteristics of systems and software that can be quantified on a
varying scale, such as response times for performance testing. These tests can be referenced to a
quality model such as the one defined in Software Engineering Software Product Quality (ISO
9126). Non-functional testing considers the external behavior of the software and in most cases uses
black-box test design techniques to accomplish that.

2.3.3 Testing of Software Structure/Architecture (Structural Testing) (K2)

Structural (white-box) testing may be performed at all test levels. Structural techniques are best used
after specification-based techniques, in order to help measure the thoroughness of testing through
assessment of coverage of a type of structure.
Coverage is the extent that a structure has been exercised by a test suite, expressed as a percentage of
the items being covered. If coverage is not 100%, then more tests may be designed to test those items
that were missed to increase coverage. Coverage techniques are covered in Chapter 4.
At all test levels, but especially in component testing and component integration testing, tools can be
used to measure the code coverage of elements, such as statements or decisions. Structural testing
may be based on the architecture of the system, such as a calling hierarchy.
Structural testing approaches can also be applied at system, system integration or acceptance testing
levels (e.g., to business models or menu structures).

2.3.4 Testing Related to Changes: Re-testing and Regression Testing (K2)

After a defect is detected and fixed, the software should be re-tested to confirm that the original
defect has been successfully removed. This is called confirmation. Debugging (locating and fixing a
defect) is a development activity, not a testing activity.
Regression testing is the repeated testing of an already tested program, after modification, to discover
any defects introduced or uncovered as a result of the change(s). These defects may be either in the
software being tested, or in another related or unrelated software component. It is performed when
the software, or its environment, is changed. The extent of regression testing is based on the risk of
not finding defects in software that was working previously.
Tests should be repeatable if they are to be used for confirmation testing and to assist regression
testing.

Regression testing may be performed at all test levels, and includes functional, non-functional and
structural testing. Regression test suites are run many times and generally evolve slowly, so
regression testing is a strong candidate for automation.

2.4

Maintenance Testing (K2)

15 minutes

Terms
Impact analysis, maintenance testing

Background
Once deployed, a software system is often in service for years or decades. During this time the
system, its configuration data, or its environment are often corrected, changed or extended. The
planning of releases in advance is crucial for successful maintenance testing. A distinction has to be
made between planned releases and hot fixes. Maintenance testing is done on an existing operational
system, and is triggered by modifications, migration, or retirement of the software or system.
Modifications include planned enhancement changes (e.g., release-based), corrective and emergency
changes, and changes of environment, such as planned operating system or database upgrades,
planned upgrade of Commercial-Off-The-Shelf software, or patches to correct newly exposed or
discovered vulnerabilities of the operating system.
Maintenance testing for migration (e.g., from one platform to another) should include operational tests
of the new environment as well as of the changed software. Migration testing (conversion testing) is
also needed when data from another application will be migrated into the system being maintained.
Maintenance testing for the retirement of a system may include the testing of data migration or
archiving if long data-retention periods are required.
In addition to testing what has been changed, maintenance testing includes regression testing to parts
of the system that have not been changed. The scope of maintenance testing is related to the risk of the
change, the size of the existing system and to the size of the change. Depending on the changes,
maintenance testing may be done at any or all test levels and for any or all test types. Determining
how the existing system may be affected by changes is called impact analysis, and is used to help
decide how much regression testing to do. The impact analysis may be used to determine the
regression test suite.
Maintenance testing can be difficult if specifications are out of date or missing, or testers with domain
knowledge are not available.

References
2.1.3 CMMI, Craig, 2002, Hetzel, 1988, IEEE 12207
2.2 Hetzel, 1988
2.2.4 Copeland, 2004, Myers, 1979
2.3.1 Beizer, 1990, Black, 2001, Copeland, 2004
2.3.2 Black, 2001, ISO 9126

2.3.3 Beizer, 1990, Copeland, 2004, Hetzel, 1988

2.3.4 Hetzel, 1988, IEEE STD 829-1998
2.4 Black, 2001, Craig, 2002, Hetzel, 1988, IEEE STD 829-1998

3.

Static Techniques (K2)

60 minutes

Learning Objectives for Static Techniques

The objectives identify what you will be able to do following the completion of each module.

3.1 Static Techniques and the Test Process (K2)

LO-3.1.1 Recognize software work products that can be examined by the different static techniques
(K1)
LO-3.1.2 Describe the importance and value of considering static techniques for the assessment of
software work products (K2)
LO-3.1.3 Explain the difference between static and dynamic techniques, considering objectives,
types of defects to be identified, and the role of these techniques within the software life
cycle (K2)

3.2 Review Process (K2)

LO-3.2.1 Recall the activities, roles and responsibilities of a typical formal review (K1)
LO-3.2.2 Explain the differences between different types of reviews: informal review, technical
review, walkthrough and inspection (K2)
LO-3.2.3 Explain the factors for successful performance of reviews (K2)

3.3 Static Analysis by Tools (K2)

LO-3.3.1 Recall typical defects and errors identified by static analysis and compare them to
reviews and dynamic testing (K1)
LO-3.3.2 Describe, using examples, the typical benefits of static analysis (K2)
LO-3.3.3 List typical code and design defects that may be identified by static analysis tools (K1)

3.1

Static Techniques and the Test Process (K2)

15 minutes

Terms
Dynamic testing, static testing

Background
Unlike dynamic testing, which requires the execution of software, static testing techniques rely on the
manual examination (reviews) and automated analysis (static analysis) of the code or other project
documentation without the execution of the code.
Reviews are a way of testing software work products (including code) and can be performed well
before dynamic test execution. Defects detected during reviews early in the life cycle (e.g., defects
found in requirements) are often much cheaper to remove than those detected by running tests on the
executing code.
A review could be done entirely as a manual activity, but there is also tool support. The main manual
activity is to examine a work product and make comments about it. Any software work product can be
reviewed, including requirements specifications, design specifications, code, test plans, test
specifications, test cases, test scripts, user guides or web pages.
Benefits of reviews include early defect detection and correction, development productivity
improvements, reduced development timescales, reduced testing cost and time, lifetime cost
reductions, fewer defects and improved communication. Reviews can find omissions, for example, in
requirements, which are unlikely to be found in dynamic testing.
Reviews, static analysis and dynamic testing have the same objective identifying defects. They are
complementary; the different techniques can find different types of defects effectively and efficiently.
Compared to dynamic testing, static techniques find causes of failures (defects) rather than the
failures themselves.
Typical defects that are easier to find in reviews than in dynamic testing include: deviations from
standards, requirement defects, design defects, insufficient maintainability and incorrect interface
specifications.

3.2

Review Process (K2)

25 minutes

Terms
Entry criteria, formal review, informal review, inspection, metric, moderator, peer review, reviewer,
scribe, technical review, walkthrough

Background
The different types of reviews vary from informal, characterized by no written instructions for
reviewers, to systematic, characterized by team participation, documented results of the review, and
documented procedures for conducting the review. The formality of a review process is related to
factors such as the maturity of the development process, any legal or regulatory requirements or the
need for an audit trail.
The way a review is carried out depends on the agreed objectives of the review (e.g., find defects,
gain understanding, educate testers and new team members, or discussion and decision by consensus).

3.2.1 Activities of a Formal Review (K1)

A typical formal review has the following main activities:
1. Planning
Defining the review criteria
Selecting the personnel
Allocating roles
Defining the entry and exit criteria for more formal review types (e.g., inspections)
Selecting which parts of documents to review
Checking entry criteria (for more formal review types)
2. Kick-off
Distributing documents
Explaining the objectives, process and documents to the participants
3. Individual preparation
Preparing for the review meeting by reviewing the document(s)
Noting potential defects, questions and comments
4. Examination/evaluation/recording of results (review meeting)
Discussing or logging, with documented results or minutes (for more formal review types)
Noting defects, making recommendations regarding handling the defects, making decisions about
the defects
Examining/evaluating and recording issues during any physical meetings or tracking any group

electronic communications
5. Rework
Fixing defects found (typically done by the author)
Recording updated status of defects (in formal reviews)
6. Follow-up
Checking that defects have been addressed
Gathering metrics
Checking on exit criteria (for more formal review types)

3.2.2 Roles and Responsibilities (K1)

A typical formal review will include the roles below:
Manager: decides on the execution of reviews, allocates time in project schedules and
determines if the review objectives have been met.
Moderator: the person who leads the review of the document or set of documents, including
planning the review, running the meeting, and following-up after the meeting. If necessary, the
moderator may mediate between the various points of view and is often the person upon whom
the success of the review rests.
Author: the writer or person with chief responsibility for the document(s) to be reviewed.
Reviewers: individuals with a specific technical or business background (also called checkers
or inspectors) who, after the necessary preparation, identify and describe findings (e.g., defects)
in the product under review. Reviewers should be chosen to represent different perspectives and
roles in the review process, and should take part in any review meetings.
Scribe (or recorder): documents all the issues, problems and open points that were identified
during the meeting.
Looking at software products or related work products from different perspectives and using
checklists can make reviews more effective and efficient. For example, a checklist based on various
perspectives such as user, maintainer, tester or operations, or a checklist of typical requirements
problems may help to uncover previously undetected issues.

3.2.3 Types of Reviews (K2)

A single software product or related work product may be the subject of more than one review. If
more than one type of review is used, the order may vary. For example, an informal review may be
carried out before a technical review, or an inspection may be carried out on a requirements
specification before a walkthrough with customers. The main characteristics, options and purposes of
common review types are:
Informal Review
No formal process
May take the form of pair programming or a technical lead reviewing designs and code

Results may be documented

Varies in usefulness depending on the reviewers
Main purpose: inexpensive way to get some benefit

Walkthrough
Meeting led by author
May take the form of scenarios, dry runs, peer group participation
Open-ended sessions
Optional pre-meeting preparation of reviewers
Optional preparation of a review report including list of findings

Optional scribe (who is not the author)

May vary in practice from quite informal to very formal
Main purposes: learning, gaining understanding, finding defects

Technical Review
Documented, defined defect-detection process that includes peers and technical experts with
optional management participation
May be performed as a peer review without management participation
Ideally led by trained moderator (not the author)
Pre-meeting preparation by reviewers
Optional use of checklists
Preparation of a review report which includes the list of findings, the verdict whether the
software product meets its requirements and, where appropriate, recommendations related to
findings
May vary in practice from quite informal to very formal
Main purposes: discussing, making decisions, evaluating alternatives, finding defects, solving
technical problems and checking conformance to specifications, plans, regulations, and standards
Inspection
Led by trained moderator (not the author)
Usually conducted as a peer examination
Defined roles
Includes metrics gathering
Formal process based on rules and checklists
Specified entry and exit criteria for acceptance of the software product
Pre-meeting preparation
Inspection report including list of findings
Formal follow-up process (with optional process improvement components)
Optional reader

Main purpose: finding defects

Walkthroughs, technical reviews and inspections can be performed within a peer group, i.e.,
colleagues at the same organizational level. This type of review is called a peer review.

3.2.4 Success Factors for Reviews (K2)

Success factors for reviews include:
Each review has clear predefined objectives
The right people for the review objectives are involved
Testers are valued reviewers who contribute to the review and also learn about the product
which enables them to prepare tests earlier
Defects found are welcomed and expressed objectively
People issues and psychological aspects are dealt with (e.g., making it a positive experience for
the author)
The review is conducted in an atmosphere of trust; the outcome will not be used for the
evaluation of the participants
Review techniques are applied that are suitable to achieve the objectives and to the type and
level of software work products and reviewers
Checklists or roles are used if appropriate to increase effectiveness of defect identification
Training is given in review techniques, especially the more formal techniques such as inspection
Management supports a good review process (e.g., by incorporating adequate time for review
activities in project schedules)
There is an emphasis on learning and process improvement

3.3

Static Analysis by Tools (K2)

20 minutes

Terms
Compiler, complexity, control flow, data flow, static analysis

Background
The objective of static analysis is to find defects in software source code and software models. Static
analysis is performed without actually executing the software being examined by the tool; dynamic
testing does execute the software code. Static analysis can locate defects that are hard to find in
dynamic testing. As with reviews, static analysis finds defects rather than failures. Static analysis
tools analyze program code (e.g., control flow and data flow), as well as generated output such as
HTML and XML.
The value of static analysis is:
Early detection of defects prior to test execution
Early warning about suspicious aspects of the code or design by the calculation of metrics, such
as a high complexity measure
Identification of defects not easily found by dynamic testing
Detecting dependencies and inconsistencies in software models such as links
Improved maintainability of code and design
Prevention of defects, if lessons are learned in development
Typical defects discovered by static analysis tools include:
Referencing a variable with an undefined value
Inconsistent interfaces between modules and components
Variables that are not used or are improperly declared
Unreachable (dead) code
Missing and erroneous logic (potentially infinite loops)
Overly complicated constructs
Programming standards violations
Security vulnerabilities
Syntax violations of code and software models
Static analysis tools are typically used by developers (checking against predefined rules or
programming standards) before and during component and integration testing or when checking-in
code to configuration management tools, and by designers during software modeling. Static analysis
tools may produce a large number of warning messages, which need to be well-managed to allow the
most effective use of the tool.

Compilers may offer some support for static analysis, including the calculation of metrics.

References
3.2 IEEE 1028
3.2.2 Gilb, 1993, van Veenendaal, 2004
3.2.4 Gilb, 1993, IEEE 1028
3.3 van Veenendaal, 2004

4.

Test Design Techniques (K4)

285 minutes

Learning Objectives for Test Design Techniques

The objectives identify what you will be able to do following the completion of each module.
4.1 The Test Development Process (K3)
LO-4.1.1 Differentiate between a test design specification, test case specification and test
procedure specification (K2)
LO-4.1.2 Compare the terms test condition, test case and test procedure (K2)
LO-4.1.3 Evaluate the quality of test cases in terms of clear traceability to the requirements and
expected results (K2)
LO-4.1.4 Translate test cases into a well-structured test procedure specification at a level of detail
relevant to the knowledge of the testers (K3)
4.2 Categories of Test Design Techniques (K2)
LO-4.2.1 Recall reasons that both specification-based (black-box) and structure-based (white-box)
test design techniques are useful and list the common techniques for each (K1)
LO-4.2.2 Explain the characteristics, commonalities, and differences between specification-based
testing, structure-based testing and experience-based testing (K2)
4.3 Specification-based or Black-box Techniques (K3)
LO-4.3.1 Write test cases from given software models using equivalence partitioning, boundary
value analysis, decision tables and state transition diagrams/tables (K3)
LO-4.3.2 Explain the main purpose of each of the four testing techniques, what level and type of
testing could use the technique, and how coverage may be measured (K2)
LO-4.3.3 Explain the concept of use case testing and its benefits (K2)
4.4 Structure-based or White-box Techniques (K4)
LO-4.4.1 Describe the concept and value of code coverage (K2)
LO-4.4.2 Explain the concepts of statement and decision coverage, and give reasons why these
concepts can also be used at test levels other than component testing (e.g., on business
procedures at system level) (K2)
LO-4.4.3 Write test cases from given control flows using statement and decision test design
techniques (K3)
LO-4.4.4 Assess statement and decision coverage for completeness with respect to defined exit
criteria. (K4)
4.5 Experience-based Techniques (K2)
LO-4.5.1 Recall reasons for writing test cases based on intuition, experience and knowledge about
common defects (K1)
LO-4.5.2 Compare experience-based techniques with specification-based testing techniques (K2)
4.6 Choosing Test Techniques (K2)

LO-4.6.1 Classify test design techniques according to their fitness to a given context, for the test
basis, respective models and software characteristics (K2)

4.1

The Test Development Process (K3)

15 minutes

Terms
Test case specification, test design, test execution schedule, test procedure specification, test script,
traceability

Background
The test development process described in this section can be done in different ways, from very
informal with little or no documentation, to very formal (as it is described below). The level of
formality depends on the context of the testing, including the maturity of testing and development
processes, time constraints, safety or regulatory requirements, and the people involved.
During test analysis, the test basis documentation is analyzed in order to determine what to test, i.e.,
to identify the test conditions. A test condition is defined as an item or event that could be verified by
one or more test cases (e.g., a function, transaction, quality characteristic or structural element).
Establishing traceability from test conditions back to the specifications and requirements enables both
effective impact analysis when requirements change, and determining requirements coverage for a set
of tests. During test analysis the detailed test approach is implemented to select the test design
techniques to use based on, among other considerations, the identified risks (see Chapter 5 for more
on risk analysis).
During test design the test cases and test data are created and specified. A test case consists of a set of
input values, execution preconditions, expected results and execution postconditions, defined to cover
a certain test objective(s) or test condition(s). The Standard for Software Test Documentation (IEEE
STD 829-1998) describes the content of test design specifications (containing test conditions) and
test case specifications.
Expected results should be produced as part of the specification of a test case and include outputs,
changes to data and states, and any other consequences of the test. If expected results have not been
defined, then a plausible, but erroneous, result may be interpreted as the correct one. Expected results
should ideally be defined prior to test execution.
During test implementation the test cases are developed, implemented, prioritized and organized in
the test procedure specification (IEEE STD 829-1998). The test procedure specifies the sequence of
actions for the execution of a test. If tests are run using a test execution tool, the sequence of actions is
specified in a test script (which is an automated test procedure).
The various test procedures and automated test scripts are subsequently formed into a test execution
schedule that defines the order in which the various test procedures, and possibly automated test

scripts, are executed. The test execution schedule will take into account such factors as regression
tests, prioritization, and technical and logical dependencies.

4.2

Categories of Test Design Techniques (K2)

15 minutes

Terms
Black-box test design technique, experience-based test design technique, test design technique, whitebox test design technique

Background
The purpose of a test design technique is to identify test conditions, test cases, and test data.
It is a classic distinction to denote test techniques as black-box or white-box. Black-box test design
techniques (also called specification-based techniques) are a way to derive and select test conditions,
test cases, or test data based on an analysis of the test basis documentation. This includes both
functional and non-functional testing. Black-box testing, by definition, does not use any information
regarding the internal structure of the component or system to be tested. White-box test design
techniques (also called structural or structure-based techniques) are based on an analysis of the
structure of the component or system. Black-box and white-box testing may also be combined with
experience-based techniques to leverage the experience of developers, testers and users to determine
what should be tested.
Some techniques fall clearly into a single category; others have elements of more than one category.
This syllabus refers to specification-based test design techniques as black-box techniques and
structure-based test design techniques as white-box techniques. In addition experience-based test
design techniques are covered.
Common characteristics of specification-based test design techniques include:
Models, either formal or informal, are used for the specification of the problem to be solved, the
software or its components
Test cases can be derived systematically from these models
Common characteristics of structure-based test design techniques include:
Information about how the software is constructed is used to derive the test cases (e.g., code and
detailed design information)
The extent of coverage of the software can be measured for existing test cases, and further test
cases can be derived systematically to increase coverage
Common characteristics of experience-based test design techniques include:
The knowledge and experience of people are used to derive the test cases
The knowledge of testers, developers, users and other stakeholders about the software, its usage
and its environment is one source of information

Knowledge about likely defects and their distribution is another source of information

4.3 Specification-based or Black-box Techniques

(K3)

150
minutes

Terms
Boundary value analysis, decision table testing, equivalence partitioning, state transition testing, use
case testing

4.3.1 Equivalence Partitioning (K3)

In equivalence partitioning, inputs to the software or system are divided into groups that are expected
to exhibit similar behavior, so they are likely to be processed in the same way. Equivalence partitions
(or classes) can be found for both valid data, i.e., values that should be accepted and invalid data,
i.e., values that should be rejected. Partitions can also be identified for outputs, internal values, timerelated values (e.g., before or after an event) and for interface parameters (e.g., integrated
components being tested during integration testing). Tests can be designed to cover all valid and
invalid partitions. Equivalence partitioning is applicable at all levels of testing.
Equivalence partitioning can be used to achieve input and output coverage goals. It can be applied to
human input, input via interfaces to a system, or interface parameters in integration testing.

4.3.2 Boundary Value Analysis (K3)

Behavior at the edge of each equivalence partition is more likely to be incorrect than behavior within
the partition, so boundaries are an area where testing is likely to yield defects. The maximum and
minimum values of a partition are its boundary values. A boundary value for a valid partition is a
valid boundary value; the boundary of an invalid partition is an invalid boundary value. Tests can be
designed to cover both valid and invalid boundary values. When designing test cases, a test for each
boundary value is chosen.
Boundary value analysis can be applied at all test levels. It is relatively easy to apply and its defectfinding capability is high. Detailed specifications are helpful in determining the interesting
boundaries.
This technique is often considered as an extension of equivalence partitioning or other black-box test
design techniques. It can be used on equivalence classes for user input on screen as well as, for
example, on time ranges (e.g., time out, transactional speed requirements) or table ranges (e.g., table
size is 256*256).

4.3.3 Decision Table Testing (K3)

Decision tables are a good way to capture system requirements that contain logical conditions, and to
document internal system design. They may be used to record complex business rules that a system is
to implement. When creating decision tables, the specification is analyzed, and conditions and actions

of the system are identified. The input conditions and actions are most often stated in such a way that
they must be true or false (Boolean). The decision table contains the triggering conditions, often
combinations of true and false for all input conditions, and the resulting actions for each combination
of conditions. Each column of the table corresponds to a business rule that defines a unique
combination of conditions and which result in the execution of the actions associated with that rule.
The coverage standard commonly used with decision table testing is to have at least one test per
column in the table, which typically involves covering all combinations of triggering conditions.
The strength of decision table testing is that it creates combinations of conditions that otherwise might
not have been exercised during testing. It may be applied to all situations when the action of the
software depends on several logical decisions.

4.3.4 State Transition Testing (K3)

A system may exhibit a different response depending on current conditions or previous history (its
state). In this case, that aspect of the system can be shown with a state transition diagram. It allows
the tester to view the software in terms of its states, transitions between states, the inputs or events
that trigger state changes (transitions) and the actions which may result from those transitions. The
states of the system or object under test are separate, identifiable and finite in number.
A state table shows the relationship between the states and inputs, and can highlight possible
transitions that are invalid.
Tests can be designed to cover a typical sequence of states, to cover every state, to exercise every
transition, to exercise specific sequences of transitions or to test invalid transitions.
State transition testing is much used within the embedded software industry and technical automation
in general. However, the technique is also suitable for modeling a business object having specific
states or testing screen-dialogue flows (e.g., for Internet applications or business scenarios).

4.3.5 Use Case Testing (K2)

Tests can be derived from use cases. A use case describes interactions between actors (users or
systems), which produce a result of value to a system user or the customer. Use cases may be
described at the abstract level (business use case, technology-free, business process level) or at the
system level (system use case on the system functionality level). Each use case has preconditions
which need to be met for the use case to work successfully. Each use case terminates with
postconditions which are the observable results and final state of the system after the use case has
been completed. A use case usually has a mainstream (i.e., most likely) scenario and alternative
scenarios.
Use cases describe the process flows through a system based on its actual likely use, so the test
cases derived from use cases are most useful in uncovering defects in the process flows during realworld use of the system. Use cases are very useful for designing acceptance tests with customer/user
participation. They also help uncover integration defects caused by the interaction and interference of
different components, which individual component testing would not see. Designing test cases from
use cases may be combined with other specification-based test techniques.

4.4

Structure-based or White-box Techniques (K4)

60 minutes

Terms
Code coverage, decision coverage, statement coverage, structure-based testing

Background
Structure-based or white-box testing is based on an identified structure of the software or the system,
as seen in the following examples:
Component level: the structure of a software component, i.e., statements, decisions, branches or
even distinct paths
Integration level: the structure may be a call tree (a diagram in which modules call other
modules)
System level: the structure may be a menu structure, business process or web page structure
In this section, three code-related structural test design techniques for code coverage, based on
statements, branches and decisions, are discussed. For decision testing, a control flow diagram may
be used to visualize the alternatives for each decision.

4.4.1 Statement Testing and Coverage (K4)

In component testing, statement coverage is the assessment of the percentage of executable statements
that have been exercised by a test case suite. The statement testing technique derives test cases to
execute specific statements, normally to increase statement coverage.
Statement coverage is determined by the number of executable statements covered by (designed or
executed) test cases divided by the number of all executable statements in the code under test.

4.4.2 Decision Testing and Coverage (K4)

Decision coverage, related to branch testing, is the assessment of the percentage of decision outcomes
(e.g., the True and False options of an IF statement) that have been exercised by a test case suite. The
decision testing technique derives test cases to execute specific decision outcomes. Branches
originate from decision points in the code and show the transfer of control to different locations in the
code.
Decision coverage is determined by the number of all decision outcomes covered by (designed or
executed) test cases divided by the number of all possible decision outcomes in the code under test.
Decision testing is a form of control flow testing as it follows a specific flow of control through the
decision points. Decision coverage is stronger than statement coverage; 100% decision coverage
guarantees 100% statement coverage, but not vice versa.

4.4.3 Other Structure-based Techniques (K1)

There are stronger levels of structural coverage beyond decision coverage, for example, condition
coverage and multiple condition coverage.
The concept of coverage can also be applied at other test levels For example, at the integration level
the percentage of modules, components or classes that have been exercised by a test case suite could
be expressed as module, component or class coverage.
Tool support is useful for the structural testing of code.

4.5

Experience-based Techniques (K2)

30 minutes

Terms
Exploratory testing, (fault) attack

Background
Experience-based testing is where tests are derived from the testers skill and intuition and their
experience with similar applications and technologies. When used to augment systematic techniques,
these techniques can be useful in identifying special tests not easily captured by formal techniques,
especially when applied after more formal approaches. However, this technique may yield widely
varying degrees of effectiveness, depending on the testers experience.
A commonly used experience-based technique is error guessing. Generally testers anticipate defects
based on experience. A structured approach to the error guessing technique is to enumerate a list of
possible defects and to design tests that attack these defects. This systematic approach is called fault
attack. These defect and failure lists can be built based on experience, available defect and failure
data, and from common knowledge about why software fails.
Exploratory testing is concurrent test design, test execution, test logging and learning, based on a test
charter containing test objectives, and carried out within time-boxes. It is an approach that is most
useful where there are few or inadequate specifications and severe time pressure, or in order to
augment or complement other, more formal testing. It can serve as a check on the test process, to help
ensure that the most serious defects are found.

4.6

Choosing Test Techniques (K2)

15 minutes

Terms
No specific terms.

Background
The choice of which test techniques to use depends on a number of factors, including the type of
system, regulatory standards, customer or contractual requirements, level of risk, type of risk, test
objective, documentation available, knowledge of the testers, time and budget, development life
cycle, use case models and previous experience with types of defects found.
Some techniques are more applicable to certain situations and test levels; others are applicable to all
test levels.
When creating test cases, testers generally use a combination of test techniques including process,
rule and data-driven techniques to ensure adequate coverage of the object under test.

References
4.1 Craig, 2002, Hetzel, 1988, IEEE STD 829-1998
4.2 Beizer, 1990, Copeland, 2004
4.3.1 Copeland, 2004, Myers, 1979
4.3.2 Copeland, 2004, Myers, 1979
4.3.3 Beizer, 1990, Copeland, 2004
4.3.4 Beizer, 1990, Copeland, 2004
4.3.5 Copeland, 2004
4.4.3 Beizer, 1990, Copeland, 2004
4.5 Kaner, 2002
4.6 Beizer, 1990, Copeland, 2004

5.

Test Management (K3)

170 minutes

Learning Objectives for Test Management

The objectives identify what you will be able to do following the completion of each module.

5.1 Test Organization (K2)

LO-5.1.1
LO-5.1.2
LO-5.1.3
LO-5.1.4

Recognize the importance of independent testing (K1)

Explain the benefits and drawbacks of independent testing within an organization (K2)
Recognize the different team members to be considered for the creation of a test team(K1)
Recall the tasks of a typical test leader and tester (K1)

5.2 Test Planning and Estimation (K3)

LO-5.2.1 Recognize the different levels and objectives of test planning (K1)
LO-5.2.2 Summarize the purpose and content of the test plan, test design specification and test
procedure documents according to the Standard for Software Test Documentation (IEEE
Std 829-1998) (K2)
LO-5.2.3 Differentiate between conceptually different test approaches, such as analytical, modelbased, methodical, process/standard compliant, dynamic/heuristic, consultative and
regression-averse (K2)
LO-5.2.4 Differentiate between the subject of test planning for a system and scheduling test
execution (K2)
LO-5.2.5 Write a test execution schedule for a given set of test cases, considering prioritization, and
technical and logical dependencies (K3)
LO-5.2.6 List test preparation and execution activities that should be considered during test
planning (K1)
LO-5.2.7 Recall typical factors that influence the effort related to testing (K1)
LO-5.2.8 Differentiate between two conceptually different estimation approaches: the metricsbased approach and the expert-based approach (K2)
LO-5.2.9 Recognize/justify adequate entry and exit criteria for specific test levels and groups of test
cases (e.g., for integration testing, acceptance testing or test cases for usability testing)
(K2)

5.3 Test Progress Monitoring and Control (K2)

LO-5.3.1 Recall common metrics used for monitoring test preparation and execution (K1)
LO-5.3.2 Explain and compare test metrics for test reporting and test control (e.g., defects found
and fixed, and tests passed and failed) related to purpose and use (K2)
LO-5.3.3 Summarize the purpose and content of the test summary report document according to the
Standard for Software Test Documentation (IEEE Std 829-1998) (K2)

5.4 Configuration Management (K2)

LO-5.4.1 Summarize how configuration management supports testing (K2)

5.5 Risk and Testing (K2)

LO-5.5.1 Describe a risk as a possible problem that would threaten the achievement of one or more
stakeholders project objectives (K2)
LO-5.5.2 Remember that the level of risk is determined by likelihood (of happening) and impact
(harm resulting if it does happen) (K1)
LO-5.5.3 Distinguish between the project and product risks (K2)
LO-5.5.4 Recognize typical product and project risks (K1)
LO-5.5.5 Describe, using examples, how risk analysis and risk management may be used for test
planning (K2)

5.6 Incident Management (K3)

LO-5.6.1 Recognize the content of an incident report according to the Standard for Software Test
Documentation (IEEE Std 829-1998) (K1)
LO-5.6.2 Write an incident report covering the observation of a failure during testing. (K3)

5.1

Test Organization (K2)

30 minutes

Terms
Tester, test leader, test manager

5.1.1 Test Organization and Independence (K2)

The effectiveness of finding defects by testing and reviews can be improved by using independent
testers. Options for independence include the following:
No independent testers; developers test their own code
Independent testers within the development teams
Independent test team or group within the organization, reporting to project management or
executive management
Independent testers from the business organization or user community
Independent test specialists for specific test types such as usability testers, security testers or
certification testers (who certify a software product against standards and regulations)
Independent testers outsourced or external to the organization
For large, complex or safety critical projects, it is usually best to have multiple levels of testing, with
some or all of the levels done by independent testers. Development staff may participate in testing,
especially at the lower levels, but their lack of objectivity often limits their effectiveness. The
independent testers may have the authority to require and define test processes and rules, but testers
should take on such process-related roles only in the presence of a clear management mandate to do
so.
The benefits of independence include:
Independent testers see other and different defects, and are unbiased
An independent tester can verify assumptions people made during specification and
implementation of the system
Drawbacks include:
Isolation from the development team (if treated as totally independent)
Developers may lose a sense of responsibility for quality
Independent testers may be seen as a bottleneck or blamed for delays in release
Testing tasks may be done by people in a specific testing role, or may be done by someone in another
role, such as a project manager, quality manager, developer, business and domain expert,
infrastructure or IT operations.

5.1.2 Tasks of the Test Leader and Tester (K1)

In this syllabus two test positions are covered, test leader and tester. The activities and tasks
performed by people in these two roles depend on the project and product context, the people in the
roles, and the organization.
Sometimes the test leader is called a test manager or test coordinator. The role of the test leader may
be performed by a project manager, a development manager, a quality assurance manager or the
manager of a test group. In larger projects two positions may exist: test leader and test manager.
Typically the test leader plans, monitors and controls the testing activities and tasks as defined in
Section 1.4.
Typical test leader tasks may include:
Coordinate the test strategy and plan with project managers and others
Write or review a test strategy for the project, and test policy for the organization
Contribute the testing perspective to other project activities, such as integration planning
Plan the tests considering the context and understanding the test objectives and risks
including selecting test approaches, estimating the time, effort and cost of testing, acquiring
resources, defining test levels, cycles, and planning incident management
Initiate the specification, preparation, implementation and execution of tests, monitor the test
results and check the exit criteria
Adapt planning based on test results and progress (sometimes documented in status reports) and
take any action necessary to compensate for problems
Set up adequate configuration management of testware for traceability
Introduce suitable metrics for measuring test progress and evaluating the quality of the testing and
the product
Decide what should be automated, to what degree, and how
Select tools to support testing and organize any training in tool use for testers
Decide about the implementation of the test environment
Write test summary reports based on the information gathered during testing
Typical tester tasks may include:
Review and contribute to test plans
Analyze, review and assess user requirements, specifications and models for testability
Create test specifications
Set up the test environment (often coordinating with system administration and network
management)
Prepare and acquire test data
Implement tests on all test levels, execute and log the tests, evaluate the results and document the
deviations from expected results
Use test administration or management tools and test monitoring tools as required
Automate tests (may be supported by a developer or a test automation expert)
Measure performance of components and systems (if applicable)

Review tests developed by others

People who work on test analysis, test design, specific test types or test automation may be
specialists in these roles. Depending on the test level and the risks related to the product and the
project, different people may take over the role of tester, keeping some degree of independence.
Typically testers at the component and integration level would be developers, testers at the
acceptance test level would be business experts and users, and testers for operational acceptance
testing would be operators.

5.2

Test Planning and Estimation (K3)

40 minutes

Terms
Test approach, test strategy

5.2.1 Test Planning (K2)

This section covers the purpose of test planning within development and implementation projects, and
for maintenance activities. Planning may be documented in a master test plan and in separate test
plans for test levels such as system testing and acceptance testing. The outline of a test-planning
document is covered by the Standard for Software Test Documentation (IEEE Std 829-1998).
Planning is influenced by the test policy of the organization, the scope of testing, objectives, risks,
constraints, criticality, testability and the availability of resources. As the project and test planning
progress, more information becomes available and more detail can be included in the plan.
Test planning is a continuous activity and is performed in all life cycle processes and activities.
Feedback from test activities is used to recognize changing risks so that planning can be adjusted.

5.2.2 Test Planning Activities (K3)

Test planning activities for an entire system or part of a system may include:
Determining the scope and risks and identifying the objectives of testing
Defining the overall approach of testing, including the definition of the test levels and entry and
exit criteria
Integrating and coordinating the testing activities into the software life cycle activities
(acquisition, supply, development, operation and maintenance)
Making decisions about what to test, what roles will perform the test activities, how the test
activities should be done, and how the test results will be evaluated
Scheduling test analysis and design activities
Scheduling test implementation, execution and evaluation
Assigning resources for the different activities defined
Defining the amount, level of detail, structure and templates for the test documentation
Selecting metrics for monitoring and controlling test preparation and execution, defect resolution
and risk issues
Setting the level of detail for test procedures in order to provide enough information to support
reproducible test preparation and execution

5.2.3 Entry Criteria (K2)

Entry criteria define when to start testing such as at the beginning of a test level or when a set of tests
is ready for execution.
Typically entry criteria may cover the following:
Test environment availability and readiness
Test tool readiness in the test environment
Testable code availability
Test data availability

5.2.4 Exit Criteria (K2)

Exit criteria define when to stop testing such as at the end of a test level or when a set of tests has
achieved specific goal.
Typically exit criteria may cover the following:
Thoroughness measures, such as coverage of code, functionality or risk
Estimates of defect density or reliability measures
Cost
Residual risks, such as defects not fixed or lack of test coverage in certain areas
Schedules such as those based on time to market

5.2.5 Test Estimation (K2)

Two approaches for the estimation of test effort are:
The metrics-based approach: estimating the testing effort based on metrics of former or similar
projects or based on typical values
The expert-based approach: estimating the tasks based on estimates made by the owner of the
tasks or by experts
Once the test effort is estimated, resources can be identified and a schedule can be drawn up.
The testing effort may depend on a number of factors, including:
Characteristics of the product: the quality of the specification and other information used for test
models (i.e., the test basis), the size of the product, the complexity of the problem domain, the
requirements for reliability and security, and the requirements for documentation
Characteristics of the development process: the stability of the organization, tools used, test
process, skills of the people involved, and time pressure
The outcome of testing: the number of defects and the amount of rework required

5.2.6 Test Strategy, Test Approach (K2)

The test approach is the implementation of the test strategy for a specific project. The test approach is
defined and refined in the test plans and test designs. It typically includes the decisions made based
on the (test) projects goal and risk assessment. It is the starting point for planning the test process, for

selecting the test design techniques and test types to be applied, and for defining the entry and exit
criteria.
The selected approach depends on the context and may consider risks, hazards and safety, available
resources and skills, the technology, the nature of the system (e.g., custom built vs. COTS), test
objectives, and regulations.
Typical approaches include:
Analytical approaches, such as risk-based testing where testing is directed to areas of greatest
risk
Model-based approaches, such as stochastic testing using statistical information about failure
rates (such as reliability growth models) or usage (such as operational profiles)
Methodical approaches, such as failure-based (including error guessing and fault attacks),
experience-based, checklist-based, and quality characteristic-based
Process- or standard-compliant approaches, such as those specified by industry-specific
standards or the various agile methodologies
Dynamic and heuristic approaches, such as exploratory testing where testing is more reactive to
events than pre-planned, and where execution and evaluation are concurrent tasks
Consultative approaches, such as those in which test coverage is driven primarily by the advice
and guidance of technology and/or business domain experts outside the test team
Regression-averse approaches, such as those that include reuse of existing test material,
extensive automation of functional regression tests, and standard test suites
Different approaches may be combined, for example, a risk-based dynamic approach.

5.3

Test Progress Monitoring and Control (K2)

20 minutes

Terms
Defect density, failure rate, test control, test monitoring, test summary report

5.3.1 Test Progress Monitoring (K1)

The purpose of test monitoring is to provide feedback and visibility about test activities. Information
to be monitored may be collected manually or automatically and may be used to measure exit criteria,
such as coverage. Metrics may also be used to assess progress against the planned schedule and
budget. Common test metrics include:
Percentage of work done in test case preparation (or percentage of planned test cases prepared)
Percentage of work done in test environment preparation
Test case execution (e.g., number of test cases run/not run, and test cases passed/failed)
Defect information (e.g., defect density, defects found and fixed, failure rate, and re-test results)
Test coverage of requirements, risks or code
Subjective confidence of testers in the product
Dates of test milestones
Testing costs, including the cost compared to the benefit of finding the next defect or to run the
next test

5.3.2 Test Reporting (K2)

Test reporting is concerned with summarizing information about the testing endeavor, including:
What happened during a period of testing, such as dates when exit criteria were met
Analyzed information and metrics to support recommendations and decisions about future
actions, such as an assessment of defects remaining, the economic benefit of continued testing,
outstanding risks, and the level of confidence in the tested software
The outline of a test summary report is given in Standard for Software Test Documentation (IEEE
Std 829-1998).
Metrics should be collected during and at the end of a test level in order to assess:
The adequacy of the test objectives for that test level
The adequacy of the test approaches taken
The effectiveness of the testing with respect to the objectives

5.3.3 Test Control (K2)

Test control describes any guiding or corrective actions taken as a result of information and metrics
gathered and reported. Actions may cover any test activity and may affect any other software life

cycle activity or task.

Examples of test control actions include:
Making decisions based on information from test monitoring
Re-prioritizing tests when an identified risk occurs (e.g., software delivered late)
Changing the test schedule due to availability or unavailability of a test environment
Setting an entry criterion requiring fixes to have been re-tested (confirmation tested) by a
developer before accepting them into a build

5.4

Configuration Management (K2)

10 minutes

Terms
Configuration management, version control

Background
The purpose of configuration management is to establish and maintain the integrity of the products
(components, data and documentation) of the software or system through the project and product life
cycle.
For testing, configuration management may involve ensuring the following:
All items of testware are identified, version controlled, tracked for changes, related to each other
and related to development items (test objects) so that traceability can be maintained throughout
the test process
All identified documents and software items are referenced unambiguously in test documentation
For the tester, configuration management helps to uniquely identify (and to reproduce) the tested item,
test documents, the tests and the test harness(es).
During test planning, the configuration management procedures and infrastructure (tools) should be
chosen, documented and implemented.

5.5

Risk and Testing (K2)

30 minutes

Terms
Product risk, project risk, risk, risk-based testing

Background
Risk can be defined as the chance of an event, hazard, threat or situation occurring and resulting in
undesirable consequences or a potential problem. The level of risk will be determined by the
likelihood of an adverse event happening and the impact (the harm resulting from that event).

5.5.1 Project Risks (K2)

Project risks are the risks that surround the projects capability to deliver its objectives, such as:
Organizational factors:
Skill, training and staff shortages
Personnel issues
Political issues, such as:
Problems with testers communicating their needs and test results
Failure by the team to follow up on information found in testing and reviews (e.g., not
improving development and testing practices)
Improper attitude toward or expectations of testing (e.g., not appreciating the value of finding
defects during testing)

Technical issues:
Problems in defining the right requirements
The extent to which requirements cannot be met given existing constraints
Test environment not ready on time
Late data conversion, migration planning and development and testing data conversion/migration
tools
Low quality of the design, code, configuration data, test data and tests

Supplier issues:
Failure of a third party
Contractual issues

When analyzing, managing and mitigating these risks, the test manager is following well-established
project management principles. The Standard for Software Test Documentation (IEEE Std 8291998) outline for test plans requires risks and contingencies to be stated.

5.5.2 Product Risks (K2)

Potential failure areas (adverse future events or hazards) in the software or system are known as
product risks, as they are a risk to the quality of the product. These include:
Failure-prone software delivered
The potential that the software/hardware could cause harm to an individual or company
Poor software characteristics (e.g., functionality, reliability, usability and performance)
Poor data integrity and quality (e.g., data migration issues, data conversion problems, data
transport problems, violation of data standards)
Software that does not perform its intended functions
Risks are used to decide where to start testing and where to test more; testing is used to reduce the
risk of an adverse effect occurring, or to reduce the impact of an adverse effect.
Product risks are a special type of risk to the success of a project. Testing as a risk-control activity
provides feedback about the residual risk by measuring the effectiveness of critical defect removal
and of contingency plans.
A risk-based approach to testing provides proactive opportunities to reduce the levels of product
risk, starting in the initial stages of a project. It involves the identification of product risks and their
use in guiding test planning and control, specification, preparation and execution of tests. In a riskbased approach the risks identified may be used to:
Determine the test techniques to be employed
Determine the extent of testing to be carried out
Prioritize testing in an attempt to find the critical defects as early as possible
Determine whether any non-testing activities could be employed to reduce risk (e.g., providing
training to inexperienced designers)
Risk-based testing draws on the collective knowledge and insight of the project stakeholders to
determine the risks and the levels of testing required to address those risks.
To ensure that the chance of a product failure is minimized, risk management activities provide a
disciplined approach to:
Assess (and reassess on a regular basis) what can go wrong (risks)
Determine what risks are important to deal with
Implement actions to deal with those risks
In addition, testing may support the identification of new risks, may help to determine what risks
should be reduced, and may lower uncertainty about risks.

5.6

Incident Management (K3)

40 minutes

Terms
Incident logging, incident management, incident report

Background
Since one of the objectives of testing is to find defects, the discrepancies between actual and
expected outcomes need to be logged as incidents. An incident must be investigated and may turn out
to be a defect. Appropriate actions to dispose incidents and defects should be defined. Incidents and
defects should be tracked from discovery and classification to correction and confirmation of the
solution. In order to manage all incidents to completion, an organization should establish an incident
management process and rules for classification.
Incidents may be raised during development, review, testing or use of a software product. They may
be raised for issues in code or the working system, or in any type of documentation including
requirements, development documents, test documents, and user information such as Help or
installation guides.
Incident reports have the following objectives:
Provide developers and other parties with feedback about the problem to enable identification,
isolation and correction as necessary
Provide test leaders a means of tracking the quality of the system under test and the progress of
the testing
Provide ideas for test process improvement
Details of the incident report may include:
Date of issue, issuing organization, and author
Expected and actual results
Identification of the test item (configuration item) and environment
Software or system life cycle process in which the incident was observed
Description of the incident to enable reproduction and resolution, including logs, database dumps
or screenshots
Scope or degree of impact on stakeholder(s) interests
Severity of the impact on the system
Urgency/priority to fix
Status of the incident (e.g., open, deferred, duplicate, waiting to be fixed, fixed awaiting re-test,
closed)
Conclusions, recommendations and approvals

Global issues, such as other areas that may be affected by a change resulting from the incident
Change history, such as the sequence of actions taken by project team members with respect to
the incident to isolate, repair, and confirm it as fixed
References, including the identity of the test case specification that revealed the problem

The structure of an incident report is also covered in the Standard for Software Test Documentation
(IEEE Std 829-1998).

References
5.1.1 Black, 2001, Hetzel, 1988
5.1.2 Black, 2001, Hetzel, 1988
5.2.5 Black, 2001, Craig, 2002, IEEE Std 829-1998, Kaner 2002
5.3.3 Black, 2001, Craig, 2002, Hetzel, 1988, IEEE Std 829-1998
5.4 Craig, 2002
5.5.2 Black, 2001, IEEE Std 829-1998
5.6 Black, 2001, IEEE Std 829-1998

6.

Tool Support for Testing (K2)

80 minutes

Learning Objectives for Tool Support for Testing

The objectives identify what you will be able to do following the completion of each module.

6.1 Types of Test Tools (K2)

LO-6.1.1 Classify different types of test tools according to their purpose and to the activities of the
fundamental test process and the software life cycle (K2)
LO-6.1.3 Explain the term test tool and the purpose of tool support for testing (K2) 2

6.2 Effective Use of Tools: Potential Benefits and Risks (K2)

LO-6.2.1 Summarize the potential benefits and risks of test automation and tool support for testing
(K2)
LO-6.2.2 Remember special considerations for test execution tools, static analysis, and test
management tools (K1)

6.3 Introducing a Tool into an Organization (K1)

LO-6.3.1 State the main principles of introducing a tool into an organization (K1)
LO-6.3.2 State the goals of a proof-of-concept for tool evaluation and a piloting phase for tool
implementation (K1)
LO-6.3.3 Recognize that factors other than simply acquiring a tool are required for good tool
support (K1)

6.1

Types of Test Tools (K2)

45 minutes

Terms
Configuration management tool, coverage tool, debugging tool, dynamic analysis tool, incident
management tool, load testing tool, modeling tool, monitoring tool, performance testing tool, probe
effect, requirements management tool, review tool, security tool, static analysis tool, stress testing
tool, test comparator, test data preparation tool, test design tool, test harness, test execution tool, test
management tool, unit test framework tool

6.1.1 Tool Support for Testing (K2)

Test tools can be used for one or more activities that support testing. These include:
1. Tools that are directly used in testing such as test execution tools, test data generation tools and
result comparison tools
2. Tools that help in managing the testing process such as those used to manage tests, test results,
data, requirements, incidents, defects, etc., and for reporting and monitoring test execution
3. Tools that are used in reconnaissance, or, in simple terms: exploration (e.g., tools that monitor
file activity for an application)
4. Any tool that aids in testing (a spreadsheet is also a test tool in this meaning)
Tool support for testing can have one or more of the following purposes depending on the context:
Improve the efficiency of test activities by automating repetitive tasks or supporting manual test
activities like test planning, test design, test reporting and monitoring
Automate activities that require significant resources when done manually (e.g., static testing)
Automate activities that cannot be executed manually (e.g., large scale performance testing of
client-server applications)
Increase reliability of testing (e.g., by automating large data comparisons or simulating behavior)
The term test frameworks is also frequently used in the industry, in at least three meanings:
Reusable and extensible testing libraries that can be used to build testing tools (called test
harnesses as well)
A type of design of test automation (e.g., data-driven, keyword-driven)
Overall process of execution of testing
For the purpose of this syllabus, the term test frameworks is used in its first two meanings as
described in Section 6.1.6.

6.1.2 Test Tool Classification (K2)

There are a number of tools that support different aspects of testing. Tools can be classified based on
several criteria such as purpose, commercial / free / open-source / shareware, technology used and

so forth. Tools are classified in this syllabus according to the testing activities that they support.
Some tools clearly support one activity; others may support more than one activity, but are classified
under the activity with which they are most closely associated. Tools from a single provider,
especially those that have been designed to work together, may be bundled into one package.
Some types of test tools can be intrusive, which means that they can affect the actual outcome of the
test. For example, the actual timing may be different due to the extra instructions that are executed by
the tool, or you may get a different measure of code coverage. The consequence of intrusive tools is
called the probe effect.
Some tools offer support more appropriate for developers (e.g., tools that are used during component
and component integration testing). Such tools are marked with (D) in the list below.

6.1.3 Tool Support for Management of Testing and Tests (K1)

Management tools apply to all test activities over the entire software life cycle.
Test Management Tools
These tools provide interfaces for executing tests, tracking defects and managing requirements, along
with support for quantitative analysis and reporting of the test objects. They also support tracing the
test objects to requirement specifications and might have an independent version control capability or
an interface to an external one.
Requirements Management Tools
These tools store requirement statements, store the attributes for the requirements (including priority),
provide unique identifiers and support tracing the requirements to individual tests. These tools may
also help with identifying inconsistent or missing requirements.
Incident Management Tools (Defect Tracking Tools)
These tools store and manage incident reports, i.e., defects, failures, change requests or perceived
problems and anomalies, and help in managing the life cycle of incidents, optionally with support for
statistical analysis.
Configuration Management Tools
Although not strictly test tools, these are necessary for storage and version management of testware
and related software especially when configuring more than one hardware/software environment in
terms of operating system versions, compilers, browsers, etc.

6.1.4 Tool Support for Static Testing (K1)

Static testing tools provide a cost effective way of finding more defects at an earlier stage in the
development process.
Review Tools

These tools assist with review processes, checklists, review guidelines and are used to store and
communicate review comments and report on defects and effort. They can be of further help by
providing aid for online reviews for large or geographically dispersed teams.
Static Analysis Tools (D)
These tools help developers and testers find defects prior to dynamic testing by providing support for
enforcing coding standards (including secure coding), analysis of structures and dependencies. They
can also help in planning or risk analysis by providing metrics for the code (e.g., complexity).
Modeling Tools (D)
These tools are used to validate software models (e.g., physical data model (PDM) for a relational
database), by enumerating inconsistencies and finding defects. These tools can often aid in generating
some test cases based on the model.

6.1.5 Tool Support for Test Specification (K1)

Test Design Tools
These tools are used to generate test inputs or executable tests and/or test oracles from requirements,
graphical user interfaces, design models (state, data or object) or code.
Test Data Preparation Tools
Test data preparation tools manipulate databases, files or data transmissions to set up test data to be
used during the execution of tests to ensure security through data anonymity.

6.1.6 Tool Support for Test Execution and Logging (K1)

Test Execution Tools
These tools enable tests to be executed automatically, or semi-automatically, using stored inputs and
expected outcomes, through the use of a scripting language and usually provide a test log for each test
run. They can also be used to record tests, and usually support scripting languages or GUI-based
configuration for parameterization of data and other customization in the tests.
Test Harness/Unit Test Framework Tools (D)
A unit test harness or framework facilitates the testing of components or parts of a system by
simulating the environment in which that test object will run, through the provision of mock objects as
stubs or drivers.
Test Comparators
Test comparators determine differences between files, databases or test results. Test execution tools
typically include dynamic comparators, but post-execution comparison may be done by a separate
comparison tool. A test comparator may use a test oracle, especially if it is automated.
Coverage Measurement Tools (D)

These tools, through intrusive or non-intrusive means, measure the percentage of specific types of
code structures that have been exercised (e.g., statements, branches or decisions, and module or
function calls) by a set of tests.
Security Testing Tools
These tools are used to evaluate the security characteristics of software. This includes evaluating the
ability of the software to protect data confidentiality, integrity, authentication, authorization,
availability, and non-repudiation. Security tools are mostly focused on a particular technology,
platform, and purpose.

6.1.7 Tool Support for Performance and Monitoring (K1)

Dynamic Analysis Tools (D)
Dynamic analysis tools find defects that are evident only when software is executing, such as time
dependencies or memory leaks. They are typically used in component and component integration
testing, and when testing middleware.
Performance Testing/Load Testing/Stress Testing Tools
Performance testing tools monitor and report on how a system behaves under a variety of simulated
usage conditions in terms of number of concurrent users, their ramp-up pattern, frequency and relative
percentage of transactions. The simulation of load is achieved by means of creating virtual users
carrying out a selected set of transactions, spread across various test machines commonly known as
load generators.
Monitoring Tools
Monitoring tools continuously analyze, verify and report on usage of specific system resources, and
give warnings of possible service problems.

6.1.8 Tool Support for Specific Testing Needs (K1)

Data Quality Assessment
Data is at the center of some projects such as data conversion/migration projects and applications
like data warehouses and its attributes can vary in terms of criticality and volume. In such contexts,
tools need to be employed for data quality assessment to review and verify the data conversion and
migration rules to ensure that the processed data is correct, complete and complies with a pre-defined
context-specific standard.
Other testing tools exist for usability testing.

6.2 Effective Use of Tools: Potential Benefits and Risks

(K2)

20
minutes

Terms
Data-driven testing, keyword-driven testing, scripting language

6.2.1 Potential Benefits and Risks of Tool Support for Testing (for all tools) (K2)
Simply purchasing or leasing a tool does not guarantee success with that tool. Each type of tool may
require additional effort to achieve real and lasting benefits. There are potential benefits and
opportunities with the use of tools in testing, but there are also risks.
Potential benefits of using tools include:
Repetitive work is reduced (e.g., running regression tests, re-entering the same test data, and
checking against coding standards)
Greater consistency and repeatability (e.g., tests executed by a tool in the same order with the
same frequency, and tests derived from requirements)
Objective assessment (e.g., static measures, coverage)
Ease of access to information about tests or testing (e.g., statistics and graphs about test progress,
incident rates and performance)
Risks of using tools include:
Unrealistic expectations for the tool (including functionality and ease of use)
Underestimating the time, cost and effort for the initial introduction of a tool (including training
and external expertise)
Underestimating the time and effort needed to achieve significant and continuing benefits from the
tool (including the need for changes in the testing process and continuous improvement of the
way the tool is used)
Underestimating the effort required to maintain the test assets generated by the tool
Over-reliance on the tool (replacement for test design or use of automated testing where manual
testing would be better)
Neglecting version control of test assets within the tool
Neglecting relationships and interoperability issues between critical tools, such as requirements
management tools, version control tools, incident management tools, defect tracking tools and
tools from multiple vendors
Risk of tool vendor going out of business, retiring the tool, or selling the tool to a different
vendor
Poor response from vendor for support, upgrades, and defect fixes
Risk of suspension of open-source / free tool project

Unforeseen, such as the inability to support a new platform

6.2.2 Special Considerations for Some Types of Tools (K1)

Test Execution Tools
Test execution tools execute test objects using automated test scripts. This type of tool often requires
significant effort in order to achieve significant benefits.
Capturing tests by recording the actions of a manual tester seems attractive, but this approach does not
scale to large numbers of automated test scripts. A captured script is a linear representation with
specific data and actions as part of each script. This type of script may be unstable when unexpected
events occur.
A data-driven testing approach separates out the test inputs (the data), usually into a spreadsheet, and
uses a more generic test script that can read the input data and execute the same test script with
different data. Testers who are not familiar with the scripting language can then create the test data for
these predefined scripts.
There are other techniques employed in data-driven techniques, where instead of hard-coded data
combinations placed in a spreadsheet, data is generated using algorithms based on configurable
parameters at run time and supplied to the application. For example, a tool may use an algorithm,
which generates a random user ID, and for repeatability in pattern, a seed is employed for controlling
randomness.
In a keyword-driven testing approach, the spreadsheet contains keywords describing the actions to be
taken (also called action words), and test data. Testers (even if they are not familiar with the scripting
language) can then define tests using the keywords, which can be tailored to the application being
tested.
Technical expertise in the scripting language is needed for all approaches (either by testers or by
specialists in test automation).
Regardless of the scripting technique used, the expected results for each test need to be stored for
later comparison.
Static Analysis Tools
Static analysis tools applied to source code can enforce coding standards, but if applied to existing
code may generate a large quantity of messages. Warning messages do not stop the code from being
translated into an executable program, but ideally should be addressed so that maintenance of the
code is easier in the future. A gradual implementation of the analysis tool with initial filters to
exclude some messages is an effective approach.
Test Management Tools
Test management tools need to interface with other tools or spreadsheets in order to produce useful
information in a format that fits the needs of the organization.

6.3

Introducing a Tool into an Organization (K1)

15 minutes

Terms
No specific terms.

Background
The main considerations in selecting a tool for an organization include:
Assessment of organizational maturity, strengths and weaknesses and identification of
opportunities for an improved test process supported by tools
Evaluation against clear requirements and objective criteria
A proof-of-concept, by using a test tool during the evaluation phase to establish whether it
performs effectively with the software under test and within the current infrastructure or to
identify changes needed to that infrastructure to effectively use the tool
Evaluation of the vendor (including training, support and commercial aspects) or service support
suppliers in case of non-commercial tools
Identification of internal requirements for coaching and mentoring in the use of the tool
Evaluation of training needs considering the current test teams test automation skills
Estimation of a cost-benefit ratio based on a concrete business case
Introducing the selected tool into an organization starts with a pilot project, which has the following
objectives:
Learn more detail about the tool
Evaluate how the tool fits with existing processes and practices, and determine what would need
to change
Decide on standard ways of using, managing, storing and maintaining the tool and the test assets
(e.g., deciding on naming conventions for files and tests, creating libraries and defining the
modularity of test suites)
Assess whether the benefits will be achieved at reasonable cost
Success factors for the deployment of the tool within an organization include:
Rolling out the tool to the rest of the organization incrementally
Adapting and improving processes to fit with the use of the tool
Providing training and coaching/mentoring for new users
Defining usage guidelines
Implementing a way to gather usage information from the actual use
Monitoring tool use and benefits
Providing support for the test team for a given tool

Gathering lessons learned from all teams

References
6.2.2 Buwalda, 2001, Fewster, 1999
6.3 Fewster, 1999
________________________
2

LO-6.1.2 Intentionally skipped

7. References
Standards
ISTQB Glossary of Terms used in Software Testing Version 2.1
[CMMI] Chrissis, M.B., Konrad, M. and Shrum, S. (2004) CMMI, Guidelines for Process Integration
and Product Improvement, Addison Wesley: Reading, MA See Section 2.1
[IEEE Std 829-1998] IEEE Std 829 (1998) IEEE Standard for Software Test Documentation, See
Sections 2.3, 2.4, 4.1, 5.2, 5.3, 5.5, 5.6
[IEEE 1028] IEEE Std 1028 (2008) IEEE Standard for Software Reviews and Audits, See Section
3.2
[IEEE 12207] IEEE 12207/ISO/IEC 12207-2008, Software life cycle processes, See Section 2.1
[ISO 9126] ISO/IEC 9126-1:2001, Software Engineering Software Product Quality, See Section
2.3

Books
[Beizer, 1990] Beizer, B. (1990) Software Testing Techniques (2nd edition), Van Nostrand Reinhold:
Boston See Sections 1.2, 1.3, 2.3, 4.2, 4.3, 4.4, 4.6
[Black, 2001] Black, R. (2001) Managing the Testing Process (3rd edition), John Wiley & Sons:
New York See Sections 1.1, 1.2, 1.4, 1.5, 2.3, 2.4, 5.1, 5.2, 5.3, 5.5, 5.6
[Buwalda, 2001] Buwalda, H. et al. (2001) Integrated Test Design and Automation, Addison
Wesley: Reading, MA See Section 6.2
[Copeland, 2004] Copeland, L. (2004) A Practitioners Guide to Software Test Design, Artech
House: Norwood, MA See Sections 2.2, 2.3, 4.2, 4.3, 4.4, 4.6
[Craig, 2002] Craig, Rick D. and Jaskiel, Stefan P. (2002) Systematic Software Testing, Artech
House: Norwood, MA See Sections 1.4.5, 2.1.3, 2.4, 4.1, 5.2.5, 5.3, 5.4
[Fewster, 1999] Fewster, M. and Graham, D. (1999) Software Test Automation, Addison Wesley:
Reading, MA See Sections 6.2, 6.3
[Gilb, 1993]: Gilb, Tom and Graham, Dorothy (1993) Software Inspection, Addison Wesley:
Reading, MA See Sections 3.2.2, 3.2.4
[Hetzel, 1988] Hetzel, W. (1988) Complete Guide to Software Testing, QED: Wellesley, MA See
Sections 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 2.4, 4.1, 5.1, 5.3
[Kaner, 2002] Kaner, C., Bach, J. and Pettticord, B. (2002) Lessons Learned in Software Testing,
John Wiley & Sons: New York See Sections 1.1, 4.5, 5.2
[Myers 1979] Myers, Glenford J. (1979) The Art of Software Testing, John Wiley & Sons: New York
See Sections 1.2, 1.3, 2.2, 4.3

[van Veenendaal, 2004] van Veenendaal, E. (ed.) (2004) The Testing Practitioner (Chapters 6, 8,
10), UTN Publishers: The Netherlands See Sections 3.2, 3.3

8. Appendix A Syllabus Background

History of this Document
This document was prepared between 2004 and 2011 by a Working Group comprised of members
appointed by the International Software Testing Qualifications Board (ISTQB). It was initially
reviewed by a selected review panel, and then by representatives drawn from the international
software testing community. The rules used in the production of this document are shown in Appendix
C.
This document is the syllabus for the International Foundation Certificate in Software Testing, the first
level international qualification approved by the ISTQB (www.istqb.org).

Objectives of the Foundation Certificate Qualification

To gain recognition for testing as an essential and professional software engineering

specialization
To provide a standard framework for the development of testers careers
To enable professionally qualified testers to be recognized by employers, customers and peers,
and to raise the profile of testers
To promote consistent and good testing practices within all software engineering disciplines
To identify testing topics that are relevant and of value to industry
To enable software suppliers to hire certified testers and thereby gain commercial advantage
over their competitors by advertising their tester recruitment policy
To provide an opportunity for testers and those with an interest in testing to acquire an
internationally recognized qualification in the subject

Objectives of the International Qualification (adapted from ISTQB meeting at

Sollentuna, November 2001)

To be able to compare testing skills across different countries

To enable testers to move across country borders more easily
To enable multinational/international projects to have a common understanding of testing issues
To increase the number of qualified testers worldwide
To have more impact/value as an internationally-based initiative than from any country-specific
approach
To develop a common international body of understanding and knowledge about testing through
the syllabus and terminology, and to increase the level of knowledge about testing for all
participants
To promote testing as a profession in more countries
To enable testers to gain a recognized qualification in their native language
To enable sharing of knowledge and resources across countries

To provide international recognition of testers and this qualification due to participation from
many countries

Entry Requirements for this Qualification

The entry criterion for taking the ISTQB Foundation Certificate in Software Testing examination is
that candidates have an interest in software testing. However, it is strongly recommended that
candidates also:
Have at least a minimal background in either software development or software testing, such as
six months experience as a system or user acceptance tester or as a software developer
Take a course that has been accredited to ISTQB standards (by one of the ISTQB-recognized
National Boards).

Background and History of the Foundation Certificate in Software Testing

The independent certification of software testers began in the UK with the British Computer Societys
Information Systems Examination Board (ISEB), when a Software Testing Board was set up in 1998
(www.bcs.org.uk/iseb). In 2002, ASQF in Germany began to support a German tester qualification
scheme (www.asqf.de). This syllabus is based on the ISEB and ASQF syllabi; it includes
reorganized, updated and additional content, and the emphasis is directed at topics that will provide
the most practical help to testers.
An existing Foundation Certificate in Software Testing (e.g., from ISEB, ASQF or an ISTQBrecognized National Board) awarded before this International Certificate was released, will be
deemed to be equivalent to the International Certificate. The Foundation Certificate does not expire
and does not need to be renewed. The date it was awarded is shown on the Certificate.
Within each participating country, local aspects are controlled by a national ISTQB-recognized
Software Testing Board. Duties of National Boards are specified by the ISTQB, but are implemented
within each country. The duties of the country boards are expected to include accreditation of training
providers and the setting of exams.

9. Appendix B Learning Objectives/Cognitive Level of

Knowledge
The following learning objectives are defined as applying to this syllabus. Each topic in the syllabus
will be examined according to the learning objective for it.

Level 1: Remember (K1)

The candidate will recognize, remember and recall a term or concept.
Keywords: Remember, retrieve, recall, recognize, know
Example
Can recognize the definition of failure as:
Non-delivery of service to an end user or any other stakeholder or
Actual deviation of the component or system from its expected delivery, service or result

Level 2: Understand (K2)

The candidate can select the reasons or explanations for statements related to the topic, and can
summarize, compare, classify, categorize and give examples for the testing concept.
Keywords: Summarize, generalize, abstract, classify, compare, map, contrast, exemplify, interpret,
translate, represent, infer, conclude, categorize, construct models
Examples
Can explain the reason why tests should be designed as early as possible:
To find defects when they are cheaper to remove
To find the most important defects first
Can explain the similarities and differences between integration and system testing:
Similarities: testing more than one component, and can test non-functional aspects
Differences: integration testing concentrates on interfaces and interactions, and system testing
concentrates on whole-system aspects, such as end-to-end processing

Level 3: Apply (K3)

The candidate can select the correct application of a concept or technique and apply it to a given
context.
Keywords: Implement, execute, use, follow a procedure, apply a procedure
Example
Can identify boundary values for valid and invalid partitions
Can select test cases from a given state transition diagram in order to cover all transitions

Level 4: Analyze (K4)

The candidate can separate information related to a procedure or technique into its constituent parts
for better understanding, and can distinguish between facts and inferences. Typical application is to
analyze a document, software or project situation and propose appropriate actions to solve a problem
or task.
Keywords: Analyze, organize, find coherence, integrate, outline, parse, structure, attribute,
deconstruct, differentiate, discriminate, distinguish, focus, select
Example
Analyze product risks and propose preventive and corrective mitigation activities
Describe which portions of an incident report are factual and which are inferred from results

Reference
(For the cognitive levels of learning objectives)
Anderson, L. W. and Krathwohl, D. R. (eds) (2001) A Taxonomy for Learning, Teaching, and
Assessing: A Revision of Blooms Taxonomy of Educational Objectives, Allyn & Bacon

10. Appendix C Rules Applied to the ISTQB

Foundation Syllabus
The rules listed here were used in the development and review of this syllabus. (A TAG is shown
after each rule as a shorthand abbreviation of the rule.)

10.1.1 General Rules

SG1. The syllabus should be understandable and absorbable by people with zero to six months (or
more) experience in testing. (6-MONTH)
SG2. The syllabus should be practical rather than theoretical. (PRACTICAL)
SG3. The syllabus should be clear and unambiguous to its intended readers. (CLEAR)
SG4. The syllabus should be understandable to people from different countries, and easily
translatable into different languages. (TRANSLATABLE)
SG5. The syllabus should use American English. (AMERICAN-ENGLISH)

10.1.2 Current Content

SC1. The syllabus should include recent testing concepts and should reflect current best practices in
software testing where this is generally agreed. The syllabus is subject to review every three to five
years. (RECENT)
SC2. The syllabus should minimize time-related issues, such as current market conditions, to enable it
to have a shelf life of three to five years. (SHELF-LIFE).

10.1.3 Learning Objectives

LO1. Learning objectives should distinguish between items to be recognized/remembered (cognitive
level K1), items the candidate should understand conceptually (K2), items the candidate should be
able to practice/use (K3), and items the candidate should be able to use to analyze a document,
software or project situation in context (K4). (KNOWLEDGE-LEVEL)
LO2. The description of the content should be consistent with the learning objectives. (LOCONSISTENT)
LO3. To illustrate the learning objectives, sample exam questions for each major section should be
issued along with the syllabus. (LO-EXAM)

10.1.4 Overall Structure

ST1. The structure of the syllabus should be clear and allow cross-referencing to and from other
parts, from exam questions and from other relevant documents. (CROSS-REF)
ST2. Overlap between sections of the syllabus should be minimized. (OVERLAP)
ST3. Each section of the syllabus should have the same structure. (STRUCTURE-CONSISTENT)

ST4. The syllabus should contain version, date of issue and page number on every page. (VERSION)
ST5. The syllabus should include a guideline for the amount of time to be spent in each section (to
reflect the relative importance of each topic). (TIME-SPENT)

References
SR1. Sources and references will be given for concepts in the syllabus to help training providers find
out more information about the topic. (REFS)
SR2. Where there are not readily identified and clear sources, more detail should be provided in the
syllabus. For example, definitions are in the Glossary, so only the terms are listed in the syllabus.
(NON-REF DETAIL)

Sources of Information
Terms used in the syllabus are defined in the ISTQB Glossary of Terms used in Software Testing. A
version of the Glossary is available from ISTQB.
A list of recommended books on software testing is also issued in parallel with this syllabus. The
main book list is part of the References section.

11. Appendix D Notice to Training Providers

Each major subject heading in the syllabus is assigned an allocated time in minutes. The purpose of
this is both to give guidance on the relative proportion of time to be allocated to each section of an
accredited course, and to give an approximate minimum time for the teaching of each section.
Training providers may spend more time than is indicated and candidates may spend more time again
in reading and research. A course curriculum does not have to follow the same order as the syllabus.
The syllabus contains references to established standards, which must be used in the preparation of
training material. Each standard used must be the version quoted in the current version of this
syllabus. Other publications, templates or standards not referenced in this syllabus may also be used
and referenced, but will not be examined.
All K3 and K4 Learning Objectives require a practical exercise to be included in the training
materials.

12. Appendix E Release Notes

Release 2010
1.

2.
3.
4.
5.
6.
7.
8.
9.
10.

11.

Changes to Learning Objectives (LO) include some clarification

a. Wording changed for the following LOs (content and level of LO remains unchanged):
LO-1.2.2, LO-1.3.1, LO-1.4.1, LO-1.5.1, LO-2.1.1, LO-2.1.3, LO-2.4.2, LO-4.1.3,
LO-4.2.1, LO-4.2.2, LO-4.3.1, LO-4.3.2, LO-4.3.3, LO-4.4.1, LO-4.4.2, LO-4.4.3,
LO-4.6.1, LO-5.1.2, LO-5.2.2, LO-5.3.2, LO-5.3.3, LO-5.5.2, LO-5.6.1, LO-6.1.1,
LO-6.2.2, LO-6.3.2.
b. LO-1.1.5 has been reworded and upgraded to K2. Because a comparison of terms of
defect related terms can be expected.
c. LO-1.2.3 (K2) has been added. The content was already covered in the 2007
syllabus.
d. LO-3.1.3 (K2) now combines the content of LO-3.1.3 and LO-3.1.4.
e. LO-3.1.4 has been removed from the 2010 syllabus, as it is partially redundant with
LO-3.1.3.
f. LO-3.2.1 has been reworded for consistency with the 2010 syllabus content.
g. LO-3.3.2 has been modified, and its level has been changed from K1 to K2, for
consistency with LO-3.1.2.
h. LO 4.4.4 has been modified for clarity, and has been changed from a K3 to a K4.
Reason: LO-4.4.4 had already been written in a K4 manner.
i. LO-6.1.2 (K1) was dropped from the 2010 syllabus and was replaced with LO-6.1.3
(K2). There is no LO-6.1.2 in the 2010 syllabus.
Consistent use for test approach according to the definition in the glossary. The term test strategy
will not be required as term to recall.
Chapter 1.4 now contains the concept of traceability between test basis and test cases.
Chapter 2.x now contains test objects and test basis.
Re-testing is now the main term in the glossary instead of confirmation testing.
The aspect data quality and testing has been added at several locations in the syllabus: data
quality and risk in Chapter 2.2, 5.5, 6.1.8.
Chapter 5.2.3 Entry Criteria are added as a new subchapter. Reason: Consistency to Exit Criteria
(-> entry criteria added to LO-5.2.9).
Consistent use of the terms test strategy and test approach with their definition in the glossary.
Chapter 6.1 shortened because the tool descriptions were too large for a 45 minute lesson.
IEEE Std 829:2008 has been released. This version of the syllabus does not yet consider this
new edition. Section 5.2 refers to the document Master Test Plan. The content of the Master Test
Plan is covered by the concept that the document Test Plan covers different levels of planning:
Test plans for the test levels can be created as well as a test plan on the project level covering
multiple test levels. Latter is named Master Test Plan in this syllabus and in the ISTQB Glossary.
Code of Ethics has been moved from the CTAL to CTFL.

Release 2011
Changes made with the maintenance release 2011
1. General: Working Party replaced by Working Group
2. Replaced post-conditions by postconditions in order to be consistent with the ISTQB Glossary
2.1.
3. First occurrence: ISTQB replaced by ISTQB
4. Introduction to this Syllabus: Descriptions of Cognitive Levels of Knowledge removed, because
this was redundant to Appendix B.
5. Section 1.6: Because the intent was not to define a Learning Objective for the Code of Ethics,
the cognitive level for the section has been removed.
6. Section 2.2.1, 2.2.2, 2.2.3 and 2.2.4, 3.2.3: Fixed formatting issues in lists.
7. Section 2.2.2 The word failure was not correct for isolate failures to a specific component
. Therefore replaced with defect in that sentence.
8. Section 2.3: Corrected formatting of bullet list of test objectives related to test terms in section
Test Types (K2).
9. Section 2.3.4: Updated description of debugging to be consistent with Version 2.1 of the ISTQB
Glossary.
10. Section 2.4 removed word extensive from includes extensive regression testing, because the
extensive depends on the change (size, risks, value, etc.) as written in the next sentence.
11. Section 3.2: The word including has been removed to clarify the sentence.
12. Section 3.2.1: Because the activities of a formal review had been incorrectly formatted, the
review process had 12 main activities instead of six, as intended. It has been changed back to
six, which makes this section compliant with the Syllabus 2007 and the ISTQB Advanced Level
Syllabus 2007.
13. Section 4: Word developed replaced by defined because test cases get defined and not
developed.
14. Section 4.2: Text change to clarify how black-box and white-box testing could be used in
conjunction with experience-based techniques.
15. Section 4.3.5 text change ..between actors, including users and the system.. to between
actors (users or systems), .
16. Section 4.3.5 alternative path replaced by alternative scenario.
17. Section 4.4.2: In order to clarify the term branch testing in the text of Section 4.4, a sentence to
clarify the focus of branch testing has been changed.
18. Section 4.5, Section 5.2.6: The term experienced-based testing has been replaced by the
correct term experience-based.
19. Section 6.1: Heading 6.1.1 Understanding the Meaning and Purpose of Tool Support for Testing
(K2) replaced by 6.1.1 Tool Support for Testing (K2).
20. Section 7 / Books: The 3rd edition of [Black,2001] listed, replacing 2nd edition.
21. Appendix D: Chapters requiring exercises have been replaced by the generic requirement that all
Learning Objectives K3 and higher require exercises. This is a requirement specified in the
ISTQB Accreditation Process (Version 1.26).

22. Appendix E: The changed learning objectives between Version 2007 and 2010 are now correctly
listed.

13. Index
action word 63
alpha testing. 24, 27
architecture 15, 21, 22, 25, 28, 29
archiving 17, 30
automation 29
benefits of independence 47
benefits of using tool 62
beta testing 24, 27
black-box technique 37, 39, 40
black-box test design technique 39
black-box testing 28
bottom-up 25
boundary value analysis 40
bug 11
captured script 62
checklists 34, 35
choosing test technique 44
code coverage 28, 29, 37, 42, 58
commercial off the shelf (COTS) 22
compiler 36
complexity 11, 36, 50, 59
component integration testing 22, 25, 29, 59, 60
component testing 22, 24, 25, 27, 29, 37, 41, 42
configuration management 45, 48, 52
Configuration management tool 58
confirmation testing 13, 15, 16, 21, 28, 29
contract acceptance testing 27
control flow 28, 36, 37, 42
coverage 15, 24, 28, 29, 37, 38, 39, 40, 42, 50, 51, 58, 60, 62
coverage tool 58
custom-developed software 27
data flow 36
data-driven approach 63
data-driven testing 62
debugging 13, 24, 29, 58
debugging tool 24, 58
decision coverage 37, 42
decision table testing 40, 41
decision testing 42

defect 10, 11, 13, 14, 16, 18, 21, 24, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 39, 40, 41, 43, 44, 45, 47,
49, 50, 51, 53, 54, 55, 59, 60, 69
defect density 50, 51
defect tracking tool 59
development 8, 11, 12, 13, 14, 18, 21, 22, 24, 29, 32, 33, 36, 38, 44, 47, 49, 50, 52, 53, 55, 59, 67
development model 21, 22
drawbacks of independence 47
driver 24
dynamic analysis tool 58, 60
dynamic testing 13, 31, 32, 36
emergency change. 30
enhancement 27, 30
entry criteria 33
equivalence partitioning 40
error 10, 11, 18, 43, 50
error guessing 18, 43, 50
exhaustive testing 14
exit criteria 13, 15, 16, 33, 35, 45, 48, 49, 50, 51
expected result 16, 38, 48, 63
experience-based technique 37, 39, 43
experience-based test design technique 39
exploratory testing 43, 50
factory acceptance testing 27
failure 10, 11, 13, 14, 18, 21, 24, 26, 32, 36, 43, 46, 50, 51, 53, 54, 69
failure rate 50, 51
fault 10, 11, 43
fault attack 43
field testing 24, 27
follow-up 33, 34, 35
formal review 31, 33
functional requirement 24, 26
functional specification 28
functional task 25
functional test 28
functional testing 28
functionality 24, 25, 28, 50, 53, 62
impact analysis 21, 30, 38
incident 15, 16, 17, 19, 24, 46, 48, 55, 58, 59, 62
incident logging 55
incident management 48, 55, 58
incident management tool 58, 59
incident report 46, 55
independence 18, 47, 48
informal review 31, 33, 34
inspection 31, 33, 34, 35

inspection leader 33
integration 13, 22, 24, 25, 27, 29, 36, 40, 41, 42, 45, 48, 59, 60, 69
integration testing 22, 24, 25, 29, 36, 40, 45, 59, 60, 69
interoperability testing 28
introducing a tool into an organization 57, 64
ISO 9126 11, 29, 30, 65
development model 22
iterative-incremental development model 22
keyword-driven approach 63
keyword-driven testing 62
kick-off 33
learning objective 8, 9, 10, 21, 31, 37, 45, 57, 69, 70, 71
load testing 28, 58, 60
load testing tool 58
maintainability testing 28
maintenance testing 21, 30
management tool 48, 58, 59, 63
maturity 17, 33, 38, 64
metric 33, 35, 45
mistake 10, 11, 16
modelling tool 59
moderator 33, 34, 35
monitoring tool 48, 58
non-functional requirement 21, 24, 26
non-functional testing 11, 28
objectives for testing 13
off-the-shelf 22
operational acceptance testing 27
operational test 13, 23, 30
patch 30
peer review 33, 34, 35
performance testing 28, 58
performance testing tool 58, 60
pesticide paradox 14
portability testing 28
probe effect 58
procedure 16
product risk 18, 45, 53, 54
project risk 12, 45, 53
prototyping 22
quality 8, 10, 11, 13, 19, 28, 37, 38, 47, 48, 50, 53, 55, 59
rapid application development (RAD) 22
Rational Unified Process (RUP) 22
recorder 34
regression testing 15, 16, 21, 28, 29, 30

Regulation acceptance testing 27

reliability 11, 13, 28, 50, 53, 58
reliability testing 28
requirement 13, 22, 24, 32, 34
requirements management tool 58
requirements specification 26, 28
responsibilities 24, 31, 33
re-testing. 29, See confirmation testing, See confirmation testing review13, 19, 31, 32, 33, 34, 35, 36,
47, 48, 53, 55, 58, 67, 71
review tool 58
reviewer 33, 34
risk 11, 12, 13, 14, 25, 26, 29, 30, 38, 44, 45, 49, 50, 51, 53, 54
risk-based approach 54
risk-based testing 50, 53, 54
risks 11, 25, 49, 53
risks of using tool 62
robustness testing 24
roles 8, 31, 33, 34, 35, 47, 48, 49
root cause 10, 11
scribe 33, 34
scripting language 60, 62, 63
security 27, 28, 36, 47, 50, 58
security testing 28
security tool 58, 60
simulators 24
site acceptance testing 27
software development 8, 11, 21, 22
software development model 22
special considerations for some types of tool 62
test case 38
specification-based technique 29, 39, 40
specification-based testing 37
stakeholders 12, 13, 16, 18, 26, 39, 45, 54
state transition testing 40, 41
statement coverage 42
statement testing 42
static analysis 32, 36
static analysis tool 31, 36, 58, 59, 63
static technique 31, 32
static testing 13, 32
stress testing 28, 58, 60
stress testing tool 58, 60
structural testing 24, 28, 29, 42
structure-based technique 39, 42
structure-based test design technique 42

structure-based testing 37, 42

stub. 24
success factors 35
system integration testing 22, 25
system testing 13, 22, 24, 25, 26, 27, 49, 69
technical review 31, 33, 34, 35
test analysis 15, 38, 48, 49
test approach 38, 48, 50, 51
test basis 15
test case 13, 14, 15, 16, 24, 28, 32, 37, 38, 39, 40, 41, 42, 45, 51, 55, 59, 69
test case specification 37, 38, 55
test cases 28
test closure 10, 15, 16
test condition 38
test conditions 13, 15, 16, 28, 38, 39
test control 15, 45, 51
test coverage 15, 50
test data 15, 16, 38, 48, 58, 60, 62, 63
test data preparation tool 58, 60
test design 13, 15, 22, 37, 38, 39, 43, 48, 58, 62
test design specification 45
test design technique 37, 38, 39
test design tool 58, 59
Test Development Process 38
test effort 50
test environment 15, 16, 17, 24, 26, 48, 51
test estimation 50
test execution 13, 15, 16, 32, 36, 38, 43, 45,
57, 58, 60
test execution schedule 38
test execution tool 16, 38, 57, 58, 60, 62
test harness 16, 24, 52, 58, 60
test implementation 16, 38, 49
test leader 18, 45, 47, 55
test leader tasks 47
test level 21, 22, 24, 28, 29, 30, 37, 40, 42, 44, 45, 48, 49
test log 15, 16, 43, 60
test management 45, 58
test management tool 58, 63
test manager 8, 47, 53
test monitoring 48, 51
test objective 13, 22, 28, 43, 44, 48, 51
test oracle 60
test organization 47
test plan 15, 16, 32, 45, 48, 49, 52, 53, 54

test planning 15, 16, 45, 49, 52, 54

test planning activities 49
test procedure 15, 16, 37, 38, 45, 49
test procedure specification 37, 38
test progress monitoring 51
test report 45, 51
test reporting 45, 51
test script 16, 32, 38
test strategy 47
test suite 29
test summary report 15, 16, 45, 48, 51
test tool classification 58
test type 21, 28, 30, 48, 75
test-driven development 24
tester 10, 13, 18, 34, 41, 43, 45, 47, 48, 52, 62, 67
tester tasks 48
test-first approach 24
testing and quality 11
testing principles 10, 14
testware 15, 16, 17, 48, 52
tool support 24, 32, 42, 57, 62
tool support for management of testing and tests 59
tool support for performance and monitoring 60
tool support for static testing 59
tool support for test execution and logging60
tool support for test specification 59
tool support for testing 57, 62
top-down 25
traceability 38, 48, 52
transaction processing sequences 25
types of test tool 57, 58
unit test framework 24, 58, 60
unit test framework tool 58, 60
upgrades 30
usability 11, 27, 28, 45, 47, 53
usability testing 28, 45
use case test 37, 40
use case testing 37, 40, 41
use cases 22, 26, 28, 41
user acc ept ance testing 27
validation 22
verification 22
version control 52
V-model 22
walkthrough 31, 33, 34

white-box test design technique 39, 42

white-box testing 28, 42

Standard glossary of terms used in Software Testing

Version 2.1 (dd. April 1st, 2010)
Produced by the Glossary Working Party
International Software Testing Qualifications Board

Editor : Erik van Veenendaal (The Netherlands)

Copyright Notice
This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Contributors
Rex Black (USA)
Enst Dring (Norway)
Sigrid Eldh (Sweden)
Isabel Evans (UK)
Simon Frankish (UK)
David Fuller (Australia)
Annu George (India)
Dorothy Graham (UK)
Mats Grindal (Sweden)
Matthias Hamburg (Germany)
Julian Harty (UK)
David Hayman (UK)
Bernard Homes (France)
Ian Howles (UK)
Juha Itkonen (Finland)
Paul Jorgensen (US)
Vipul Kocher (India)
Fernando Lamas de Oliveira (Portugal)
Tilo Linz (Germany)
Gustavo Marquez Sosa (Spain)
Don Mills (UK)
Peter Morgan (UK)
Thomas Mller (Switzerland)
Avi Ofer (Israel)
Dale Perry (USA)
Horst Pohlmann (Germany)
Meile Posthuma (The Netherlands)
Erkki Pyhnen (Finland)
Maaret Pyhjrvi (Finland)
Andy Redwood (UK)
Stuart Reid (UK)
Piet de Roo (The Netherlands)
Steve Sampson (UK)

Shane Saunders (UK)

Hans Schaefer (Norway)
Jurrin Seubers (The Netherlands)
Dave Sherratt (UK)
Mike Smith (UK)
Andreas Spillner (Germany)
Lucjan Stapp (Poland)
Richard Taylor (UK)
Geoff Thompson (UK)
Stephanie Ulrich (Germany)
Matti Vuori (Finland)
Gearrel Welvaart (The Netherlands)
Paul Weymouth (UK)
Pete Williams (UK)

Change History
Version 1.3 d.d. May, 31st 2007
New terms added:
- action word driven testing
- bug tracking tool
- coverage measurement tool
- modeling tool
- monkey testing
- scripted testing
- specification-based technique
- stress testing tool
- structure-based technique
- unit test framework
- white box technique

Version 2.0 d.d. December, 2nd 2007

New terms added:
- attack
- buffer
- buffer overflow
- bug taxonomy
- classification tree
- control flow analysis
- continuous representation
- cost of quality
- defect based technique
- defect based test design technique
- defect taxonomy
- error seeding tool
- Failure Mode, Effect and Criticality
Analysis (FMECA)
- false-fail result
- false-pass result
- false-negative result
- false-positive result
- fault attack
- fault seeding
- fault seeding tool
- hazard analysis
- hyperlink
- hyperlink tool
- load profile
- operational acceptance testing
- operational profile
- orthogonal array
- orthogonal array testing
- pairwise testing
- performance profiling

Terms changed:
- basic block
- control flow graph
- defect management tool
- independence of testing
- project risk
- risk-based testing
- test comparator
- test process

Terms changed:
- bebugging
- error seeding
- Failure Mode and Effect Analysis
(FMEA)
- Fault Tree Analysis (FTA)
- load testing
- horizontal traceability
- modified multiple condition testing
- process cycle test
- product risk
- root cause
- specification-based technique
- stress testing
- test charter

- pointer
- procedure testing
- process improvement
- production acceptance testing
- qualification
- reliability growth model
- retrospective meeting
- risk level
- risk type
- root cause analysis
- safety critical system
- software attack
- Software Failure Mode and Effect
Analysis (SFMEA)
- Software Failure Mode Effect and
Criticality Analysis (SFMECA)
- Software Fault Tree Analysis (SFTA)
- software lifecycle
- staged representation
- system of systems
- test design
- test estimation
- test implementation
- Test Maturity Model Integration (TMMi)
- test progress report
- test rig
- test schedule
- test session
- wild pointer

Version 2.1 d.d. April 1st, 2010

New terms added:
- accuracy testing
- acting (IDEAL)
- agile manifesto
- agile software development
- assessment report
- assessor
- balanced scorecard
- call graph
- causal analysis
- cause-effect diagram
- change management
- charter
- checklist-based testing
- clear box testing
- codependent behavior
- content-based model
- corporate dashboard
- critical success factor
- critical testing processes

Terms changed:
- attack
- buffer overflow
- control flow analysis
- cyclomatic complexity
- desk checking
- entry point
- exit point
- Failure Mode, Effects, and Criticality
Analysis (FMECA)
- invalid testing
- memory leak
- modeling tool
- monkey testing
- operational acceptance testing
- performance testing tool
- risk-based testing
- risk type
- scribe
- Software Failure Mode, Effects, and Criticality Analysis
(SFMECA)

- CTP
- dashboard
- dd-path
- Deming cycle
- diagnosing (IDEAL)
- EFQM excellence model
- emotional intelligence
- establishing (IDEAL)
- extreme programming
- fishbone diagram
- Goal Question Metric
- GQM
- hyperlink test tool
- IDEAL
- indicator
- initiating (IDEAL)
- Ishikawa diagram
- lead assessor
- learning (IDEAL)
- lifecycle model
- load testing tool
- manufacturing-based quality
- maturity level
- maturity model
- Mean Time Between Failures
- Mean Time To Repair
- mind-map
- MTBF
- MTTR
- Pareto analysis
- post-project meeting
- process assessment
- process model
- product-based quality
- project retrospective
- quality gate
- Rational Unified Process
- risk category
- RUP
- scorecard
- SCRUM
- session-based test management
- session-based testing
- Software Process Improvement
- SPI
- standard
- STEP
- structure-based test design technique
- suitability testing
- SUMI
- Systematic Test and Evaluation
Process

- static analysis
- static testing
- system of systems
- test estimation
- use case
- use case testing

- test deliverable
- test improvement plan
- Test Process Group
- test process improvement manifesto
- test process improver
- Total Quality Management
- TPG
- TQM
- transactional analysis
- transcendent-based quality
- user-based quality
- value-based quality
- WBS
- Work Breakdown Structure

Table of Contents
Foreword
1. Introduction
2. Scope
3. Arrangement
4. Normative references
5. Trademarks
6. Definitions
A
B
C
D
E
F
G
H
I
K
L
M
N
O
P
R
Q
S
T
U
V
W
Annex A (Informative)
Annex B (Method of commenting on this glossary)

Foreword
In compiling this glossary the working party has sought the views and comments of as broad a
spectrum of opinion as possible in industry, commerce and government bodies and organizations,
with the aim of producing an international testing standard which would gain acceptance in as wide a
field as possible. Total agreement will rarely, if ever, be achieved in compiling a document of this
nature. Contributions to this glossary have been received from the testing communities in Australia,
Belgium, Finland, France, Germany, India, Israel, The Netherlands, Norway, Portugal, Spain,
Sweden, Switzerland, United Kingdom, and USA.
Many (software) testers have used BS 7925-1 since its original publication in 1998. It has served
also as a major reference for the Information Systems Examination Board (ISEB) qualification at both
Foundation and Practitioner level. The standard was initially developed with a bias towards
component testing, but, since its publication, many comments and proposals for new definitions have
been submitted to both improve and expand the standard to cover a wider range of software testing.
The ISTQB testing glossary has incorporated many of these suggested updates. It is used as a
reference document for the International Software Testing Qualification Board (ISTQB) software
testing qualification scheme.

1. Introduction
Much time and effort is wasted both within and between industry, commerce, government and
professional and academic institutions when ambiguities arise as a result of the inability to
differentiate adequately between such terms as statement coverage and decision coverage; test
suite, test specification and test plan and similar terms which form an interface between various
sectors of society. Moreover, the professional or technical use of these terms is often at variance,
with different meanings attributed to them.

2. Scope
This document presents concepts, terms and definitions designed to aid communication in (software)
testing and related disciplines.

3. Arrangement
The glossary has been arranged in a single section of definitions ordered alphabetically. Some terms
are preferred to other synonymous ones, in which case, the definition of the preferred term appears,
with the synonymous ones referring to that. For example structural testing refers to white box
testing. For synonyms, the See indicator is used
See also cross-references are also used. They assist the user to quickly navigate to the right index
term. See also cross-references are constructed for relationships such as broader term to a
narrower term, and overlapping meaning between two terms.

4. Normative references
At the time of publication, the edition indicated was valid. All standards are subject to revision, and
parties to agreements based upon this Standard are encouraged to investigate the possibility of
applying the most recent edition of the standards listed below. Members of IEC and ISO maintain
registers of currently valid International Standards.
BS 7925-2:1998. Software Component Testing.
DO-178B:1992. Software Considerations in Airborne Systems and Equipment Certification,
Requirements and Technical Concepts for Aviation (RTCA SC167).
IEEE 610.12:1990. Standard Glossary of Software Engineering Terminology.
IEEE 829:1998. Standard for Software Test Documentation.

IEEE 1008:1993. Standard for Software Unit Testing.

IEEE 1012:2004 Standard for Verification and Validation Plans
IEEE 1028:1997. Standard for Software Reviews and Audits.
IEEE 1044:1993. Standard Classification for Software Anomalies.
IEEE 1219:1998. Software Maintenance.
ISO/IEC 2382-1:1993. Data processing - Vocabulary - Part 1: Fundamental terms.
ISO 9000:2005. Quality Management Systems Fundamentals and Vocabulary.
ISO/IEC 9126-1:2001. Software Engineering Software Product Quality Part 1: Quality
characteristics and sub-characteristics.
ISO/IEC 12207:1995. Information Technology Software Lifecycle Processes.
ISO/IEC 14598-1:1999. Information Technology Software Product Evaluation - Part 1:
General Overview.
ISO 15504-9: 1998. Information Technology Software Process Assessment Part 9:
Vocabulary

5. Trademarks
In this document the following trademarks are used:
CMM, CMMI and IDEAL are registered trademarks of Carnegie Mellon University
EFQM is a registered trademark of the EFQM Foundation
Rational Unified Process is a registered trademark of Rational Software Corporation
STEP is a registered trademark of Software Quality Engineering
TMap, TPA and TPI are registered trademarks of Sogeti Nederland BV
TMM is a registered service mark of Illinois Institute of Technology
TMMi is a registered trademark of the TMMi Foundation

6. Definitions
A
abstract test case: See high level test case.
acceptance: See acceptance testing.
acceptance criteria: The exit criteria that a component or system must satisfy in order to be accepted
by a user, customer, or other authorized entity. [IEEE 610]
acceptance testing: Formal testing with respect to user needs, requirements, and business processes
conducted to determine whether or not a system satisfies the acceptance criteria and to enable the
user, customers or other authorized entity to determine whether or not to accept the system. [After
IEEE 610]
accessibility testing: Testing to determine the ease by which users with disabilities can use a
component or system. [Gerrard]
accuracy: The capability of the software product to provide the right or agreed results or effects with
the needed degree of precision. [ISO 9126] See also functionality testing.
accuracy testing: The process of testing to determine the accuracy of a software product
acting (IDEAL): The phase within the IDEAL model where the improvements are developed, put
into practice, and deployed across the organization. The acting phase consists of the activities:
create solution, pilot/test solution, refine solution and implement solution. See also IDEAL.
action word driven testing: See keyword driven testing
actual outcome: See actual result.
actual result: The behavior produced/observed when a component or system is tested.
ad hoc review: See informal review.
ad hoc testing: Testing carried out informally; no formal test preparation takes place, no recognized
test design technique is used, there are no expectations for results and arbitrariness guides the test
execution activity.
adaptability: The capability of the software product to be adapted for different specified
environments without applying actions or means other than those provided for this purpose for the
software considered. [ISO 9126] See also portability.
agile manifesto: A statement on the values that underpin agile software development. The values are:
- individuals and interactions over processes and tools
- working software over comprehensive documentation
- customer collaboration over contract negotiation
- responding to change over following a plan.
agile software development: A group of software development methodologies based on iterative
incremental development, where requirements and solutions evolve through collaboration between

self-organizing cross-functional teams.

agile testing: Testing practice for a project using agile methodologies, such as extreme programming
(XP), treating development as the customer of testing and emphasizing the test-first design
paradigm. See also test driven development.
algorithm test: [TMap] See branch testing.
alpha testing: Simulated or actual operational testing by potential users/customers or an independent
test team at the developers site, but outside the development organization. Alpha testing is often
employed for off-the-shelf software as a form of internal acceptance testing.
analyzability: The capability of the software product to be diagnosed for deficiencies or causes of
failures in the software, or for the parts to be modified to be identified. [ISO 9126] See also
maintainability.
analyzer: See static analyzer.
anomaly: Any condition that deviates from expectation based on requirements specifications, design
documents, user documents, standards, etc. or from someones perception or experience.
Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or
use of software products or applicable documentation. [IEEE 1044] See also bug, defect,
deviation, error, fault, failure, incident, problem.
arc testing: See branch testing.
assessment report: A document summarizing the assessment results, e.g. conclusions,
recommendations and findings. See also process assessment.
assessor: A person who conducts an assessment; any member of an assessment team.
attack: Directed and focused attempt to evaluate the quality, especially reliability, of a test object by
attempting to force specific failures to occur. See also negative testing.
attractiveness: The capability of the software product to be attractive to the user. [ISO 9126] See
also usability.
audit: An independent evaluation of software products or processes to ascertain compliance to
standards, guidelines, specifications, and/or procedures based on objective criteria, including
documents that specify:
(1) the form or content of the products to be produced
(2) the process by which the products shall be produced
(3) how compliance to standards or guidelines shall be measured. [IEEE 1028]
audit trail: A path by which the original input to a process (e.g. data) can be traced back through the
process, taking the process output as a starting point. This facilitates defect analysis and allows a
process audit to be carried out. [After TMap]
automated testware: Testware used in automated testing, such as tool scripts.
availability: The degree to which a component or system is operational and accessible when required
for use. Often expressed as a percentage. [IEEE 610]

B
back-to-back testing: Testing in which two or more variants of a component or system are executed
with the same inputs, the outputs compared, and analyzed in cases of discrepancies. [IEEE 610]
balanced scorecard: A strategic performance management tool for measuring whether the operational
activities of a company are aligned with its objectives in terms of business vision and strategy.
See also corporate dashboard, scorecard.
baseline: A specification or software product that has been formally reviewed or agreed upon, that
thereafter serves as the basis for further development, and that can be changed only through a
formal change control process. [After IEEE 610]
basic block: A sequence of one or more consecutive executable statements containing no branches.
Note: A node in a control flow graph represents a basic block.
basis test set: A set of test cases derived from the internal structure of a component or specification
to ensure that 100% of a specified coverage criterion will be achieved.
bebugging: [Abbott] See fault seeding.
behavior: The response of a component or system to a set of input values and preconditions.
benchmark test: (1) A standard against which measurements or comparisons can be made. (2) A test
that is be used to compare components or systems to each other or to a standard as in (1). [After
IEEE 610]
bespoke software: Software developed specifically for a set of users or customers. The opposite is
off-the-shelf software.
best practice: A superior method or innovative practice that contributes to the improved performance
of an organization under given context, usually recognized as best by other peer organizations.
beta testing: Operational testing by potential and/or existing users/customers at an external site not
otherwise involved with the developers, to determine whether or not a component or system
satisfies the user/customer needs and fits within the business processes. Beta testing is often
employed as a form of external acceptance testing for off-the-shelf software in order to acquire
feedback from the market.
big-bang testing: A type of integration testing in which software elements, hardware elements, or
both are combined all at once into a component or an overall system, rather than in stages. [After
IEEE 610] See also integration testing.
black box technique: See black box test design technique.
black box test design technique: Procedure to derive and/or select test cases based on an analysis
of the specification, either functional or non-functional, of a component or system without
reference to its internal structure.
black box testing: Testing, either functional or non-functional, without reference to the internal
structure of the component or system.
blocked test case: A test case that cannot be executed because the preconditions for its execution are
not fulfilled.

bottom-up testing: An incremental approach to integration testing where the lowest level
components are tested first, and then used to facilitate the testing of higher level components. This
process is repeated until the component at the top of the hierarchy is tested. See also integration
testing.
boundary value: An input value or output value which is on the edge of an equivalence partition or at
the smallest incremental distance on either side of an edge, for example the minimum or maximum
value of a range.
boundary value analysis: A black box test design technique in which test cases are designed based
on boundary values. See also boundary value.
boundary value coverage: The percentage of boundary values that have been exercised by a test
suite.
boundary value testing: See boundary value analysis.
branch: A basic block that can be selected for execution based on a program construct in which one
of two or more alternative program paths is available, e.g. case, jump, go to, if-then-else.
branch condition: See condition.
branch condition combination coverage: See multiple condition coverage.
branch condition combination testing: See multiple condition testing.
branch condition coverage: See condition coverage.
branch coverage: The percentage of branches that have been exercised by a test suite. 100% branch
coverage implies both 100% decision coverage and 100% statement coverage.
branch testing: A white box test design technique in which test cases are designed to execute
branches.
buffer: A device or storage area used to store data temporarily for differences in rates of data flow,
time or occurrence of events, or amounts of data that can be handled by the devices or processes
involved in the transfer or use of the data. [IEEE 610]
buffer overflow: A memory access failure due to the attempt by a process to store data beyond the
boundaries of a fixed length buffer, resulting in overwriting of adjacent memory areas or the
raising of an overflow exception. See also buffer.
bug: See defect.
bug report: See defect report.
bug taxonomy: See defect taxonomy.
bug tracking tool: See defect management tool.
business process-based testing: An approach to testing in which test cases are designed based on
descriptions and/or knowledge of business processes.

C
call graph: An abstract representation of calling relationships between subroutines in a program.
Capability Maturity Model (CMM): A five level staged framework that describes the key elements
of an effective software process. The Capability Maturity Model covers best-practices for
planning, engineering and managing software development and maintenance. [CMM] See also
Capability Maturity Model Integration (CMMI).
Capability Maturity Model Integration (CMMI): A framework that describes the key elements of
an effective product development and maintenance process. The Capability Maturity Model
Integration covers best-practices for planning, engineering and managing product development and
maintenance. CMMI is the designated successor of the CMM. [CMMI] See also Capability
Maturity Model (CMM).
capture/playback tool: A type of test execution tool where inputs are recorded during manual testing
in order to generate automated test scripts that can be executed later (i.e. replayed). These tools
are often used to support automated regression testing.
capture/replay tool: See capture/playback tool.
CASE: Acronym for Computer Aided Software Engineering.
CAST: Acronym for Computer Aided Software Testing. See also test automation.
causal analysis: The analysis of defects to determine their root cause. [CMMI]
cause-effect analysis: See cause-effect graphing.
cause-effect decision table: See decision table.
cause-effect diagram: A graphical representation used to organize and display the interrelationships
of various possible root causes of a problem. Possible causes of a real or potential defect or
failure are organized in categories and subcategories in a horizontal tree-structure, with the
(potential) defect or failure as the root node. [After Juran]
cause-effect graph: A graphical representation of inputs and/or stimuli (causes) with their
associated outputs (effects), which can be used to design test cases.
cause-effect graphing: A black box test design technique in which test cases are designed from
cause-effect graphs. [BS 7925/2]
certification: The process of confirming that a component, system or person complies with its
specified requirements, e.g. by passing an exam.
change control: See configuration control.
change control board: See configuration control board.change management: (1) A structured
approach to transitioning individuals, teams, and organizations from a current state to a desired
future state. (2) Controlled way to effect a change, or a proposed change, to a product or service.
See also configuration management.
changeability: The capability of the software product to enable specified modifications to be
implemented. [ISO 9126] See also maintainability.
charter: See test charter.

checker: See reviewer.

checklist-based testing: An experience-based test design technique whereby the experienced tester
uses a high-level list of items to be noted, checked, or remembered, or a set of rules or criteria
against which a product has to be verified. See also experience-based testing.
Chows coverage metrics: See N-switch coverage. [Chow]
classification tree: A tree showing equivalence partitions hierarchically ordered, which is used to
design test cases in the classification tree method. See also classification tree method.
classification tree method: A black box test design technique in which test cases, describedby
means of a classification tree, are designed to execute combinations of representatives of input
and/or output domains. [Grochtmann]
clear-box testing: See white-box testing.
code: Computer instructions and data definitions expressed in a programming language or in a form
output by an assembler, compiler or other translator. [IEEE 610]
code analyzer: See static code analyzer.
code coverage: An analysis method that determines which parts of the software have been executed
(covered) by the test suite and which parts have not been executed, e.g. statement coverage,
decision coverage or condition coverage.
code-based testing: See white box testing.
codependent behavior: Excessive emotional or psychological dependence on another person,
specifically in trying to change that persons current (undesirable) behavior while supporting them
in continuing that behavior. For example, in software testing, complaining about late delivery to
test and yet enjoying the necessary heroism working additional hours to make up time when
delivery is running late, therefore reinforcing the lateness.
co-existence: The capability of the software product to co-exist with other independent software in a
common environment sharing common resources. [ISO 9126] See also portability.
commercial off-the-shelf software: See off-the-shelf software.
comparator: See test comparator.
compatibility testing: See interoperability testing.
compiler: A software tool that translates programs expressed in a high order language into their
machine language equivalents. [IEEE 610]
complete testing: See exhaustive testing.
completion criteria: See exit criteria.
complexity: The degree to which a component or system has a design and/or internal structure that is
difficult to understand, maintain and verify. See also cyclomatic complexity.
compliance: The capability of the software product to adhere to standards, conventions or regulations
in laws and similar prescriptions. [ISO 9126]
compliance testing: The process of testing to determine the compliance of the component or system.
component: A minimal software item that can be tested in isolation.

component integration testing: Testing performed to expose defects in the interfaces and interaction
between integrated components.
component specification: A description of a components function in terms of its output values for
specified input values under specified conditions, and required non-functional behavior (e.g.
resource-utilization).
component testing: The testing of individual software components. [After IEEE 610]
compound condition: Two or more single conditions joined by means of a logical operator (AND,
OR or XOR), e.g. A>B AND C>1000.
concrete test case: See low level test case.
concurrency testing: Testing to determine how the occurrence of two or more activities within the
same interval of time, achieved either by interleaving the activities or by simultaneous execution,
is handled by the component or system. [After IEEE 610]
condition: A logical expression that can be evaluated as True or False, e.g. A>B. See also test
condition.
condition combination coverage: See multiple condition coverage.
condition combination testing: See multiple condition testing.
condition coverage: The percentage of condition outcomes that have been exercised by a test suite.
100% condition coverage requires each single condition in every decision statement to be tested
as True and False.
condition determination coverage: The percentage of all single condition outcomes that
independently affect a decision outcome that have been exercised by a test case suite. 100%
condition determination coverage implies 100% decision condition coverage.
condition determination testing: A white box test design technique in which test cases are designed
to execute single condition outcomes that independently affect a decision outcome.
condition outcome: The evaluation of a condition to True or False.
condition testing: A white box test design technique in which test cases are designed to execute
condition outcomes.
confidence test: See smoke test.
configuration: The composition of a component or system as defined by the number, nature, and
interconnections of its constituent parts.
configuration auditing: The function to check on the contents of libraries of configuration items, e.g.
for standards compliance. [IEEE 610]
configuration control: An element of configuration management, consisting of the evaluation, coordination, approval or disapproval, and implementation of changes to configuration items after
formal establishment of their configuration identification. [IEEE 610]
configuration control board (CCB): A group of people responsible for evaluating and approving or
disapproving proposed changes to configuration items, and for ensuring implementation of
approved changes. [IEEE 610]

configuration identification: An element of configuration management, consisting of selecting the

configuration items for a system and recording their functional and physical characteristics in
technical documentation. [IEEE 610]
configuration item: An aggregation of hardware, software or both, that is designated for
configuration management and treated as a single entity in the configuration management process.
[IEEE 610]
configuration management: A discipline applying technical and administrative direction and
surveillance to: identify and document the functional and physical characteristics of a
configuration item, control changes to those characteristics, record and report change processing
and implementation status, and verify compliance with specified requirements. [IEEE 610]
configuration management tool: A tool that provides support for the identification and control of
configuration items, their status over changes and versions, and the release of baselines consisting
of configuration items.
configuration testing: See portability testing.
confirmation testing: See re-testing.
conformance testing: See compliance testing.
consistency: The degree of uniformity, standardization, and freedom from contradiction among the
documents or parts of a component or system. [IEEE 610]
content-based model: A process model providing a detailed description of good engineering
practices, e.g. test practices.
continuous representation: A capability maturity model structure wherein capability levels provide
a recommended order for approaching process improvement within specified process areas.
[CMMI]
control flow: A sequence of events (paths) in the execution through a component or system.
control flow analysis: A form of static analysis based on a representation of unique paths (sequences
of events) in the execution through a component or system. Control flow analysis evaluates the
integrity of control flow structures, looking for possible control flow anomalies such as closed
loops or logically unreachable process steps.
control flow graph: An abstract representation of all possible sequences of events (paths) in the
execution through a component or system.
control flow path: See path.
conversion testing: Testing of software used to convert data from existing systems for use in
replacement systems.
corporate dashboard: A dashboard-style representation of the status of corporate performance data.
See also balanced scorecard, dashboard.
cost of quality: The total costs incurred on quality activities and issues and often split into prevention
costs, appraisal costs, internal failure costs and external failure costs.
COTS: Acronym for Commercial Off-The-Shelf software. See off-the-shelf software.

coverage: The degree, expressed as a percentage, to which a specified coverage item has been
exercised by a test suite.
coverage analysis: Measurement of achieved coverage to a specified coverage item during test
execution referring to predetermined criteria to determine whether additional testing is required
and if so, which test cases are needed.
coverage item: An entity or property used as a basis for test coverage, e.g. equivalence partitions or
code statements.
coverage measurement tool: See coverage tool.
coverage tool: A tool that provides objective measures of what structural elements, e.g. statements,
branches have been exercised by a test suite.
critical success factor: An element which is necessary for an organization or project to achieve its
mission. They are the critical factors or activities required for ensuring the success. See also
content-based model.
Critical Testing Processes: A content-based model for test process improvement built around twelve
critical processes. These include highly visible processes, by which peers and management judge
competence and mission-critical processes in which performance affects the companys profits
and reputation.
CTP: See Critical Testing Processes.
custom software: See bespoke software.
cyclomatic complexity: The number of independent paths through a program. Cyclomatic complexity
is defined as: L N + 2P, where
- L = the number of edges/links in a graph
- N = the number of nodes in a graph
- P = the number of disconnected parts of the graph (e.g. a called graph or subroutine) [After
McCabe]
cyclomatic number: See cyclomatic complexity.

D
daily build: a development activity where a complete system is compiled and linked every day
(usually overnight), so that a consistent system is available at any time including all latest changes.
dashboard: A representation of dynamic measurements of operational performance for some
organization or activity, using metrics represented via metaphores such as visual dials,
counters, and other devices resembling those on the dashboard of an automobile, so that the
effects of events or activities can be easily understood and related to operational goals. See also
corporate dashboard, scorecard.
data definition: An executable statement where a variable is assigned a value.
data driven testing: A scripting technique that stores test input and expected results in a table or
spreadsheet, so that a single control script can execute all of the tests in the table. Data driven
testing is often used to support the application of test execution tools such as capture/playback
tools. [Fewster and Graham] See also keyword driven testing.
data flow: An abstract representation of the sequence and possible changes of the state of data
objects, where the state of an object is any of: creation, usage, or destruction. [Beizer]
data flow analysis: A form of static analysis based on the definition and usage of variables.
data flow coverage: The percentage of definition-use pairs that have been exercised by a test suite.
data flow testing: A white box test design technique in which test cases are designed to execute
definition and use pairs of variables.
data integrity testing: See database integrity testing.
database integrity testing: Testing the methods and processes used to access and manage the
data(base), to ensure access methods, processes and data rules function as expected and that
during access to the database, data is not corrupted or unexpectedly deleted, updated or created.
dd-path: A path of execution (usually through a graph representing a program, such as a flow-chart)
that does not include any conditional nodes such as the path of execution between two decisions.
dead code: See unreachable code.
debugger: See debugging tool.
debugging: The process of finding, analyzing and removing the causes of failures in software.
debugging tool: A tool used by programmers to reproduce failures, investigate the state of programs
and find the corresponding defect. Debuggers enable programmers to execute programs step by
step, to halt a program at any program statement and to set and examine program variables.
decision: A program point at which the control flow has two or more alternative routes. A node with
two or more links to separate branches.
decision condition coverage: The percentage of all condition outcomes and decision outcomes that
have been exercised by a test suite. 100% decision condition coverage implies both 100%
condition coverage and 100% decision coverage.
decision condition testing: A white box test design technique in which test cases are designed to
execute condition outcomes and decision outcomes.

decision coverage: The percentage of decision outcomes that have been exercised by a test suite.
100% decision coverage implies both 100% branch coverage and 100% statement coverage.
decision outcome: The result of a decision (which therefore determines the branches to be taken).
decision table: A table showing combinations of inputs and/or stimuli (causes) with their associated
outputs and/or actions (effects), which can be used to design test cases.
decision table testing: A black box test design technique in which test cases are designed to execute
the combinations of inputs and/or stimuli (causes) shown in a decision table. [Veenendaal04] See
also decision table.
decision testing: A white box test design technique in which test cases are designed to execute
decision outcomes.
defect: A flaw in a component or system that can cause the component or system to fail to perform its
required function, e.g. an incorrect statement or data definition. A defect, if encountered during
execution, may cause a failure of the component or system.
defect based technique: See defect based test design technique.
defect based test design technique: A procedure to derive and/or select test cases targeted at one or
more defect categories, with tests being developed from what is known about the specific defect
category. See also defect taxonomy.
defect density: The number of defects identified in a component or system divided by the size of the
component or system (expressed in standard measurement terms, e.g. lines-of-code, number of
classes or function points).
Defect Detection Percentage (DDP): The number of defects found by a test phase, divided by the
number found by that test phase and any other means afterwards.
defect management: The process of recognizing, investigating, taking action and disposing of
defects. It involves recording defects, classifying them and identifying the impact. [After IEEE
1044]
defect management tool: A tool that facilitates the recording and status tracking of defects and
changes. They often have workflow-oriented facilities to track and control the allocation,
correction and re-testing of defects and provide reporting facilities. See also incident
management tool.
defect masking: An occurrence in which one defect prevents the detection of another. [After IEEE
610]
defect report: A document reporting on any flaw in a component or system that can cause the
component or system to fail to perform its required function. [After IEEE 829]
defect taxonomy: A system of (hierarchical) categories designed to be a useful aid for reproducibly
classifying defects.
defect tracking tool: See defect management tool.
definition-use pair: The association of the definition of a variable with the use of that variable.
Variable uses include computational (e.g. multiplication) or to direct the execution of a path
(predicate use).

deliverable: Any (work) product that must be delivered to someone other than the (work) products
author.
Deming cycle: An iterative four-step problem-solving process, (plan-do-check-act), typically used in
process improvement. [After Deming]
design-based testing: An approach to testing in which test cases are designed based on the
architecture and/or detailed design of a component or system (e.g. tests of interfaces between
components or systems).
desk checking: Testing of software or a specification by manual simulation of its execution. See also
static testing.
development testing: Formal or informal testing conducted during the implementation of a
component or system, usually in the development environment by developers. [After IEEE 610]
deviation: See incident.
deviation report: See incident report.
diagnosing (IDEAL): The phase within the IDEAL model where it is determined where one is,
relative to where one wants to be. The diagnosing phase consists of the activities: characterize
current and desired states and develop recommendations. See also IDEAL.
dirty testing: See negative testing.
documentation testing: Testing the quality of the documentation, e.g. user guide or installation guide.
domain: The set from which valid input and/or output values can be selected.
driver: A software component or test tool that replaces a component that takes care of the control
and/or the calling of a component or system. [After TMap]
dynamic analysis: The process of evaluating behavior, e.g. memory performance, CPU usage, of a
system or component during execution. [After IEEE 610]
dynamic analysis tool: A tool that provides run-time information on the state of the software code.
These tools are most commonly used to identify unassigned pointers, check pointer arithmetic and
to monitor the allocation, use and de-allocation of memory and to flag memory leaks.
dynamic comparison: Comparison of actual and expected results, performed while the software is
being executed, for example by a test execution tool.
dynamic testing: Testing that involves the execution of the software of a component or system.

E
efficiency:The capability of the software product to provide appropriate performance, relative to the
amount of resources used under stated conditions. [ISO 9126]
efficiency testing:The process of testing to determine the efficiency of a software product.
EFQM (European Foundation for Quality Management) excellence model:A non-prescriptive
framework for an organisations quality management system, defined and owned by the European
Foundation for Quality Management, based on five Enabling criteria (covering what an
organisation does), and four Results criteria (covering what an organisation achieves).
elementary comparison testing:A black box test design technique in which test cases are designed
to execute combinations of inputs using the concept of condition determination coverage. [TMap]
emotional intelligence:The ability, capacity, and skill to identify, assess, and manage the emotions of
ones self, of others, and of groups.
emulator:A device, computer program, or system that accepts the same inputs and produces the same
outputs as a given system. [IEEE 610] See also simulator.
entry criteria:The set of generic and specific conditions for permitting a process to go forward with
a defined task, e.g. test phase. The purpose of entry criteria is to prevent a task from starting which
would entail more (wasted) effort compared to the effort needed to remove the failed entry
criteria. [Gilb and Graham]
entry point:An executable statement or process step which defines a point at which a given process
is intended to begin..
equivalence class:See equivalence partition.
equivalence partition:A portion of an input or output domain for which the behavior of a component
or system is assumed to be the same, based on the specification.
equivalence partition coverage:The percentage of equivalence partitions that have been exercised
by a test suite.
equivalence partitioning:A black box test design technique in which test cases are designed to
execute representatives from equivalence partitions. In principle test cases are designed to cover
each partition at least once.
error:A human action that produces an incorrect result. [After IEEE 610]
error guessing:A test design technique where the experience of the tester is used to anticipate what
defects might be present in the component or system under test as a result of errors made, and to
design tests specifically to expose them.
error seeding:See fault seeding.
error seeding tool:See fault seeding tool.
error tolerance:The ability of a system or component to continue normal operation despite the
presence of erroneous inputs. [After IEEE 610].
establishing (IDEAL):The phase within the IDEAL model where the specifics of how an
organization will reach its destination are planned. The establishing phase consists of the

activities: set priorities, develop approach and plan actions. See also IDEAL.
evaluation:See testing.
exception handling:Behavior of a component or system in response to erroneous input, from either a
human user or from another component or system, or to an internal failure.
executable statement:A statement which, when compiled, is translated into object code, and which
will be executed procedurally when the program is running and may perform an action on data.
exercised:A program element is said to be exercised by a test case when the input value causes the
execution of that element, such as a statement, decision, or other structural element.
exhaustive testing:A test approach in which the test suite comprises all combinations of input values
and preconditions.
exit criteria:The set of generic and specific conditions, agreed upon with the stakeholders, for
permitting a process to be officially completed. The purpose of exit criteria is to prevent a task
from being considered completed when there are still outstanding parts of the task which have not
been finished. Exit criteria are used to report against and to plan when to stop testing. [After Gilb
and Graham]
exit point:An executable statement or process step which defines a point at which a given process is
intended to cease..
expected outcome:See expected result.
expected result:The behavior predicted by the specification, or another source, of the component or
system under specified conditions.
experience-based technique:See experience-based test design technique.
experience-based test design technique:Procedure to derive and/or select test cases based on the
testers experience, knowledge and intuition.
exploratory testing:An informal test design technique where the tester actively controls the design of
the tests as those tests are performed and uses information gained while testing to design new and
better tests. [After Bach]
extreme programming:A software engineering methodology used within agile software development
whereby core practices are programming in pairs, doing extensive code review, unit testing of all
code, and simplicity and clarity in code. See also agile software development.

F
fail:A test is deemed to fail if its actual result does not match its expected result.
failure:Deviation of the component or system from its expected delivery, service or result. [After
Fenton]
failure mode:The physical or functional manifestation of a failure. For example, a system in failure
mode may be characterized by slow operation, incorrect outputs, or complete termination of
execution. [IEEE 610]
Failure Mode and Effect Analysis (FMEA):A systematic approach to risk identification and
analysis of identifying possible modes of failure and attempting to prevent their occurrence. See
also Failure Mode, Effect and Criticality Analysis (FMECA).
Failure Mode, Effects, and Criticality Analysis (FMECA):An extension of FMEA, as in addition to
the basic FMEA, it includes a criticality analysis, which is used to chart the probability of failure
modes against the severity of their consequences. The result highlights failure modes with
relatively high probability and severity of consequences, allowing remedial effort to be directed
where it will produce the greatest value. See also Failure Mode and Effect Analysis (FMEA).
failure rate:The ratio of the number of failures of a given category to a given unit of measure, e.g.
failures per unit of time, failures per number of transactions, failures per number of computer runs.
[IEEE 610]
false-fail result: A test result in which a defect is reported although no such defect actually exists in
the test object.
false-pass result: A test result which fails to identify the presence of a defect that is actually present
in the test object.
false-positive result: See false-fail result.
false-negative result: See false-pass result.
fault:See defect.
fault attack:See attack.
fault density:See defect density.
Fault Detection Percentage (FDP):See Defect Detection Percentage (DDP).
fault masking:See defect masking.
fault seeding:The process of intentionally adding known defects to those already in the component or
system for the purpose of monitoring the rate of detection and removal, and estimating the number
of remaining defects. [IEEE 610]
fault seeding tool:A tool for seeding (i.e. intentionally inserting) faults in a component or system.
fault tolerance:The capability of the software product to maintain a specified level of performance
in cases of software faults (defects) or of infringement of its specified interface. [ISO 9126] See
also reliability, robustness.
Fault Tree Analysis (FTA):A technique used to analyze the causes of faults (defects). The technique
visually models how logical relationships between failures, human errors, and external events can

combine to cause specific faults to disclose.

feasible path:A path for which a set of input values and preconditions exists which causes it to be
executed.
feature:An attribute of a component or system specified or implied by requirements documentation
(for example reliability, usability or design constraints). [After IEEE 1008]
field testing:See beta testing.
finite state machine:A computational model consisting of a finite number of states and transitions
between those states, possibly with accompanying actions. [IEEE 610]
finite state testing:See state transition testing.
fishbone diagram:See cause-effect diagram.
formal review:A review characterized by documented procedures and requirements, e.g. inspection.
frozen test basis:A test basis document that can only be amended by a formal change control
process. See also baseline.
Function Point Analysis (FPA):Method aiming to measure the size of the functionality of an
information system. The measurement is independent of the technology. This measurement may be
used as a basis for the measurement of productivity, the estimation of the needed resources, and
project control.
functional integration:An integration approach that combines the components or systems for the
purpose of getting a basic functionality working early. See also integration testing.
functional requirement:A requirement that specifies a function that a component or system must
perform. [IEEE 610]
functional test design technique:Procedure to derive and/or select test cases based on an analysis of
the specification of the functionality of a component or system without reference to its internal
structure. See also black box test design technique.
functional testing:Testing based on an analysis of the specification of the functionality of a
component or system. See also black box testing.
functionality:The capability of the software product to provide functions which meet stated and
implied needs when the software is used under specified conditions. [ISO 9126]
functionality testing:The process of testing to determine the functionality of a software product.

G
glass box testing:See white box testing.
Goal Question Metric:An approach to software measurement using a three-level model: conceptual
level (goal), operational level (question) and quantitative level (metric).
GQM:See Goal Question Metric.

H
hazard analysis: A technique used to characterize the elements of risk. The result of a hazard analysis
will drive the methods used for development and testing of a system. See also risk analysis.
heuristic evaluation:A static usability test technique to determine the compliance of a user interface
with recognized usability principles (the so-called heuristics).
high level test case:A test case without concrete (implementation level) values for input data and
expected results. Logical operators are used; instances of the actual values are not yet defined
and/or available. See also low level test case.
horizontal traceability:The tracing of requirements for a test level through the layers of test
documentation (e.g. test plan, test design specification, test case specification and test procedure
specification or test script).
hyperlink: A pointer within a web page that leads to other web pages.
hyperlink test tool: A tool used to check that no broken hyperlinks are present on a web site.

I
IDEAL:An organizational improvement model that serves as a roadmap for initiating, planning, and
implementing improvement actions. The IDEAL model is named for the five phases it describes:
initiating, diagnosing, establishing, acting, and learning impact analysis:The assessment of change
to the layers of development documentation, test documentation and components, in order to
implement a given change to specified requirements.
incident:Any event occurring that requires investigation. [After IEEE 1008]
incident logging:Recording the details of any incident that occurred, e.g. during testing.
incident management:The process of recognizing, investigating, taking action and disposing of
incidents. It involves logging incidents, classifying them and identifying the impact. [After IEEE
1044]
incident management tool:A tool that facilitates the recording and status tracking of incidents. They
often have workflow-oriented facilities to track and control the allocation, correction and retesting of incidents and provide reporting facilities. See also defect management tool.
incident report:A document reporting on any event that occurred, e.g. during the testing, which
requires investigation. [After IEEE 829]
incremental development model:A development lifecycle where a project is broken into a series of
increments, each of which delivers a portion of the functionality in the overall project
requirements. The requirements are prioritized and delivered in priority order in the appropriate
increment. In some (but not all) versions of this lifecycle model, each subproject follows a mini
V-model with its own design, coding and testing phases.
incremental testing:Testing where components or systems are integrated and tested one or some at a
time, until all the components or systems are integrated and tested.
independence of testing:Separation of responsibilities, which encourages the accomplishment of
objective testing. [After DO-178b]
indicator:A measure that can be used to estimate or predict another measure. [ISO 14598]
infeasible path:A path that cannot be exercised by any set of possible input values.
informal review:A review not based on a formal (documented) procedure. initiating (IDEAL):The
phase within the IDEAL model where the groundwork is laid for a successful improvement effort.
The initiating phase consists of the activities: set context, build sponsorship and charter
infrastructure. See also IDEAL.
input:A variable (whether stored within a component or outside) that is read by a component.
input domain:The set from which valid input values can be selected. See also domain.
input value:An instance of an input. See also input.
inspection:A type of peer review that relies on visual examination of documents to detect defects,
e.g. violations of development standards and non-conformance to higher level documentation. The
most formal review technique and therefore always based on a documented procedure. [After
IEEE 610, IEEE 1028] See also peer review.
inspection leader:See moderator.

inspector:See reviewer.
installability:The capability of the software product to be installed in a specified environment [ISO
9126]. See also portability.
installability testing:The process of testing the installability of a software product. See also
portability testing.
installation guide:Supplied instructions on any suitable media, which guides the installer through the
installation process. This may be a manual guide, step-by-step procedure, installation wizard, or
any other similar process description.
installation wizard:Supplied software on any suitable media, which leads the installer through the
installation process. It normally runs the installation process, provides feedback on installation
results, and prompts for options.
instrumentation:The insertion of additional code into the program in order to collect information
about program behavior during execution, e.g. for measuring code coverage.
instrumenter:A software tool used to carry out instrumentation.
intake test:A special instance of a smoke test to decide if the component or system is ready for
detailed and further testing. An intake test is typically carried out at the start of the test execution
phase. See also smoke test.
integration:The process of combining components or systems into larger assemblies.
integration testing:Testing performed to expose defects in the interfaces and in the interactions
between integrated components or systems. See also component integration testing, system
integration testing.
integration testing in the large:See system integration testing.
integration testing in the small:See component integration testing.
interface testing:An integration test type that is concerned with testing the interfaces between
components or systems.
interoperability:The capability of the software product to interact with one or more specified
components or systems. [After ISO 9126] See also functionality.
interoperability testing:The process of testing to determine the interoperability of a software
product. See also functionality testing.
invalid testing:Testing using input values that should be rejected by the component or system. See
also error tolerance, negative testing.
Ishikawa diagram:See cause-effect diagram.
isolation testing:Testing of individual components in isolation from surrounding components, with
surrounding components being simulated by stubs and drivers, if needed.
item transmittal report:See release note.
iterative development model:A development lifecycle where a project is broken into a usually large
number of iterations. An iteration is a complete development loop resulting in a release (internal
or external) of an executable product, a subset of the final product under development, which
grows from iteration to iteration to become the final product.

K
key performance indicator:See performance indicator.
keyword driven testing:A scripting technique that uses data files to contain not only test data and
expected results, but also keywords related to the application being tested. The keywords are
interpreted by special supporting scripts that are called by the control script for the test. See also
data driven testing.

L
LCSAJ:A Linear Code Sequence And Jump, consists of the following three items (conventionally
identified by line numbers in a source code listing): the start of the linear sequence of executable
statements, the end of the linear sequence, and the target line to which control flow is transferred
at the end of the linear sequence.
LCSAJ coverage:The percentage of LCSAJs of a component that have been exercised by a test suite.
100% LCSAJ coverage implies 100% decision coverage.
LCSAJ testing:A white box test design technique in which test cases are designed to execute
LCSAJs.
lead assessor:The person who leads an assessment. In some cases, for instance CMMi and TMMi
when formal assessments are conducted, the lead-assessor must be accredited and formally
trained.
learnability:The capability of the software product to enable the user to learn its application. [ISO
9126] See also usability.
learning (IDEAL):The phase within the IDEAL model where one learns from experiences and
improves ones ability to adopt new processes and technologies in the future. The learning phase
consists of the activities: analyze and validate, and propose future actions. See also IDEAL.
level test plan: A test plan that typically addresses one test level. See also test plan.
lifecycle model:A partitioning of the life of a product or project into phases. [CMMI] See also
software lifecycle.
link testing:See component integration testing.
load profile:A specification of the activity which a component or system being tested may experience
in production. A load profile consists of a designated number of virtual users who process a
defined set of transactions in a specified time period and according to a predefined operational
profile. See also operational profile.
load testing:A type of performance testing conducted to evaluate the behavior of a component or
system with increasing load, e.g. numbers of parallel users and/or numbers of transactions, to
determine what load can be handled by the component or system. See also performance testing,
stress testing.
load testing tool:See performance testing tool.
logic-coverage testing:See white box testing. [Myers]
logic-driven testing:See white box testing.
logical test case:See high level test case.
low level test case:A test case with concrete (implementation level) values for input data and
expected results. Logical operators from high level test cases are replaced by actual values that
correspond to the objectives of the logical operators. See also high level test case.

M
maintainability:The ease with which a software product can be modified to correct defects, modified
to meet new requirements, modified to make future maintenance easier, or adapted to a changed
environment. [ISO 9126]
maintainability testing:The process of testing to determine the maintainability of a software product.
maintenance:Modification of a software product after delivery to correct defects, to improve
performance or other attributes, or to adapt the product to a modified environment. [IEEE 1219]
maintenance testing:Testing the changes to an operational system or the impact of a changed
environment to an operational system.
management review:A systematic evaluation of software acquisition, supply, development,
operation, or maintenance process, performed by or on behalf of management that monitors
progress, determines the status of plans and schedules, confirms requirements and their system
allocation, or evaluates the effectiveness of management approaches to achieve fitness for
purpose. [After IEEE 610, IEEE 1028]
manufacturing-based quality:A view of quality, whereby quality is measured by the degree to which
a product or service conforms to its intended design and requirements. Quality arises from the
process(es) used. [After Garvin] See also product-based quality, transcendent-based quality,
user-based quality, value-based quality.
master test plan:A test plan that typically addresses multiple test levels. See also test plan.
maturity:(1) The capability of an organization with respect to the effectiveness and efficiency of its
processes and work practices. See also Capability Maturity Model, Test Maturity Model. (2)
The capability of the software product to avoid failure as a result of defects in the software. [ISO
9126] See also reliability.
maturity level:Degree of process improvement across a predefined set of process areas in which all
goals in the set are attained. [TMMi]
maturity model:A structured collection of elements that describe certain aspects of maturity in an
organization, and aid in the definition and understanding of an organizations processes. A maturity
model often provides a common language, shared vision and framework for prioritizing
improvement actions.
Mean Time Between Failures: The arithmetic mean (average) time between failures of a system.
The MTBF is typically part of a reliability growth model that assumes the failed system is
immediately repaired, as a part of a defect fixing process. See also reliability growth model.
Mean Time To Repair:The arithmetic mean (average) time a system will take to recover from any
failure. This typically includes testing to insure that the defect has been resolved.
measure:The number or category assigned to an attribute of an entity by making a measurement. [ISO
14598]
measurement:The process of assigning a number or category to an entity to describe an attribute of
that entity. [ISO 14598]

measurement scale:A scale that constrains the type of data analysis that can be performed on it. [ISO
14598]
memory leak:A memory access failure due to a defect in a programs dynamic store allocation logic
that causes it to fail to release memory after it has finished using it, eventually causing the program
and/or other concurrent processes to fail due to lack of memory.
metric:A measurement scale and the method used for measurement. [ISO 14598]
migration testing:See conversion testing.
milestone:A point in time in a project at which defined (intermediate) deliverables and results should
be ready.
mind-map:A diagram used to represent words, ideas, tasks, or other items linked to and arranged
around a central key word or idea. Mind maps are used to generate, visualize, structure, and
classify ideas, and as an aid in study, organization, problem solving, decision making, and writing.
mistake:See error.
modeling tool:A tool that supports the creation, amendment and verification of models of the
software or system [Graham].
moderator:The leader and main person responsible for an inspection or other review process.
modified condition decision coverage:See condition determination coverage.
modified condition decision testing:See condition determination testing.
modified multiple condition coverage:See condition determination coverage.
modified multiple condition testing:See condition determination testing.
module:See component.
module testing:See component testing.
monitor:A software tool or hardware device that runs concurrently with the component or system
under test and supervises, records and/or analyses the behavior of the component or system. [After
IEEE 610]
monitoring tool:See monitor.
monkey testing:Testing by means of a random selection from a large range of inputs and by randomly
pushing buttons, ignorant of how the product is being used.
MTBF:See Mean Time Between Failures.
MTTR:See Mean Time To Repair.
multiple condition:See compound condition.
multiple condition coverage:The percentage of combinations of all single condition outcomes within
one statement that have been exercised by a test suite. 100% multiple condition coverage implies
100% condition determination coverage.
multiple condition testing:A white box test design technique in which test cases are designed to
execute combinations of single condition outcomes (within one statement).
mutation analysis:A method to determine test suite thoroughness by measuring the extent to which a
test suite can discriminate the program from slight variants (mutants) of the program.

mutation testing:See back-to-back testing.

N
N-switch coverage:The percentage of sequences of N+1 transitions that have been exercised by a
test suite. [Chow]
N-switch testing:A form of state transition testing in which test cases are designed to execute all
valid sequences of N+1 transitions. [Chow] See also state transition testing.
negative testing:Tests aimed at showing that a component or system does not work. Negative testing
is related to the testers attitude rather than a specific test approach or test design technique, e.g.
testing with invalid input values or exceptions. [After Beizer].
non-conformity:Non fulfillment of a specified requirement. [ISO 9000]
non-functional requirement:A requirement that does not relate to functionality, but to attributes such
as reliability, efficiency, usability, maintainability and portability.
non-functional test design technique:Procedure to derive and/or select test cases for nonfunctional
testing based on an analysis of the specification of a component or system without reference to its
internal structure. See also black box test design technique.
non-functional testing:Testing the attributes of a component or system that do not relate to
functionality, e.g. reliability, efficiency, usability, maintainability and portability.

O
off-the-shelf software:A software product that is developed for the general market, i.e. for a large
number of customers, and that is delivered to many customers in identical format.
operability:The capability of the software product to enable the user to operate and control it. [ISO
9126] See also usability.
operational acceptance testing:Operational testing in the acceptance test phase, typically performed
in a (simulated) operational environment by operations and/or systems administration staff
focusing on operational aspects, e.g. recoverability, resource-behavior, installability and technical
compliance. See also operational testing.
operational environment:Hardware and software products installed at users or customers sites
where the component or system under test will be used. The software may include operating
systems, database management systems, and other applications.
operational profile: The representation of a distinct set of tasks performed by the component or
system, possibly based on user behavior when interacting with the component or system, and their
probabilities of occurence. A task is logical rather that physical and can be executed over several
machines or be executed in non-contiguous time segments.
operational profile testing:Statistical testing using a model of system operations (short duration
tasks) and their probability of typical use. [Musa]
operational testing:Testing conducted to evaluate a component or system in its operational
environment. [IEEE 610]
oracle:See test oracle.
orthogonal array: A 2-dimensional array constructed with special mathematical properties, such that
choosing any two columns in the array provides every pair combination of each number in the
array.
orthogonal array testing:A systematic way of testing all-pair combinations of variables using
orthogonal arrays. It significantly reduces the number of all combinations of variables to test all
pair combinations. See also pairwise testing.
outcome:See result.
output:A variable (whether stored within a component or outside) that is written by a component.
output domain:The set from which valid output values can be selected. See also domain.
output value:An instance of an output. See also output.

P
pair programming:A software development approach whereby lines of code (production and/or test)
of a component are written by two programmers sitting at a single computer. This implicitly means
ongoing real-time code reviews are performed.
pair testing:Two persons, e.g. two testers, a developer and a tester, or an end-user and a tester,
working together to find defects. Typically, they share one computer and trade control of it while
testing.
pairwise testing:A black box test design technique in which test cases are designed to execute all
possible discrete combinations of each pair of input parameters. See also orthogonal array
testing.
Pareto analysis:A statistical technique in decision making that is used for selection of a limited
number of factors that produce significant overall effect. In terms of quality improvement, a large
majority of problems (80%) are produced by a few key causes (20%).
partition testing:See equivalence partitioning. [Beizer]
pass:A test is deemed to pass if its actual result matches its expected result.
pass/fail criteria:Decision rules used to determine whether a test item (function) or feature has
passed or failed a test. [IEEE 829]
path:A sequence of events, e.g. executable statements, of a component or system from an entry point
to an exit point.
path coverage:The percentage of paths that have been exercised by a test suite. 100% path coverage
implies 100% LCSAJ coverage.
path sensitizing:Choosing a set of input values to force the execution of a given path.
path testing:A white box test design technique in which test cases are designed to execute paths.
peer review:A review of a software work product by colleagues of the producer of the product for
the purpose of identifying defects and improvements. Examples are inspection, technical review
and walkthrough.
performance:The degree to which a system or component accomplishes its designated functions
within given constraints regarding processing time and throughput rate. [After IEEE 610] See also
efficiency.
performance indicator:A high level metric of effectiveness and/or efficiency used to guide and
control progressive development, e.g. lead-time slip for software development. [CMMI]
performance profiling:Definition of user profiles in performance, load and/or stress testing. Profiles
should reflect anticipated or actual usage based on an operational profile of a component or
system, and hence the expected workload. See also load profile, operational profile.
performance testing:The process of testing to determine the performance of a software product. See
also efficiency testing.
performance testing tool:A tool to support performance testing that usually has two main facilities:
load generation and test transaction measurement. Load generation can simulate either multiple
users or high volumes of input data. During execution, response time measurements are taken from

selected transactions and these are logged. Performance testing tools normally provide reports
based on test logs and graphs of load against response times.
phase test plan:A test plan that typically addresses one test phase. See also test plan.
pointer:A data item that specifies the location of another data item; for example, a data item that
specifies the address of the next employee record to be processed. [IEEE 610]
portability:The ease with which the software product can be transferred from one hardware or
software environment to another. [ISO 9126]
portability testing:The process of testing to determine the portability of a software product.
postcondition:Environmental and state conditions that must be fulfilled after the execution of a test or
test procedure.
post-execution comparison:Comparison of actual and expected results, performed after the software
has finished running.
post-project meeting: See retrospective meeting.
precondition:Environmental and state conditions that must be fulfilled before the component or
system can be executed with a particular test or test procedure.
predicted outcome:See expected result.
pretest:See intake test.
priority:The level of (business) importance assigned to an item, e.g. defect.
probe effect:The effect on the component or system by the measurement instrument when the
component or system is being measured, e.g. by a performance testing tool or monitor. For
example performance may be slightly worse when performance testing tools are being used.
problem:See defect.
problem management:See defect management.
problem report:See defect report.
procedure testing:Testing aimed at ensuring that the component or system can operate in conjunction
with new or existing users business procedures or operational procedures.
process:A set of interrelated activities, which transform inputs into outputs. [ISO 12207]
process assessment:A disciplined evaluation of an organizations software processes against a
reference model. [after ISO 15504]
process cycle test:A black box test design technique in which test cases are designed to execute
business procedures and processes. [TMap] See also procedure testing.
process improvement:A program of activities designed to improve the performance and maturity of
the organizations processes, and the result of such a program. [CMMI]
process model:A framework wherein processes of the same nature are classified into a overall
model, e.g. a test improvement model.
product-based quality:A view of quality, wherein quality is based on a well-defined set of quality
attributes. These attributes must be measured in an objective and quantitative way. Differences in
the quality of products of the same type can be traced back to the way the specific quality

attributes have been implemented. [After Garvin] See also manufacturing-based quality, quality
attribute, transcendent-based quality, user-based quality, value-based quality.
product risk:A risk directly related to the test object. See also risk.
production acceptance testing:See operational acceptance testing.
program instrumenter:See instrumenter.
program testing:See component testing.
project:A project is a unique set of coordinated and controlled activities with start and finish dates
undertaken to achieve an objective conforming to specific requirements, including the constraints
of time, cost and resources. [ISO 9000]
project retrospective:A structured way to capture lessons learned and to create specific action plans
for improving on the next project or next project phase.
project risk:A risk related to management and control of the (test) project, e.g. lack of staffing, strict
deadlines, changing requirements, etc. See also risk.
project test plan:See master test plan.
pseudo-random:A series which appears to be random but is in fact generated according to some
prearranged sequence.

Q
qualification:The process of demonstrating the ability to fulfill specified requirements. Note the term
qualified is used to designate the corresponding status. [ISO 9000]
quality:The degree to which a component, system or process meets specified requirements and/or
user/customer needs and expectations. [After IEEE 610]
quality assurance:Part of quality management focused on providing confidence that quality
requirements will be fulfilled. [ISO 9000]
quality attribute:A feature or characteristic that affects an items quality. [IEEE 610]
quality characteristic:See quality attribute.
quality gate:A special milestone in a project. Quality gates are located between those phases of a
project strongly depending on the outcome of a previous phase. A quality gate includes a formal
check of the documents of the previous phase.
quality management:Coordinated activities to direct and control an organization with regard to
quality. Direction and control with regard to quality generally includes the establishment of the
quality policy and quality objectives, quality planning, quality control, quality assurance and
quality improvement. [ISO 9000]

R
random testing:A black box test design technique where test cases are selected, possibly using a
pseudo-random generation algorithm, to match an operational profile. This technique can be used
for testing non-functional attributes such as reliability and performance.
Rational Unified Process:A proprietary adaptable iterative software development process
framework consisting of four project lifecycle phases: inception, elaboration, construction and
transition.
recorder:See scribe.
record/playback tool:See capture/playback tool.
recoverability:The capability of the software product to re-establish a specified level of
performance and recover the data directly affected in case of failure. [ISO 9126] See also
reliability.
recoverability testing:The process of testing to determine the recoverability of a software product.
See also reliability testing.
recovery testing:See recoverability testing.
regression testing:Testing of a previously tested program following modification to ensure that
defects have not been introduced or uncovered in unchanged areas of the software, as a result of
the changes made. It is performed when the software or its environment is changed.
regulation testing:See compliance testing.
release note:A document identifying test items, their configuration, current status and other delivery
information delivered by development to testing, and possibly other stakeholders, at the start of a
test execution phase. [After IEEE 829]
reliability:The ability of the software product to perform its required functions under stated
conditions for a specified period of time, or for a specified number of operations. [ISO 9126]
reliability growth model:A model that shows the growth in reliability over time during continuous
testing of a component or system as a result of the removal of defects that result in reliability
failures.
reliability testing:The process of testing to determine the reliability of a software product.
replaceability:The capability of the software product to be used in place of another specified
software product for the same purpose in the same environment. [ISO 9126] See also portability.
requirement:A condition or capability needed by a user to solve a problem or achieve an objective
that must be met or possessed by a system or system component to satisfy a contract, standard,
specification, or other formally imposed document. [After IEEE 610]
requirements-based testing:An approach to testing in which test cases are designed based on test
objectives and test conditions derived from requirements, e.g. tests that exercise specific functions
or probe non-functional attributes such as reliability or usability.
requirements management tool:A tool that supports the recording of requirements, requirements
attributes (e.g. priority, knowledge responsible) and annotation, and facilitates traceability through
layers of requirements and requirements change management. Some requirements management

tools also provide facilities for static analysis, such as consistency checking and violations to predefined requirements rules.
requirements phase:The period of time in the software lifecycle during which the requirements for a
software product are defined and documented. [IEEE 610]
resource utilization:The capability of the software product to use appropriate amounts and types of
resources, for example the amounts of main and secondary memory used by the program and the
sizes of required temporary or overflow files, when the software performs its function under
stated conditions. [After ISO 9126] See also efficiency.
resource utilization testing:The process of testing to determine the resource-utilization of a software
product. See also efficiency testing.
result:The consequence/outcome of the execution of a test. It includes outputs to screens, changes to
data, reports, and communication messages sent out. See also actual result, expected result.
resumption criteria:The testing activities that must be repeated when testing is re-started after a
suspension. [After IEEE 829]
re-testing:Testing that runs test cases that failed the last time they were run, in order to verify the
success of corrective actions.
retrospective meeting:A meeting at the end of a project during which the project team members
evaluate the project and learn lessons that can be applied to the next project.
review:An evaluation of a product or project status to ascertain discrepancies from planned results
and to recommend improvements. Examples include management review, informal review,
technical review, inspection, and walkthrough. [After IEEE 1028]
review tool:A tool that provides support to the review process. Typical features include review
planning and tracking support, communication support, collaborative reviews and a repository for
collecting and reporting of metrics.
reviewer:The person involved in the review that identifies and describes anomalies in the product or
project under review. Reviewers can be chosen to represent different viewpoints and roles in the
review process.
risk:A factor that could result in future negative consequences; usually expressed as impact and
likelihood.
risk analysis:The process of assessing identified risks to estimate their impact and probability of
occurrence (likelihood).
risk-based testing:An approach to testing to reduce the level of product risks and inform
stakeholders of their status, starting in the initial stages of a project. It involves the identification
of product risks and the use of risk levels to guide the test process.
risk category: See risk type.
risk control:The process through which decisions are reached and protective measures are
implemented for reducing risks to, or maintaining risks within, specified levels.
risk identification:The process of identifying risks using techniques such as brainstorming, checklists
and failure history.

risk level:The importance of a risk as defined by its characteristics impact and likelihood. The level
of risk can be used to determine the intensity of testing to be performed. A risk level can be
expressed either qualitatively (e.g. high, medium, low) or quantitatively.
risk management:Systematic application of procedures and practices to the tasks of identifying,
analyzing, prioritizing, and controlling risk.
risk mitigation:See risk control.
risk type:A set of risks grouped by one or more common factors such as a quality attribute, cause,
location, or potential effect of risk;. A specific set of product risk types is related to the type of
testing that can mitigate (control) that risk type. For example the risk of user-interactions being
misunderstood can be mitigated by usability testing.
robustness:The degree to which a component or system can function correctly in the presence of
invalid inputs or stressful environmental conditions. [IEEE 610] See also error-tolerance, faulttolerance.
robustness testing:Testing to determine the robustness of the software product.
root cause:A source of a defect such that if it is removed, the occurence of the defect type is
decreased or removed. [CMMI]
root cause analysis:An analysis technique aimed at identifying the root causes of defects. By
directing corrective measures at root causes, it is hoped that the likelihood of defect recurrence
will be minimized.
RUP:See Rational Unified Process.

S
safety:The capability of the software product to achieve acceptable levels of risk of harm to people,
business, software, property or the environment in a specified context of use. [ISO 9126]
safety critical system:A system whose failure or malfunction may result in death or serious injury to
people, or loss or severe damage to equipment, or environmental harm.
safety testing:Testing to determine the safety of a software product.
sanity test:See smoke test.
scalability:The capability of the software product to be upgraded to accommodate increased loads.
[After Gerrard]
scalability testing:Testing to determine the scalability of the software product.
scenario testing:See use case testing.
scorecard:A representation of summarized performance measurements representing progress towards
the implementation of long-term goals. A scorecard provides static measurements of performance
over or at the end of a defined interval. See also balanced scorecard, dashboard.
scribe:The person who records each defect mentioned and any suggestions for process improvement
during a review meeting, on a logging form. The scribe should ensure that the logging form is
readable and understandable.
scripted testing:Test execution carried out by following a previously documented sequence of tests.
scripting language:A programming language in which executable test scripts are written, used by a
test execution tool (e.g. a capture/playback tool).
SCRUM:An iterative incremental framework for managing projects commonly used with agile
software development. See also agile software development.
security:Attributes of software products that bear on its ability to prevent unauthorized access,
whether accidental or deliberate, to programs and data. [ISO 9126] See also functionality.
security testing:Testing to determine the security of the software product. See also functionality
testing.
security testing tool:A tool that provides support for testing security characteristics and
vulnerabilities.
security tool:A tool that supports operational security.
serviceability testing:See maintainability testing.
session-based test management:A method for measuring and managing session-based testing, e.g.
exploratory testing.
session-based testing:An approach to testing in which test activities are planned as uninterrupted
sessions of test design and execution, often used in conjunction with exploratory testing.
severity:The degree of impact that a defect has on the development or operation of a component or
system. [After IEEE 610]

simulation:The representation of selected behavioral characteristics of one physical or abstract

system by another system. [ISO 2382/1]
simulator:A device, computer program or system used during testing, which behaves or operates like
a given system when provided with a set of controlled inputs. [After IEEE 610, DO178b] See also
emulator.
site acceptance testing:Acceptance testing by users/customers at their site, to determine whether or
not a component or system satisfies the user/customer needs and fits within the business processes,
normally including hardware as well as software.
smoke test:A subset of all defined/planned test cases that cover the main functionality of a
component or system, to ascertaining that the most crucial functions of a program work, but not
bothering with finer details. A daily build and smoke test is among industry best practices. See
also intake test.
software:Computer programs, procedures, and possibly associated documentation and data
pertaining to the operation of a computer system. [IEEE 610]
software attack:See attack.
Software Failure Mode and Effect Analysis (SFMEA):See Failure Mode and Effect Analysis
(FMEA).
Software Failure Mode, Effects, and Criticality Analysis (SFMECA):See Failure Mode, Effects,
and Criticality Analysis (FMECA).
Software Fault Tree Analysis (SFTA):See Fault Tree Analysis (FTA).
software feature:See feature.
software lifecycle:The period of time that begins when a software product is conceived and ends
when the software is no longer available for use. The software lifecycle typically includes a
concept phase, requirements phase, design phase, implementation phase, test phase, installation
and checkout phase, operation and maintenance phase, and sometimes, retirement phase. Note
these phases may overlap or be performed iteratively.
Software Process Improvement:A program of activities designed to improve the performance and
maturity of the organizations software processes and the results of such a program. [After CMMI]
software product characteristic:See quality attribute.
software quality:The totality of functionality and features of a software product that bear on its
ability to satisfy stated or implied needs. [After ISO 9126]
software quality characteristic:See quality attribute.
software test incident:See incident.
software test incident report:See incident report.
Software Usability Measurement Inventory (SUMI):A questionnaire-based usability test technique
for measuring software quality from the end users point of view. [Veenendaal04]
source statement:See statement.
specification:A document that specifies, ideally in a complete, precise and verifiable manner, the
requirements, design, behavior, or other characteristics of a component or system, and, often, the

procedures for determining whether these provisions have been satisfied. [After IEEE 610]
specification-based testing:See black box testing.
specification-based technique: See black box test design technique.
specification-based test design technique:See black box test design technique.
specified input:An input for which the specification predicts a result.
SPI:See Sofware Process Improvement.
stability:The capability of the software product to avoid unexpected effects from modifications in the
software. [ISO 9126] See also maintainability.
staged representation:A model structure wherein attaining the goals of a set of process areas
establishes a maturity level; each level builds a foundation for subsequent levels. [CMMI]
standard:Formal, possibly mandatory, set of requirements developed and used to prescribe
consistent approaches to the way of working or to provide guidelines (e.g., ISO/IEC standards,
IEEE standards, and organizational standards). [After CMMI]
standard software:See off-the-shelf software.
standards testing:See compliance testing.
state diagram:A diagram that depicts the states that a component or system can assume, and shows
the events or circumstances that cause and/or result from a change from one state to another. [IEEE
610]
state table:A grid showing the resulting transitions for each state combined with each possible event,
showing both valid and invalid transitions.
state transition:A transition between two states of a component or system.
state transition testing:A black box test design technique in which test cases are designed to execute
valid and invalid state transitions. See also N-switch testing.
statement:An entity in a programming language, which is typically the smallest indivisible unit of
execution.
statement coverage:The percentage of executable statements that have been exercised by a test
suite.
statement testing:A white box test design technique in which test cases are designed to execute
statements.
static analysis:Analysis of software artifacts, e.g. requirements or code, carried out without
execution of these software development artifacts. Static analysis is usually carried out by means
of a supporting tool.
static analysis tool:See static analyzer.
static analyzer:A tool that carries out static analysis.
static code analysis:Analysis of source code carried out without execution of that software.
static code analyzer:A tool that carries out static code analysis. The tool checks source code, for
certain properties such as conformance to coding standards, quality metrics or data flow
anomalies.

static testing:Testing of a component or system at specification or implementation level without

execution of that software, e.g. reviews or static analysis.
statistical testing:A test design technique in which a model of the statistical distribution of the input
is used to construct representative test cases. See also operational profile testing.
status accounting:An element of configuration management, consisting of the recording and reporting
of information needed to manage a configuration effectively. This information includes a listing of
the approved configuration identification, the status of proposed changes to the configuration, and
the implementation status of the approved changes. [IEEE 610]
STEP:See Systematic Test and Evaluation Process.
storage:See resource utilization.
storage testing:See resource utilization testing.
stress testing:A type of performance testing conducted to evaluate a system or component at or
beyond the limits of its anticipated or specified work loads, or with reduced availability of
resources such as access to memory or servers. [After IEEE 610] See also performance testing,
load testing.
stress testing tool:A tool that supports stress testing.
structural coverage:Coverage measures based on the internal structure of a component or system.
structural test design technique:See white box test design technique.
structural testing:See white box testing.
structure-based test design technique:See white box test design technique.
structure-based testing:See white-box testing.
structured walkthrough:See walkthrough.
stub:A skeletal or special-purpose implementation of a software component, used to develop or test a
component that calls or is otherwise dependent on it. It replaces a called component. [After IEEE
610]
subpath:A sequence of executable statements within a component.
suitability:The capability of the software product to provide an appropriate set of functions for
specified tasks and user objectives. [ISO 9126] See also functionality.
suitability testing:The process of testing to determine the suitability of a software product
suspension criteria:The criteria used to (temporarily) stop all or a portion of the testing activities on
the test items. [After IEEE 829]
syntax testing:A black box test design technique in which test cases are designed based upon the
definition of the input domain and/or output domain.
system:A collection of components organized to accomplish a specific function or set of functions.
[IEEE 610]
system integration testing:Testing the integration of systems and packages; testing interfaces to
external organizations (e.g. Electronic Data Interchange, Internet).

system of systems:Multiple heterogeneous, distributed systems that are embedded in networks at

multiple levels and in multiple interconnected domains, addressing large-scale inter-disciplinary
common problems and purposes, usually without a common management structure.
system testing:The process of testing an integrated system to verify that it meets specified
requirements. [Hetzel]
Systematic Test and Evaluation Process:A structured testing methodology, also used as a contentbased model for improving the testing process. Systematic Test and Evaluation Process (STEP)
does not require that improvements occur in a specific order. See also content-based model.

T
technical review:A peer group discussion activity that focuses on achieving consensus on the
technical approach to be taken. [Gilb and Graham, IEEE 1028] See also peer review.
test:A set of one or more test cases. [IEEE 829]
test approach:The implementation of the test strategy for a specific project. It typically includes the
decisions made that follow based on the (test) projects goal and the risk assessment carried out,
starting points regarding the test process, the test design techniques to be applied, exit criteria and
test types to be performed.
test automation:The use of software to perform or support test activities, e.g. test management, test
design, test execution and results checking.
test basis:All documents from which the requirements of a component or system can be inferred. The
documentation on which the test cases are based. If a document can be amended only by way of
formal amendment procedure, then the test basis is called a frozen test basis. [After TMap]
test bed:See test environment.
test case:A set of input values, execution preconditions, expected results and execution
postconditions, developed for a particular objective or test condition, such as to exercise a
particular program path or to verify compliance with a specific requirement. [After IEEE 610]
test case design technique:See test design technique.
test case specification:A document specifying a set of test cases (objective, inputs, test actions,
expected results, and execution preconditions) for a test item. [After IEEE 829]
test case suite:See test suite.
test charter:A statement of test objectives, and possibly test ideas about how to test. Test charters
are used in exploratory testing. See also exploratory testing.
test closure:During the test closure phase of a test process data is collected from completed
activities to consolidate experience, testware, facts and numbers. The test closure phase consists
of finalizing and archiving the testware and evaluating the test process, including preparation of a
test evaluation report. See also test process.
test comparator:A test tool to perform automated test comparison of actual results with expected
results.
test comparison:The process of identifying differences between the actual results produced by the
component or system under test and the expected results for a test. Test comparison can be
performed during test execution (dynamic comparison) or after test execution.
test completion criteria:See exit criteria.
test condition:An item or event of a component or system that could be verified by one or more test
cases, e.g. a function, transaction, feature, quality attribute, or structural element.
test control:A test management task that deals with developing and applying a set of corrective
actions to get a test project on track when monitoring shows a deviation from what was planned.
See also test management.

test coverage:See coverage.

test cycle:Execution of the test process against a single identifiable release of the test object.
test data:Data that exists (for example, in a database) before a test is executed, and that affects or is
affected by the component or system under test.
test data preparation tool:A type of test tool that enables data to be selected from existing databases
or created, generated, manipulated and edited for use in testing.
test deliverable:Any test (work) product that must be delivered to someone other than the test (work)
products author. See also deliverable.
test design:(1) See test design specification.
(2) The process of transforming general testing objectives into tangible test conditions and test cases.
test design specification:A document specifying the test conditions (coverage items) for a test item,
the detailed test approach and identifying the associated high level test cases. [After IEEE 829]
test design technique:Procedure used to derive and/or select test cases.
test design tool:A tool that supports the test design activity by generating test inputs from a
specification that may be held in a CASE tool repository, e.g. requirements management tool, from
specified test conditions held in the tool itself, or from code.
test driven development:A way of developing software where the test cases are developed, and
often automated, before the software is developed to run those test cases.
test driver:See driver.
test environment:An environment containing hardware, instrumentation, simulators, software tools,
and other support elements needed to conduct a test. [After IEEE 610]
test estimation:The calculated approximation of a result related to various aspects of testing (e.g.
effort spent, completion date, costs involved, number of test cases, etc.) which is usable even if
input data may be incomplete, uncertain, or noisy.
test evaluation report:A document produced at the end of the test process summarizing all testing
activities and results. It also contains an evaluation of the test process and lessons learned.
test execution:The process of running a test on the component or system under test, producing actual
result(s).
test execution automation:The use of software, e.g. capture/playback tools, to control the execution
of tests, the comparison of actual results to expected results, the setting up of test preconditions,
and other test control and reporting functions.
test execution phase:The period of time in a software development lifecycle during which the
components of a software product are executed, and the software product is evaluated to
determine whether or not requirements have been satisfied. [IEEE 610]
test execution schedule:A scheme for the execution of test procedures. The test procedures are
included in the test execution schedule in their context and in the order in which they are to be
executed.
test execution technique:The method used to perform the actual test execution, either manual or
automated.

test execution tool:A type of test tool that is able to execute other software using an automated test
script, e.g. capture/playback. [Fewster and Graham]
test fail:See fail.
test generator:See test data preparation tool.
test harness:A test environment comprised of stubs and drivers needed to execute a test.
test implementation:The process of developing and prioritizing test procedures, creating test data
and, optionally, preparing test harnesses and writing automated test scripts.
test improvement plan:A plan for achieving organizational test process improvement objectives
based on a thorough understanding of the current strengths and weaknesses of the organizations
test processes and test process assets. [After CMMI]
test incident:See incident.
test incident report:See incident report.
test infrastructure:The organizational artifacts needed to perform testing, consisting of test
environments, test tools, office environment and procedures.
test input:The data received from an external source by the test object during test execution. The
external source can be hardware, software or human.
test item:The individual element to be tested. There usually is one test object and many test items.
See also test object.
test item transmittal report:See release note.
test leader:See test manager.
test level:A group of test activities that are organized and managed together. A test level is linked to
the responsibilities in a project. Examples of test levels are component test, integration test,
system test and acceptance test. [After TMap]
test log:A chronological record of relevant details about the execution of tests. [IEEE 829]
test logging:The process of recording information about tests executed into a test log.
test management:The planning, estimating, monitoring and control of test activities, typically
carried out by a test manager.
test management tool:A tool that provides support to the test management and control part of a test
process. It often has several capabilities, such as testware management, scheduling of tests, the
logging of results, progress tracking, incident management and test reporting.
test manager:The person responsible for project management of testing activities and resources, and
evaluation of a test object. The individual who directs, controls, administers, plans and regulates
the evaluation of a test object.
Test Maturity Model (TMM):A five level staged framework for test process improvement, related
to the Capability Maturity Model (CMM), that describes the key elements of an effective test
process.
Test Maturity Model Integrated (TMMi):A five level staged framework for test process
improvement, related to the Capability Maturity Model Integration (CMMI), that describes the key
elements of an effective test process.

test monitoring:A test management task that deals with the activities related to periodically checking
the status of a test project. Reports are prepared that compare the actuals to that which was
planned. See also test management.
test object:The component or system to be tested. See also test item.
test objective:A reason or purpose for designing and executing a test.
test oracle:A source to determine expected results to compare with the actual result of the software
under test. An oracle may be the existing system (for a benchmark), other software, a user manual,
or an individuals specialized knowledge, but should not be the code. [After Adrion]
test outcome:See result.
test pass:See pass.
test performance indicator:A high level metric of effectiveness and/or efficiency used to guide and
control progressive test development, e.g. Defect Detection Percentage (DDP).
test phase:A distinct set of test activities collected into a manageable phase of a project, e.g. the
execution activities of a test level. [After Gerrard]
test plan:A document describing the scope, approach, resources and schedule of intended test
activities. It identifies amongst others test items, the features to be tested, the testing tasks, who
will do each task, degree of tester independence, the test environment, the test design techniques
and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring
contingency planning. It is a record of the test planning process. [After IEEE 829]
test planning:The activity of establishing or updating a test plan.
Test Point Analysis (TPA):A formula based test estimation method based on function point analysis.
[TMap]
test policy:A high level document describing the principles, approach and major objectives of the
organization regarding testing.
test procedure:See test procedure specification.
test procedure specification:A document specifying a sequence of actions for the execution of a test.
Also known as test script or manual test script. [After IEEE 829]
test process:The fundamental test process comprises test planning and control, test analysis and
design, test implementation and execution, evaluating exit criteria and reporting, and test closure
activities.
Test Process Group:A collection of (test) specialists who facilitate the definition, maintenance, and
improvement of the test processes used by an organization. [After CMMI]
Test Process Improvement (TPI):A continuous framework for test process improvement that
describes the key elements of an effective test process, especially targeted at system testing and
acceptance testing.
test process improvement manifesto:A statement that echoes the agile manifesto, and defines values
for improving the testing process. The values are:
- flexibility over detailed processes
- best Practices over templates

- deployment orientation over process orientation

- peer reviews over quality assurance (departments)
- business driven over model driven. [Veenendaal08]
test process improver:A person implementing improvements in the test process based on a test
improvement plan.
test progress report: A document summarizing testing activities and results, produced at regular
intervals, to report progress of testing activities against a baseline (such as the original test plan)
and to communicate risks and alternatives requiring a decision to management.
test record:See test log.
test recording:See test logging.
test report:See test summary report and test progress report.
test reproducibility:An attribute of a test indicating whether the same results are produced each time
the test is executed.
test requirement:See test condition.
test result:See result.
test rig:See test environment.
test run:Execution of a test on a specific version of the test object.
test run log:See test log.
test scenario:See test procedure specification.
test schedule:A list of activities, tasks or events of the test process, identifying their intended start
and finish dates and/or times, and interdependencies.
test script:Commonly used to refer to a test procedure specification, especially an automated one.
test session:An uninterrupted period of time spent in executing tests. In exploratory testing, each test
session is focused on a charter, but testers can also explore new opportunities or issues during a
session. The tester creates and executes test cases on the fly and records their progress. See also
exploratory testing.
test set:See test suite.
test situation:See test condition.
test specification:A document that consists of a test design specification, test case specification
and/or test procedure specification.
test specification technique:See test design technique.
test stage:See test level.
test strategy:A high-level description of the test levels to be performed and the testing within those
levels for an organization or programme (one or more projects).
test suite:A set of several test cases for a component or system under test, where the post condition
of one test is often used as the precondition for the next one.
test summary report:A document summarizing testing activities and results. It also contains an
evaluation of the corresponding test items against exit criteria. [After IEEE 829]

test target:A set of exit criteria.

test technique:See test design technique.
test tool:A software product that supports one or more test activities, such as planning and control,
specification, building initial files and data, test execution and test analysis. [TMap] See also
CAST.
test type:A group of test activities aimed at testing a component or system focused on a specific test
objective, i.e. functional test, usability test, regression test etc. A test type may take place on one
or more test levels or test phases. [After TMap]
testability:The capability of the software product to enable modified software to be tested. [ISO
9126] See also maintainability.
testability review:A detailed check of the test basis to determine whether the test basis is at an
adequate quality level to act as an input document for the test process. [After TMap]
testable requirements:The degree to which a requirement is stated in terms that permit establishment
of test designs (and subsequently test cases) and execution of tests to determine whether the
requirements have been met. [After IEEE 610]
tester:A skilled professional who is involved in the testing of a component or system.
testing:The process consisting of all lifecycle activities, both static and dynamic, concerned with
planning, preparation and evaluation of software products and related work products to determine
that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect
defects.
testware:Artifacts produced during the test process required to plan, design, and execute tests, such
as documentation, scripts, inputs, expected results, set-up and clear-up procedures, files,
databases, environment, and any additional software or utilities used in testing. [After Fewster and
Graham]
thread testing:A version of component integration testing where the progressive integration of
components follows the implementation of subsets of the requirements, as opposed to the
integration of components by levels of a hierarchy.
time behavior:See performance.
top-down testing:An incremental approach to integration testing where the component at the top of
the component hierarchy is tested first, with lower level components being simulated by stubs.
Tested components are then used to test lower level components. The process is repeated until the
lowest level components have been tested. See also integration testing.
Total Quality Management:An organization-wide management approach centered on quality, based
on the participation of all its members and aiming at long-term success through customer
satisfaction, and benefits to all members of the organization and to society. Total Quality
Management consists of planning, organizing, directing, control, and assurance. [After ISO 8402]
TPG:See Test Process Group.
TQM:See Total Quality Management.
traceability:The ability to identify related items in documentation and software, such as requirements
with associated tests. See also horizontal traceability, vertical traceability.

transactional analysis:The analysis of transactions between people and within peoples minds; a
transaction is defined as a stimulus plus a response. Transactions take place between people and
between the ego states (personality segments) within one persons mind.
transcendent-based quality:A view of quality, wherein quality cannot be precisely defined, but we
know it when we see it, or are aware of its absence when it is missing. Quality depends on the
perception and affective feelings of an individual or group of individuals towards a product.
[After Garvin] See also manufacturing-based quality, product-based quality, user-based quality,
value-based quality.

U
understandability:The capability of the software product to enable the user to understand whether the
software is suitable, and how it can be used for particular tasks and conditions of use. [ISO 9126]
See also usability.
unit:See component.
unit test framework:A tool that provides an environment for unit or component testing in which a
component can be tested in isolation or with suitable stubs and drivers. It also provides other
support for the developer, such as debugging capabilities. [Graham]
unit testing:See component testing.
unreachable code:Code that cannot be reached and therefore is impossible to execute.
usability:The capability of the software to be understood, learned, used and attractive to the user
when used under specified conditions. [ISO 9126]
usability testing:Testing to determine the extent to which the software product is understood, easy to
learn, easy to operate and attractive to the users under specified conditions. [After ISO 9126]
use case:A sequence of transactions in a dialogue between an actor and a component or system with
a tangible result, where an actor can be a user or anything that can exchange information with the
system.
use case testing:A black box test design technique in which test cases are designed to execute
scenarios of use cases.
user acceptance testing:See acceptance testing.
user-based quality:A view of quality, wherein quality is the capacity to satisfy needs, wants and
desires of the user(s). A product or service that does not fulfill user needs is unlikely to find any
users. This is a context dependent, contingent approach to quality since different business
characteristics require different qualities of a product. [after Garvin] See also manufacturingbased quality, product-based quality, transcendent-based quality, value-based quality.
user scenario testing:See use case testing.
user test:A test whereby real-life users are involved to evaluate the usability of a component or
system.

V
V-model:A framework to describe the software development lifecycle activities from requirements
specification to maintenance. The V-model illustrates how testing activities can be integrated into
each phase of the software development lifecycle.
validation:Confirmation by examination and through provision of objective evidence that the
requirements for a specific intended use or application have been fulfilled. [ISO 9000]
value-based quality:A view of quality, wherein quality is defined by price. A quality product or
service is one that provides desired performance at an acceptable cost. Quality is determined by
means of a decision process with stakeholders on trade-offs between time, effort and cost aspects.
[After Garvin] See also manufacturing-based quality, product-based quality, transcendentbased quality, user-based quality.
variable:An element of storage in a computer that is accessible by a software program by referring to
it by a name.
verification:Confirmation by examination and through provision of objective evidence that specified
requirements have been fulfilled. [ISO 9000]
version control:See configuration control.
vertical traceability:The tracing of requirements through the layers of development documentation to
components.
volume testing:Testing where the system is subjected to large volumes of data. See also resourceutilization testing.

W
walkthrough:A step-by-step presentation by the author of a document in order to gather information
and to establish a common understanding of its content. [Freedman and Weinberg, IEEE 1028] See
also peer review.
WBS:See Work Breakdown Structure.
white-box technique:See white-box test design technique.
white-box test design technique:Procedure to derive and/or select test cases based on an analysis of
the internal structure of a component or system.
white-box testing:Testing based on an analysis of the internal structure of the component or system.
Wide Band Delphi:An expert based test estimation technique that aims at making an accurate
estimation using the collective wisdom of the team members.
wild pointer:A pointer that references a location that is out of scope for that pointer or that does not
exist. See also pointer.
Work Breakdown Structure:An arrangement of work elements and their relationship to each other
and to the end product. [CMMI]

Annex A (Informative)
Index of sources; the following non-normative sources were used in constructing this glossary:
[Abbott] J. Abbot (1986), Software Testing Techniques, NCC Publications.
[Adrion] W. Adrion, M. Branstad and J. Cherniabsky (1982), Validation, Verification and Testing of
Computer Software, in: Computing Surveys, Vol. 14, No 2, June 1982.
[Bach] J. Bach (2004), Exploratory Testing, in: E. van Veenendaal, The Testing Practitioner 2nd
edition, UTN Publishing, ISBN 90-72194-65-9.
[Beizer] B. Beizer (1990), Software Testing Techniques, van Nostrand Reinhold, ISBN 0-44220672-0
[Chow] T. Chow (1978), Testing Software Design Modelled by Finite-Sate Machines, in: IEEE
Transactions on Software Engineering, Vol. 4, No 3, May 1978.
[CMM] M. Paulk, C. Weber, B. Curtis and M.B. Chrissis (1995), The Capability Maturity Model,
Guidelines for Improving the Software Process, Addison-Wesley, ISBN 0-201-54664-7
[CMMI] M.B. Chrissis, M. Konrad and S. Shrum (2004), CMMI, Guidelines for Process Integration
and Product Improvement, Addison Wesley, ISBN 0-321-15496-7
[Deming] D. W. Edwards (1986), Out of the Crisis, MIT Center for Advanced Engineering Study,
ISBN 0-911379-01-0
[Fenton] N. Fenton (1991), Software Metrics: a Rigorous Approach, Chapman & Hall, ISBN 053249-425-1
[Fewster and Graham] M. Fewster and D. Graham (1999), Software Test Automation, Effective use
of test execution tools, Addison-Wesley, ISBN 0-201-33140-3.
[Freedman and Weinberg] D. Freedman and G. Weinberg (1990), Walkthroughs, Inspections, and
Technical Reviews, Dorset House Publishing, ISBN 0-932633-19-6.
[Garvin] D.A. Garvin (1984), What does product quality really mean?, in: Sloan Management
Review, Vol. 26, nr. 1 1984
[Gerrard] P. Gerrard and N. Thompson (2002), Risk-Based E-Business Testing, Artech House
Publishers, ISBN 1-58053-314-0.
[Gilb and Graham] T. Gilb and D. Graham (1993), Software Inspection, Addison-Wesley, ISBN 0201-63181-4.
[Graham] D. Graham, E. van Veenendaal, I. Evans and R. Black (2007), Foundations of Software
Testing, Thomson Learning, ISBN 978-1-84480-355-2
[Grochtmann] M. Grochtmann (1994), Test Case Design Using Classification Trees, in: Conference
Proceedings STAR 1994.
[Hetzel] W. Hetzel (1988), The complete guide to software testing 2nd edition, QED Information
Sciences, ISBN 0-89435-242-3.
[Juran] J.M. Juran (1979), Quality Control Handbook, McGraw-Hill

[McCabe] T. McCabe (1976), A complexity measure, in: IEEE Transactions on Software

Engineering, Vol. 2, pp. 308-320.
[Musa] J. Musa (1998), Software Reliability Engineering Testing, McGraw-Hill Education, ISBN 007913-271-5.
[Myers] G. Myers (1979), The Art of Software Testing, Wiley, ISBN 0-471-04328-1.
[TMap] M. Pol, R. Teunissen, E. van Veenendaal (2002), Software Testing, A guide to the TMap
Approach, Addison Wesley, ISBN 0-201-745712.
[Veenendaal04] E. van Veenendaal (2004), The Testing Practitioner 2nd edition, UTN Publishing,
ISBN 90-72194-65-9.
[Veenendaal08] E. van Veendaal (2008), Test Improvement Manifesto, in: Testing Experience, Issue
04/08, December 2008

Annex B (Method of commenting on this glossary)

Comments are invited on this document so that the glossary can be further improved to satisfy the
needs of the testing community.
When making a comment, be sure to include the following information:
Your name and contact details;
The version number of the glossary (currently 2.1);
Exact part of the glossary;
For new glossary entries, also include a reference to the ISTQB syllabus that uses the term;
Supporting information, such as the reason for a proposed change, or a reference to the use of a
term.
You can submit comments in a variety of ways, which in order of preference are as follows:
1. By E-mail to eve@improveqs.nl;
2. By post to Improve Quality Services BV, attn. Mr. E. van Veenendaal, Laan van Diepenvoorde 1,
5551 TC, Waalre, The Netherlands;
3. By FAX to +31 40 20 21450, marked for the attention of Mr. E. van Veenendaal.