Ch.

1 A Perspective on Testing
Definitions and Concepts of Testing and Quality

What is Testing?

How Related?

What is Quality ?

1. Does finding a Lot of Problems through testing improve Quality? Ch. 1 A Perspective on Testing Sum2012Bernal

Introduction to Testing Software

Testing is a relatively new discipline although programmers always debugged their programs.
Testing was conducted to show that software works.

In the 1970s Glen Myers wrote a book, Art of Software Testing (1978).
He believed that the main purpose of testing is to find faults.

Are these (works .vs. faults) opposing views?

2. Why do we and why should we Test ?

Ch. 1 A Perspective on Testing Sum2012Bernal 2

Why Test ?
Because we cant assume that the software works
How do we answer the question: Does this software work?
Is it reliable ? (system is continuously functioning > 720hrs) Is it functional ?
Complete Consistent

Is it responsive ? (response time < 1 second)

In general, what is the Quality of our software?

How do we answer this?
3. How would you answer this about the program that you wrote? How would you answer this about the program your friend wrote?
Ch. 1 A Perspective on Testing Sum2012Bernal 3

Quality of Software?
Depends on what we include as software
Just the executable code ----- How about ?
the pre-populated data in the database the help text and error messages the source logic code the design document the test scenarios and test results the reference manuals the requirements document

When we talk about quality how much of the above (do we / should we) include? How would you test these different artifacts?
Ch. 1 A Perspective on Testing Sum2012Bernal 4

Approaches to ---Testing Software

Some of us hope that our software works as opposed to ensuring that our software works? Why?
Just foolish Lazy Believe that its too costly (time, resource, effort, etc.) Lack of knowledge

DO NOT use the I feel lucky or I feel confident approach to testing - - - - although you may feel that way sometimes.

Use a methodical approach to testing to back up the I feel lucky/confident feeling

Methods and metrics utilized must show VALUE Value, unfortunately, often are expressed in negative terms
Severe problems that cost loss of lives or business Problems that cost more than testing expenses and effort
5

3 major Testing approaches

1. Traditionally, testing includes executing the code with test cases. (assuming - code is the main software artifact)

2. What do we do with the non-executable software artifacts? Reviews and Inspections - Expensive in terms of human resources - A lot to maintain and keep updated

3. Can we prove that the software works or is defect free? - Theoretically, given an arbitrary program we can not show that it has no bug. - We can use Formal Proofs to show the behavior of code
6

An Informal Survey in the 90s

Professionals taking a course in testing:
60% were new to testing 20% had 1 to 5 years of experience in testing 20 % expert testers

Metric used in testing

a) Most regularly used: Counting bugs found and ranking them by severity b) Small number used : bug find rate or bug fix rate

Formal Methods used

Almost none formally trained in inspection or analysis

Test tool:
76 % had been exposed to some automated test tool

Test definitions
Most of the practicing testers could not supply good definitions of 7 testing terms; theyCh. 1 Adid the work! just Perspective on Testing Sum2012Bernal

Historical Progression of Attitudes on Software Quality

-Large Host & Centralized Systems -Single Vendor (hdw-sw-serv) -Long term development and long term investment(10 yrs) -Single platform -Systems ran by computer professionals **Product Reliability & Quality was required and expected** Even though we didnt get good quality -PC and Desktop Computing became ubiquitous -multiple vendors -quicker product development & shorter term investment -systems ran by non-computer individuals **New product was fashionable & reboot became acceptable.**

1980s & Before

1990s

-Web availability to many -Business conducted on the Web -Software and systems are not hobbies but a business again **Product Reliability & Quality is once again important** -especially SECURITY--

Ch. 1 A Perspective on Testing Sum2012Bernal

Late 1990s & 2000s

Current State (2000-2010s) on Testing

1.

Software is written by many with the entrepreneurial spirit:

Speed to market New & innovation is treasured Small organization that cant afford much more than coders

2.

Embracing Agile process and mistaking it as fast production regardless of what:

Not much documented (requirements/design) Hard to test without documented material
Why ?

3.

Lack of Trained/Good/Experienced Testers

Testers are not quite rewarded as equals to designers, but definitely gaining on and sometimes surpassing programmers (takes good cops to catch the thieves)

4.

Improvement in tools and standards making Ch. 1 A Perspective on Testing development easier and less error prone Sum2012Bernal

How 9 ?

Is Quality still an Issue, today?

What is Quality? Some common comments:
I know it when I see it Reliable Meets requirements Fit for use Easy to use Responsive Full function / full feature Classy and luxurious
How would you define quality?
Ch. 1 A Perspective on Testing Sum2012Bernal 10

Some U.S. pioneers on Quality Pioneers:

Joseph M. Juran
Quality =>Fitness for use

W. Edward Deming
Quality => Non-faulty system

More recently:
Philip Crosby
Quality => conformance with requirements Achieve quality via prevention of error Target of Quality is zero defect Measure success via cost of quality
Ch. 1 A Perspective on Testing Sum2012Bernal 11

ISO/IEC 9126 (1994) Quality model Quality is composed of several characteristics:

Functionality Reliability Usability Efficiency Maintainability Portability

What do these mean ?

How do you know if it is achieved & how would you test for these? Do these have to be specified in the requirements? ---- if they are not-------- do we ask for these during review/inspection?
Ch. 1 A Perspective on Testing Sum2012Bernal 12

Quality
Quality is a characteristic or attribute
Needs to be clearly defined and agreed to
May have sub-attributes (e.g. previous page) Needs specific metrics for each of the sub-attributes

Needs to be measured with the defined metric(s)

Needs to be tracked and analyzed Needs to be projected Needs to be controlled
Imagine what it is like without this.

Testing and Measurement are two key activities that would help us manage quality.
Explain ----- the Perspective on Testing Ch. 1 A relationship ?
Sum2012Bernal 13

Summarizing major Competitors to Quality

Schedule (first to market) Requirements (bad or missing) New and Exciting (demands of WANT not need) Price and Availability (retail customers)

Ch. 1 A Perspective on Testing Sum2012Bernal

14

Some Precept Concerning Quality & QA

Quality requirements Does Not always dictate schedule !
Market Condition often dictates schedule (especially for small companies) BUT For large and multiple release software, quality is still a factor and may affect schedule ---- albeit schedule is seldomly changed for quality Software development process today incorporates both the need for speed and quality (incorporating the notion of )
a) service cost and b) rewrite for a replacement new product.

Quality does not require zero defect Reliability

Commercial (non-life critical or mission critical) products are not developed with zero defect goal in mind. They are much more market driven --- market prefers but does not demand zero defects.
Focus on proper support Focus on main functions and heavily used areas (not all defects are the same) Focus on customer productivity (e.g. easy to learn and use) 15 Ch. 1 A Perspective on Testing Zero Defect is very expensive proposition ( time & resource)
Sum2012Bernal

Users Do Not Always Know What They Want

Users may not know all the requirements (especially for large, complex systems which require professional or legal knowledge.) Users may not have the time or interest to really focus on the requirements at the time when asked (timing problem). Users have their own fulltime jobs. Users may not know how to prioritize needs from wishes Users may not know how to articulate clearly all the requirements. (They are non-software development people.) Developers may not listen well or properly interpret the users statements. (They are not industry specialists.)
Requirements is a key factor in software development ---- why? How does it affect software quality? ----- think about definitions of Quality --- meets requirements
Ch. 1 A Perspective on Testing Sum2012Bernal

16

Requirements are not always Clear, Stable, and Doable

Not all requirements are technically feasible; sometimes the desired new technology needs to be prototyped first. Sometimes the requirements are changed, causing redesign or re-code without proper assessment of schedule impact. Requirements are not always reviewed and signed off, but sometimes given in verbal form --- especially small changes. People mistake iterative development to mean continuous change of requirements.

Whats the danger here? cost, schedule, quality

Ch. 1 A Perspective on Testing Sum2012Bernal

17

Customers often go for New and Exciting Product

If the product has all the needed features it would sell --is not necessarily true; people often WANT new & extra features. Reliability is not always enough; sometimes customers will sacrifice quality for new and exciting features.
The story of IBM OS/2 operating system and Microsofts DOS operating system (even though both was commissioned by IBM).
IBM went for Reliability of the old Host Machines for desktop PCs Microsoft went for exciting individual user interfaces

Over-emphasis of exciting features is one reason why we are regressing a little in software quality in the last ten years !
**Still, consider Apple i-phone success inon Testing its activation & other problems 18 Ch. 1 A Perspective spite of Sum2012Bernal

Price and Availability is sometimes more important to customers (especially for commodity level software) than Product Maturity or Quality.
At the commodity level software, the customers are individuals who wants the product NOW at an competitive price. (much like shopping for a home appliance such as a coffee maker, T.V. or an i-phone) Sophisticated and full feature software needs to be balanced and sometimes traded off for price and speed. Customers dont always need all the functions and product maturity they think they require ---- if the price is right!

Ch. 1 A Perspective on Testing Sum2012Bernal

19

Summarizing: major
Competitors and Adversaries to Quality when developing software
Schedule (first to market) Requirements (bad or missing) New and Exciting (demands of WANT
not need)

Price and Availability (retail customers)

Ch. 1 A Perspective on Testing Sum2012Bernal 20

Start of Quality Assurance

As software size and complexity increased in the 1960s and 1970s, many software projects started to fail.
Many software did not perform the required functions Others had performance (speed) problems Some had large number of defects that prevented users from completing their work; some just flat out wont even install !

Software Quality Crisis was recognized and Quality Assurance was born, along with the term Software Engineering (1968 NATO conference).

Ch. 1 A Perspective on Testing Sum2012Bernal

21

Software Quality Assurance (QA)

Software QA focused on 2 main areas
Software product Software process

emphasis today : e.g. CMM & CMMI

The focus on the process areas borrowed many techniques from the traditional manufacturing area
Concept of reliability (number of defects, mean time to failure, probability of failure, etc. metrics) Concept of process control in terms of looking at repeatability of process--- repeatable process produces similar product (or controllable results).
Ch. 1 A Perspective on Testing Sum2012Bernal 22

QA, Process Control, and Documentation

A period of heavy emphasis on software development process and excessive documentation dominated QA --- initially this improved the software crisis.
1. Process included multiple steps of reviews 2. Process included multiple steps of test preparation, testing, and test result analysis 3. Process was controlled by many documents and document flow which also improved project communications

But ----- the price paid was

a) speed and b) innovation.
Ch. 1 A Perspective on Testing Sum2012Bernal 23

Software Development is NOT Manufacturing (or is it?)

Software Development is extremely labor intensive
People are not uniform like machines used in manufacturing.

Software Development often requires innovation

Every software seems to be a one of its kind although more and more are becoming standardized The same set of people do not get to repeatedly 24 Ch. 1 A Perspective Testing develop the exactly sameonsoftware multiple times. Sum2012Bernal

Improve Documentation (Availability and Presentation Technique)

Improve the creation and maintenance of documents via an on-line repository or configuration manager
Centrally protected data Sharable data Managed data (data relationships are managed) track, project, control

Improve the usability and productivity by providing more and better visualization of data
Replace numbers with graphs and figures Replace words with pictures

Ch. 1 A Perspective on Testing Sum2012Bernal

25

Improve Testers Tools and Methodology

Test Methodology Improvements
Test coverage analysis Test case generation Test-Fix-Integrate Process Test results analysis Test metrics definition and measurements process Etc. Test coverage computation Test trace Test script generator Test result records keeping and automated analysis Build and integration (daily builds) Etc.
Ch. 1 A Perspective on Testing Sum2012Bernal 26

Test tools improvements

Perspective on Testing
Today we test because we know that systems have problems - - - we are fallible.
1. To find problems and find the parts that do not work 2. To understand and show the parts that do work 3. To assess the quality of the over-all product (A major QA and release management responsibility)
Ch. 1 A Perspective on Testing Sum2012Bernal 27

 Error

Some Talking Definitions (IEEE terminology based)

A mistake made by a human The mistake may be in the requirements, design, code, fix, integration, or install

Fault
Is a defect or defects in the artifact that resulted from an error There may be defects caused by errors made that may or may not be detectable (e.g. error of omission) Note

Failure
Is the manifestation of faults when the software is executed.
Running code May show up in (includes error of omission and no-code?) several places May be non-code related (e.g. reference manual) (Not in the text)
Example? (bank accnt)
28

Incident

Is the detectable symptom of failures

Ch. 1 A Perspective on Testing Sum2012Bernal

Testing in the Large

Testing is concerned with all, but may not be able to detect all
Errors Faults Failures Incidents

Testing utilize the notion of test cases to perform the activities of test(s)
Inspection of non-executables Executing the code Analyzing and formally proving the nonexecutables and the executable in a business workflow (or user) setting
Ch. 1 A Perspective on Testing Sum2012Bernal 29

Software Activities and Error Injections, Fault Passing, and Fault Removal
Error
Inspection

Requirements

Error
Inspection

faults Design

Error

Error

faults Code Testing Fixing Fixing

Faults/failures

Note that in fixing failures, one can commit errors and introduce faults
Ch. 1 A Perspective on Testing Sum2012Bernal 30

Recording each Test Case

Test Case number Test Case author A general description of the test purpose Pre-condition Test inputs Expected outputs (if any) Post-condition Test Case history:
Test execution date Test execution person Test execution result (s)
Ch. 1 A Perspective on Testing Sum2012Bernal 31

Specification vs Implementation
Specification Implementation

Expected

Actual

The ideal place where expectation and actual matches The other areas are of concern ---- especially to testers
Ch. 1 A Perspective on Testing Sum2012Bernal 32

Specification vs Implementation vs Test Cases

Area P = Blue= Programmed-Implementation P=Program->Actual 2 5 4 7 8 What do these numbered regions mean to you? T=Tested Area T = Pink=Test Cases
Ch. 1 A Perspective on Testing Sum2012Bernal 33

Area S = Yellow=Specification

S=Spec->Expected

6 1 3

Black Box vs White Box code testing

Black box testing (Functional Testing)
Look at mainly the input and outputs Mainly uses the specification as the source for designing test cases. The internal of the implementation is not included in the test case design. Hard to detect missing specification

White box testing (Structural Testing)

Look at the internals of the implementation Design test cases based on the design and code implementation Hard to detect extraneous implementation that was never specified
Ch. 1 A Perspective on Testing Sum2012Bernal 34

Recording Test Results

Use the same form describing the test case --- see earlier slide on recording test case and expand the results to include:
State Pass or Fail on the Execution Result line If failed:
1. Show output or some other indicator to demonstrate the fault or failure 2. Assess and record the severity of the fault or failure found

Ch. 1 A Perspective on Testing Sum2012Bernal

35

Failure Classification (Tsui)

Very High severity brings the systems down or a function is non-operational and there is no work around High severity a function is not operational but there is a manual work around Medium severity a function is partially operational but the work can be completed with some work around Low severity minor inconveniences but the work can be completed
Ch. 1 A Perspective on Testing Sum2012Bernal 36

Fault Classification (Beizer)

Mild misspelled word Moderate - misleading or redundant info Annoying truncated names; billing for $0.00 Disturbing some transactions not processed Serious - lose a transaction Very serious - incorrect transaction execution Extreme Frequent very serious errors Intolerable - database corruption Catastrophic System shutdown Infectious - Shutdown that spreads to others

Increasing severity

Ch. 1 A Perspective on Testing Sum2012Bernal

37

IEEE list of anomalies (faults)

Input/output faults Logic faults Computation faults Interface faults Data faults

Why do you care about these types of faults (results of errors made)? Because they give us some ideas of what to look for in designing future test cases --- in process of for other products developed by same people
Ch. 1 A Perspective on Testing Sum2012Bernal 38

Different Levels of Testing

Unit Testing Program unit A Function 1 Program unit B Component 1 Function 2 Functional Testing System Component Testing Testing

. . .
Program unit T

. .
Function 8

.
Component 3

Whole System

Ch. 1 A Perspective on Testing Sum2012Bernal

39

Still Need to Demonstrate Value of Testing

Catastrophic problems (e.g. life or business ending ones) do not need any measurements---but--- others do:
Measure the cost of problems found by customers
Cost of problem reporting/recording Cost of problem re-creation Cost of problem fix and retest Cost of solution packaging and distribution Cost of managing the customer problem-to-resolution steps

Measure the cost of discovering the problems and fixing them prior to release
Cost of planning reviews and testing Cost of executing reviews and testing Cost of fixing the problems found and retest Cost of inserting fixes and updates Cost of managing problem-to-resolution steps

Compare the above two costs AND include loss of customer good-will
Ch. 1 A Perspective on Testing Sum2012Bernal 40

Goals of Testing?
Test as much as time allows us Validate all the key areas

Very important to think about this ---

Execute as many test cases as (schedule) time allows? Test only the designated keyrequirements?

Find as much problems as possible

Test all the likely error prone areas and maximize test problems found?

Validate the requirements

Test all the requirements?

To reach a quality target

Quality Target?

State your goal(s) for testing. - - - what would you like people to say about your system ? Your goals may dictate your testing process
Ch. 1 A Perspective on Testing Sum2012Bernal 41