The Concept of Software Quality:

While to err is human, sometimes the cost of a mistake might be just too high.
History knows many examples of situations when software flaws have caused
billions of dollars in waste or even lead to casualties: from Starbucks coffee
shops being forced to give away free drinks because of a register malfunction,
to the F-35 military aircraft being unable to detect the targets correctly because
of a radar failure. In order to make sure the released software is safe and
functions as expected, the concept of software quality was introduced. It is often
defined as “the degree of conformance to explicit or implicit requirements and
expectations”. These so-called explicit and implicit expectations correspond to
the two basic levels of software quality:
•• Functional — the product’s compliance with functional (explicit)
requirements and design specifications. This aspect focuses on the practical
use of software, from the point of view of the user: its features, performance,
ease of use, absence of defects.
••Non-Functional — system’s inner characteristics and architecture, i.e.
structural (implicit) requirements. This includes the code maintainability,
understandability, efficiency, and security [2].
The structural quality of the software is usually hard to manage: It relies mostly
on the expertise of the engineering team and can be assured through code
review, analysis, and refactoring. At the same time, functional aspect can be
assured through a set of dedicated quality management activities, which
includes quality assurance, quality control and testing.

Principles of Software Testing

Testing shows presence of mistakes
Testing is aimed at detecting the defects within a piece of software.
But no matter how thoroughly the product is tested, we can never be
100 percent sure that there are no defects. We can only use testing to
reduce the number of unfound issues.
Exhaustive testing is impossible
There is no way to test all combinations of data inputs, scenarios
and preconditions within an application. For example, if a single app
screen contains 10 input fields with 3 possible value options each, this
means to cover all possible combinations, test engineers would need
to create 59,049 (310) test scenarios. And what if the app contains
50+ of such screens? In order not to spend weeks creating millions of
such less possible scenarios, it is better to focus on potentially more
significant ones.
Early testing
As mentioned above, the cost of an error grows exponentially
throughout the stages of the SDLC. Therefore, it is important to start
testing the software as soon as possible, so that the detected issues
are resolved and do not snowball.
Defect clustering
This principle is often referred to as an application of the Pareto
Principle to software testing. This means that approximately 80
percent of all errors are usually found in only 20 percent of the system
modules. Therefore, if a defect is found in a particular module of a
software program, the chances are there might be other defects. That
is why it makes sense to test that area of the product thoroughly ?
Pesticide paradox
Running the same set of tests again and again won’t help you find
more issues. As soon as the detected errors are fixed, these test
scenarios become useless. Therefore, it is important to review and
update the tests regularly in order to adapt and potentially find more
errors.
Testing is context dependent
Depending on their purpose or industry, different applications should
be tested differently. While safety could be of primary importance
for a fintech product, it is less important for a corporate website. The
latter, in its turn, puts an emphasis on usability and speed.
Absence-of-errors fallacy
The complete absence of errors in your product does not necessarily
mean its success. No matter how much time you have spent polishing
your code or improving the functionality, if your product is not useful
or does not meet the user expectations it won’t be adopted by the
target audience.
Test Plan
A Test Plan is a detailed document that describes the test strategy, objectives,
schedule, estimation, deliverables, and resources required to perform testing for
a software product. Test Plan helps us determine the effort needed to validate
the quality of the application under test. The test plan serves as a blueprint to
conduct software testing activities as a defined process, which is minutely
monitored and controlled by the test manager.

As per ISTQB definition: “Test Plan is A document describing the scope, approach,
resources, and schedule of intended test activities.”

How to write a Test Plan

You already know that making a Test Plan is the most important task of Test
Management Process. Follow the seven steps below to create a test plan as per
IEEE 829

1. Analyse the product

2. Design the Test Strategy
3. Define the Test Objectives
4. Define Test Criteria
5. Resource Planning
6. Plan Test Environment
7. Schedule & Estimation
8. Determine Test Deliverables
Step 1) Analyze the product
How can you test a product without any information about it? The answer
is Impossible. You must learn a product thoroughly before testing it.

The product under test is a banking website. You should research clients and the
end users to know their needs and expectations from the application

 Who will use the website?

 What is it used for?
 How will it work?
 What are software/ hardware the product uses?

Step 2) Develop Test Strategy

Test Strategy is a critical step in making a Test Plan in Software Testing. A Test
Strategy document, is a high-level document, which is usually developed by Test
Manager. This document defines:

 The project’s testing objectives and the means to achieve them

 Determines testing effort and costs

Step 2.1) Define Scope of Testing

How do you determine scope your project?

To determine scope, you must –

Precise customer requirement

Project Budget
Product Specification
Skills & talent of your test team

Step 2.2) Identify Testing Type

A Testing Type is a standard test procedure that gives an expected test outcome.
Each testing type is formulated to identify a specific type of product bugs. But, all
Testing Types are aimed at achieving one common goal “Early detection of all the
defects before releasing the product to the customer”

The commonly used testing types are described as following figure

Step 2.3) Document Risk & Issues

Risk is future’s uncertain event with a probability of occurrence and
a potential for loss. When the risk actually happens, it becomes the ‘issue’.

Risk Mitigation

Team member lack the Plan training course to skill up your members
required skills for website
testing.
 The project schedule is too Set Test Priority for each of the test activity.
tight; it's hard to complete
this project on time

Test Manager has poor Plan leadership training for manager

management skill

A lack of cooperation Encourage each team member in his task, and

negatively affects your inspire them to greater efforts.
employees' productivity

Wrong budget estimate and Establish the scope before beginning work, pay
cost overruns a lot of attention to project planning and
constantly track and measure the progress

Step 2.4) Create Test Logistics

In Test Logistics, the Test Manager should answer the following questions:

 Who will test?

 When will the test occur?

Step 3) Define Test Objective

Test Objective is the overall goal and achievement of the test execution. The
objective of the testing is finding as many software defects as possible; ensure
that the software under test is bug free before release.

To define the test objectives, you should do 2 following steps

1. List all the software features (functionality, performance, GUI…) which may
need to test.
2. Define the target or the goal of the test based on above features

Step 4) Define Test Criteria

Test Criteria is a standard or rule on which a test procedure or test judgment can
be based. There’re 2 types of test criteria as following
Suspension Criteria
Specify the critical suspension criteria for a test. If the suspension criteria are met
during testing, the active test cycle will be suspended until the criteria
are resolved.

Test Plan Example: If your team members report that there are 40% of test cases
failed, you should suspend testing until the development team fixes all the failed
cases.

Exit Criteria
It specifies the criteria that denote a successful completion of a test phase. The
exit criteria are the targeted results of the test and are necessary before
proceeding to the next phase of development. Example: 95% of all critical test
cases must pass.

Some methods of defining exit criteria are by specifying a targeted run

rate and pass rate.
  Run rate is ratio between number test cases executed/total test cases of
test specification. For example, the test specification has total 120 TCs, but
the tester only executed 100 TCs, So the run rate is 100/120 = 0.83 (83%)
 Pass rate is ratio between numbers test cases passed / test cases
executed. For example, in above 100 TCs executed, there’re 80 TCs that
passed, so the pass rate is 80/100 = 0.8 (80%)

This data can be retrieved in Test Metric documents.

 Run rate is mandatory to be 100% unless a clear reason is given.

 Pass rate is dependent on project scope, but achieving high pass rate is a
goal.

Step 5) Resource Planning

Resource plan is a detailed summary of all types of resources required to
complete project task. Resource could be human, equipment and materials
needed to complete a project

The resource planning is important factor of the test planning because helps
in determining the number of resources (employee, equipment…) to be used for
the project. Therefore, the Test Manager can make the correct schedule &
estimation for the project.

Step 7) Schedule & Estimation

In the article Test estimation, you already used some techniques to estimate the
effort to complete the project. Now you should include that estimation as well as
the schedule to the Test Planning

In the Test Estimation phase, suppose you break out the whole project into small
tasks and add the estimation for each task as below

Task Members Estimate

effort

Create the test Test Designer 170 man-hour

specification

Perform Test Execution Tester, Test 80 man-hour

Administrator

Test Report Tester 10 man-hour

 Test Delivery 20 man-hour

Total 280 man-

hour

Then you create the schedule to complete these tasks.

Making schedule is a common term in project management. By creating a solid

schedule in the Test Planning, the Test Manager can use it as tool for monitoring
the project progress, control the cost overruns.

To create the project schedule, the Test Manager needs several types of input as
below:

 Employee and project deadline: The working days, the project deadline,
resource availability are the factors which affected to the schedule
 Project estimation: Base on the estimation, the Test Manager knows how
long it takes to complete the project. So he can make the appropriate
project schedule
 Project Risk : Understanding the risk helps Test Manager add enough extra
time to the project schedule to deal with the risks

Step 8) Test Deliverables

Test Deliverables is a list of all the documents, tools and other components that
has to be developed and maintained in support of the testing effort.

There are different test deliverables at every phase of the software development
lifecycle.

Test deliverables are provided before testing phase.

 Test plans document.

 Test cases documents
 Test Design specifications.

Test deliverables are provided during the testing

  Test Scripts
 Simulators.
 Test Data
 Test Traceability Matrix
 Error logs and execution logs.

Test deliverables are provided after the testing cycles is over.

 Test Results/reports
 Defect Report
 Installation/ Test procedures guidelines
 Release notes

Test case and test plan are two important documents that play a significant role
in defining as well as managing the procedure of testing and offering the team
vital information regarding the testing of the software.

Test Case:
During the process of software testing, test case defines the preconditions for
test execution and offers an insight into the various specification and
requirements that are stated before the commencement of the testing. An
integral part of the process, this document is used to define various important
components of the software testing such as, testing inputs, execution
conditions, testing procedures, and the expected results. The main purpose of
this document is to help testers meet the objective of testing as well as the
requirements stated before the commencement of the process.

Types of Test Case:

1. Formal Test Case: For designing formal test cases, the team follows a
standard format, which allows them to create a test case that offers ease
of understanding to other members of the team and that cover all the
possible tests completely. Formal test cases are further divided into two
types of test cases- positive and negative, which are mainly designed to
ensure that all the requirements of the application are tested properly by
the team. Formal test case also offers various details about the software
such as preconditions, input & output data, among other things.
2. Informal Test Case: These test cases are mainly designed for applications
and software without any formal requirement or specification. Informal
test cases can be written in a manner that is considered fit by the team or
 by the individual responsible for creating test case. Moreover, in this type
of test case the input and output are now known to the team.

Importance of Test Case:

In software testing the relevance and importance of test case is monumental. It

is a vital document prepared during the early stages of software testing that
specifies the inputs, execution conditions, testing procedures, etc, of a testing
process. With the assistance of this document, the team can offer an insight into
the testing process to other members of the team as well as the client and the
concerned stakeholders. Test case allows them to establish transparency among
members and improves communication within the team. Other reasons that
make test case an integral part of software testing are:

 With the assistance of test cases testers can find defects and issues in
the test design, which is not covered by requirements.
 Helps identify usability issues, as the whole process is monitored.
 Enables the team to record the testing procedure and activities, which
can be helpful in future.
 It organizes the complete testing process.
 Helps validate if all requirements are fulfilled or not.
 It ensures accurate test coverage.

Details Covered by a Test Case:

A test case document offers miscellaneous information about the software

testing process. From the basic requirements to the testing procedure,
everything is covered in this document, which offers the team an opportunity to
communicate major details about the process with other members of the team
as well as the stakeholders. Other details included in this document are:

 Test title.
 Test scenarios.
 Test descriptions.
 Steps & procedure.
  Prerequisites.
 Actual & expected results.
 Test environment.

Things to Consider while Writing Test Cases:

As one of the most significant document prepared during the software testing
life cycle (STLC), the document for test case needs to be designed with great care
and accuracy. The details or information included in it should be described in a
manner that is understandable and precise. Hence, following are some factors
that need consideration while preparing the test case document.

 The customer requirements should be stated and defined clearly.

 It should be compact, simple and understandable, yet informative.
 Should offer details about the roles and responsibilities of the team
members.
 The document should be traceable, economical, and reusable.
 Any critical or important information related to the expected results
should be included.
 Should assure full test coverage.

Test Strategy
Test Strategy is a set of guidelines that explain the test design and determine
how testing needs to be done.

Example: A Test Strategy includes details like “Individual modules are to be

tested by the test team members”. In this case, who tests it does not matter –
so it’s generic and the change in the team member does not have to be
updated, keeping it static.

Test Strategy Document

The purpose of the test strategy is to define the testing approach, the types of
tests, test environments, and tools to be used for testing and the high-level
details of how the test strategy will be aligned with other processes.

Test strategy is intended for the complete project team that comprises of
Project Sponsors, Business SMEs, Application/ Integration Development,
System Integration partners, Data Conversion Teams, Build/Release
Management Teams such as technical leads, architecture leads, and
deployment and infrastructure teams.

Below are the important sections that a test strategy document should have:

#1) Project Overview

This section can begin by giving an overview of the organization followed by a
brief description of the project in hand. It can include below details

 What was the need for the project?

 What objectives the project will attain?
Table of Acronyms: It is better to include a table with acronyms that the
document reader might come up with while referring to the document.
#2) Requirements Scope
Requirement scope can include Application Scope and Functional scope

Application Scope defines the system under test and the impact on the system
due to new or changed functionality. Related systems can also be defined.
System Impact (New or Changed functionality) Related System

System A New enhancements and bug fixes • System B

• System C

Functional Scope defines the impact on different modules within the system.
Here each related system with respect to functionality will be explained.
System Module Functionality Related System

System C Module 1 Functionality 1 System B

Functionality 2 System C
#3) High-Level Test Plan
Test Plan is a separate document. In the test strategy, a high-level test plan can
be included. A high-level test plan can include test objectives and test scope.
Test scope should define both in scope and out of scope activities.

#4) Test Approach

This section describes the testing approach that will be followed during the
testing life cycle.

As per the above diagram testing will be conducted in two phases i.e. Test
Strategy & Planning and Test Execution. Test Strategy & Planning phase will be
one time for an overall program whereas Test execution phases will be
repeated for each Cycle of the overall program. The above diagram shows
different stages and deliverables (outcome) in each phase of the execution
approach.

Test approach should include below sub-sections

a) Test Schedule: Explain the proposed project timeline in this subsection
b) Functional Testing Approach: Using this subsection provides an overview of
each phase and the respective entry and exit criteria. Different testing phases
are Unit testing, System testing, System Integration testing, User Acceptance
Testing, and End-to-End testing.
c) Testing Key Performance Indicators :
  Test case prioritization: Define the test case prioritization approach so
that in case of time constraints, high priority scenarios can be executed
by the test team. There should be an agreement between project
stakeholders regarding the possible risks involved in not executing all the
planned scenarios.
 Defect prioritization: Defect prioritization strategy is the next topic to
cover here. Define priority level and give the description to each level
like critical, high, medium, etc. Also
 Defect Turnaround Time: Defect turnaround time is defined as the time
between when the defect was first raised and when the defect is fixed
and comes for a retest. Quick turnaround ensures speedy testing and
adherence to the project timeline. For each defect priority level, define
the turnaround time.
Priority Level Defect Turnaround Time

1 - Critical Response Time: 2 hours or less

Fix Ready for Migration: 1 business days or less

#5) Test Coverage

This section describes the processes that the QA team will follow in order to
optimize the coverage of business/functional requirements in test scenarios
and test cases. Requirement Traceability Matrix: (RTM) can be used to trace all
the requirements with respective test scenarios and test cases.
#6) Test Environment
Define the different QA environments available. Mention what testing will be
done in which environment and by whom. Create an environment backup plan
to take care of emergencies. Access to each environment should be regulated
and called out with clarity.

Testing tools that are going to be used also can be mentioned in this section.

Activity Tool Remarks

Test management HP ALM Mention the reason for using this tool

Defect management JIRA Mention the reason for using this tool
#7) QA Deliverables and Metrics
List out all the QA deliverables
S. No. Deliverable

1 Test Strategy Document

2 Requirement Traceability Matrix

3 ST Test Scripts

4 Test Summary Report

5 Automation eligible scenario list

List out all the QA metrics

Metric Reports in which
Metric Metric
# Metric Name unit of the metrics to be
Definition Formula
measure used

1 Requirements The coverage Ratio of # of % Weekly QA status

Coverage of requirements report,
Metrics(RCM) requirements tested to # of Test summary
by QA requirements report
identified

2 Test Coverage The coverage Ratio of % Daily

of test case number of Execution report,
executed test cases Weekly QA status
executed report,
/number of Test summary
test cases report
planned

#8) Defect Management

Clearly define a defect management strategy by creating a defect workflow,
defect tracking methodology & defect triage process. Mention defect
responsibility for each tester’s roles. Periodic defect analysis and root cause
analysis will improve the overall quality of testing

#9) Communication Management

Set guidelines for status reports, status meetings, and onsite-offshore
communication.

#10) Assumptions, Risks, and Dependencies

Describe assumptions on which the project is based. These may include timing,
resources, and system capabilities. Describe any dependencies such as other
projects, availability of temporary resources, other deadlines that may impact
the project

#11) Appendix
Include stuff like Roles & Responsibilities, Work Time Zone, and References in
this section

Test Plan Vs Test Strategy

TEST PLAN TEST STRATEGY

It is derived from software requirement It is derived from the Business

specification(SRS). Requirement document(BRS).

It is prepared by the test lead or It is developed by the project

manager. manager or the Business analyst.

Test plan id, features to be tested, test Objectives and scope,

techniques, testing tasks, features pass documentation formats, test
or fail criteria, test deliverables, processes, team reporting
responsibilities, and schedule, etc. are structure, client communication
the components of the test plan. strategy, etc. are the components
of test strategy.

If there is a new feature or a change in Test strategy maintains the

the requirement that is happened then standards while preparing the
the test plan document gets updated.
TEST PLAN TEST STRATEGY

document. It is also called as Static

document.

We can prepare the test plan In smaller projects, test strategy is

individually. often found as a section of a test
plan.

We can prepare a Test plan at the We can use Test strategy at

project level. multiple projects.

It describes how to test , when to test, It describes what type of technique

who will test and what to test. to follow and which module to test.

We can describe about the specifications Test strategy describes about the
by using a Test Plan. general approaches.

Test Plan will change over the course of Test Strategy usually will not
the project. change once approved.

Test plan is written after requirement Test strategy is made before the
sign off. test plan.

Test plans can be of different types. There will be only one test strategy
There will be a master test plan and document for a project.
separate test plan for different types of
testing like system test plan,
performance test plan, etc.

Test plan should be clear and concise. Test strategy provides overall
guidance for the project in hand.
 Types of Software testing
Software testing types are the approaches and techniques that
are applied at a given level using an appropriate method to
address the test requirements in the most efficient manner.
They are vast in number while serving different objectives.

Manual Testing
In manual testing, software testing is done manually without the use of automated tools or
applications available in the market. In this form of testing, the software tester tests or
checks for bugs like the end-user and checks the project for identifying any abnormal
behavior or bugs in it. Various diverse phases are there for testing a project manually. These
are:

 unit-testing
 integration testing
 system testing and
 user acceptance testing

Automation Testing
In automation testing (also termed as Software Test Automation), the software tester has to
write different scripts and apply other 3rd party software to test the software. Automation
testing involves the manual process done automatically. It is implemented for re-running
the test situations and states done manually and at the same time at a fast pace and
repeatedly performed to check whether any bug or error is left behind or not from the
previous test. This type of testing also deals with checking load, performance stats, CPU and
memory usage, and activities. Automation testing is more advantageous than manual
testing as it saves the tester's time and money
 Black Box Testing
This testing is also known as Behavioral Testing. The software tests the internal structural,
design, and implementation and UI and UX of the product being tested, which is unknown
to the tester. Black box testing is both functional or non-functional, but most of the time, it
is usually functional.
This testing technique is named black box because the software or the product is not known
or acknowledged in advance to the tester, and hence you can say the tester's eye is blind-
folded like a black box, and you can see nothing inside. This technique of testing tries to find
errors in these below-mentioned categories:

 Inaccurate or missing functioning of the software.

 Error in the interface.
 Errors in concepts and data structures implementation.
 Database related errors.
 Performance or behavioral errors.
 Errors in starting or termination of a product.

Levels of Black Box Testing

This testing technique is related to the below-mentioned software testing methodologies:

 Integration Testing
 System Testing
 Acceptance Testing

It is to be noted that the higher the level of testing is, the bigger and complex the box is to
test, and hence further black box testing comes into play.

White Box Testing

White Box testing is also termed as Open or transparent-box Testing or Glass-box testing.
Sometimes from a developer's point of view, it's known as Code-oriented Testing or
structural testing. This type of testing technique deals with testing the internal structure,
logic design, and implementation of different modules. Here, the tester uses his / her
preferred input or exercises paths via code to determine the proper or exact output. As it is
also termed as code-oriented testing, it contains technical tests and script-based testing as
part of its testing phase.

Levels of White Box Testing

This testing technique deals with the below-mentioned software testing methodologies:

 Unit Testing: Tests for paths inside a unit and module

 Integration Testing: Tests for paths among different units
 System Testing: Tests for paths among the sub-systems of a product

Gray Box Testing

In this software testing technique, it combines the concept of both Black box and white box
testing. In Grey box testing, the inside of your product is partly known to the tester. This has
partial access to data-structures residing internally for designing different test cases and
tests from a user's perspective or like a black-box tester. Grey box testing has its name grey
because it's like a semi-transparent box from a tester's eye, and combining the color of
black and white gives the shades of grey
 Levels of Testing
The four Levels of software testing goes from Unit testing towards Integration testing on the
following list:

 Unit Testing
 Integration Testing
 System Testing
 Acceptance Testing

1. Unit Testing: In this testing level, individual sections or parts of software or product are
tested. The idea of this is to confirm every part or unit of the product after the test.
2. Integration Testing: In this software testing level, individual parts need to combine and
test as a single cluster. This testing level's main idea is for exposing the faults while
interacting between integrated units of the project.
3. System Testing: In this software testing level, the whole, integrated software or project is
tested. The principle for this testing is to assess the system's conformity with its intended
requirements.
4. Acceptance Testing: At this software testing level, a system needs to be tested for
adequacy. This test is purposefully done for evaluating the compliance of the system with
business requirements.
What is Program Correctness?
Correctness from software engineering perspective can be defined as the adherence to the
specifications that determine how users can interact with the software and how the software
should behave when it is used correctly.
If the software behaves incorrectly, it might take considerable amount of time to achieve the
task or sometimes it is impossible to achieve it.

Important rules:
Below are some of the important rules for effective programming which are consequences
of the program correctness theory.
 Defining the problem completely.
 Develop the algorithm and then the program logic.
 Reuse the proved models as much as possible.
 Prove the correctness of algorithms during the design phase.
 Developers should pay attention to the clarity and simplicity of your program.
 Verifying each part of a program as soon as it is developed.

Testing Automation & Testing Tools

What is an Automated Software Testing?

Software Test automation makes use of specialized tools to control the execution of tests
and compares the actual results against the expected result. Usually regression tests, which
are repetitive actions, are automated.
Testing Tools not only help us to perform regression tests but also helps us to automate data
set up generation, product installation, GUI interaction, defect logging, etc.

Criteria for Tool Selection:

For automating any application, the following parameters should be considered.
 Data driven capabilities
 Debugging and logging capabilities
 Platform independence
  Extensibility & Customizability
 E-mail Notifications
 Version control friendly
 Support unattended test runs

Types of Frameworks:
Typically, there are 4 test automation frameworks that are adopted while automating the
applications.
 Data Driven Automation Framework
 Keyword Driven Automation Framework
 Modular Automation Framework
 Hybrid Automation Framework

Popular Tools that are used for Functional automation:

Product Vendor URL

Quick Test Professional HP www.hp.com/go/qtp

Rational Robot IBM http://www-

03.ibm.com/software/products/us/en/robot/

Coded UI Microsoft http://msdn.microsoft.com/en-

us/library/dd286726.aspx

Selenium Open Source http://docs.seleniumhq.org/

Auto IT Open Source http://www.autoitscript.com/site/

Popular Tools that are used for Non-Functional automation:

Product Vendor URL

Load Runner HP www.hp.com/go/LoadRunner

Jmeter Apache jmeter.apache.org/

Burp Suite PortSwigger http://portswigger.net/burp/

Acunetix Acunetix http://www.acunetix.com/