Content

Types of Software Testing

• Unit Testing
• Integration Testing
• System Testing
• Smoke & Sanity Testing
• Acceptance Testing
• Regression Testing
• Testing in DevOps
Unit Testing
 Unit Testing
• Objective of unit testing is to isolate a section of code and verify its
correctness.

• Goal of unit testing is to isolate each part of the program and show that the
individual parts are correct.

• Done during the development (coding) of an application. Usually performed by

the developer.

• Skimping on unit testing leads to higher defect fixing costs during system testing
or integration testing.

• Proper unit testing done during the development stage saves both time and
money in the end. It can be automated or performed manually.

• Relies on mock objects – stubs and drivers - being created to test sections of
code that are not yet part of a complete application.

• stubs and drivers do not implement the entire programming logic of the software
module but just simulate data communication with the calling module.
 Automated Unit Testing
• A developer could write another section of code in the application just to test
the function - then later comment out and finally remove the test code when
the application is done.

• They could also isolate the function to test it more rigorously by copying and
pasting the function to its own testing environment e.g., a Unit Test
Framework

• During the execution of the test cases, the framework logs those that fail any
criterion.

• Many frameworks will also automatically flag and report a summary of these
failed test cases.

• Isolating the code also helps in revealing unnecessary dependencies between

the code being tested and other units or data spaces in the product.
 Unit Testing & Extreme Programming
• In Extreme Programming, unit testing involves the extensive use of testing
frameworks in order to create automated unit tests.

Under Extreme programming:

i. Tests are written before the code

ii. Rely heavily on testing frameworks
iii. All classes in the applications are tested
iv. Quick and easy integration is made possible

• Programmers think that integration testing will catch all errors and not a unit
test.

• Once units are integrated, very simple errors which could have been very easily
found and fixed in the unit tested take a very long time to be traced and fixed.
 Unit Testing Cont…
Benefits of Unit Testing

• developers looking to learn what functionality is provided by a unit and how

to use it can look at the unit tests to gain a basic understanding of the unit
API.
• unit testing allows the programmer to refactor code at a later date, and make
sure the module still works correctly (i.e., Regression testing).

• the procedure is to write test cases for all functions and methods so that
whenever a change causes a fault, it can be quickly identified and fixed.

• due to the modular nature of the unit testing, we can test parts of the project
without waiting for others to be completed.

Limitations of Unit Testing:

• Unit testing can't be expected to catch every error in a program unit testing
by its very nature focuses on a unit of code - can't catch integration errors or
broad system level errors.
• It's recommended unit testing be used in conjunction with other testing
activities.
 Unit Testing Cont…

Best Practices of Unit Testing

• unit test cases should be independent. In case of any enhancements or

changes in requirements, unit test cases should not be affected. test only one
code at a time.

• follow clear and consistent naming conventions for your unit tests in case of
change in code in any module, ensure there is a corresponding unit test case
for the module and the module passes the tests bugs identified during unit
testing must be fixed before proceeding to the next phase in SDLC.

• adopt a "test as your code" approach - the more code you write without testing
the more paths you have to check for errors.
Integration Testing
 Integration Testing
• individual software modules are integrated logically and tested as a group.

• a typical software project consists of multiple software modules, coded by

different programmers - integration testing focuses on checking data
communication amongst these modules.

Although each software module is unit tested, defects still exist for various
reasons like :

a module in general is designed by an individual software developer whose

understanding and programming logic may differ from other programmers

integration testing becomes necessary to verify the software modules work in unity
at the time of module development, there are wide chances of change in
requirements by the clients - these new requirements may not be unit tested, and
hence integration testing becomes necessary.

interfaces of the software modules with the database could be erroneous

external Hardware interfaces, if any, could be erroneous `

inadequate exception handling could cause issues

 Integration Testing Cont…
• Integration Test Case
it focuses mainly on the interfaces & flow of data/information between the
modules.

priority is given for the integrating links rather than the unit functions which are
already tested.

For example: application has 3 modules: 'Login Page', 'Mailbox' and 'Delete
mails' and each of them are integrated logically.
 Integration Testing Cont…

Approaches/Strategies of Integration Testing:

1. Big Bang Approach

2. Incremental Approach
i. Top Down Approach
ii. Bottom-Up Approach
iii. Sandwich Approach - Combination of Top Down and Bottom Up

1. Big Bang Approach

-All components are integrated together at once, and then tested.

Advantages

-Convenient for small systems.

 Integration Testing Cont…

1. Big Bang Approach cont…

Disadvantages

▪ fault localization is difficult.

▪ given the sheer number of interfaces that need to be tested in this approach,
some interface links to be tested could be missed easily.
▪ since the integration testing can commence only after "all" the modules are
designed, the testing team will have less time for execution in the testing
phase.
▪ since all modules are tested at once, high-risk critical modules are not
isolated and tested on priority
 Integration Testing Cont…
2. Incremental Approach

▪ testing is done by joining two or more modules that are logically related.
▪ the process uses stubs and drivers.

i. Bottom-up Integration:

▪ Each module at lower levels is tested with higher modules until all modules
are tested.
▪ Uses drivers for testing
 Integration Testing Cont…
Bottom-up Integration cont..

Advantages:

▪ fault localization is easier.

▪ no time is wasted waiting for all modules to be developed unlike the Big-
bang approach

Disadvantages:

▪ critical modules (at the top level of software architecture) which control the
flow of applications are tested last and may be prone to defects.
▪ An early prototype is not possible
 Integration Testing Cont…
ii. Top down Integration:

- testing takes place from top to down following the control flow of the
software system.
- uses stubs for testing

Advantages:

• fault localization is easier.

• possibility to obtain an early prototype.
• critical modules are tested on priority - major design flaws could be found
and fixed first

Disadvantages:

• needs many stubs.

• modules at lower level may be tested inadequately.
 Integration Testing Cont…
Integration Testing Procedure:

Irrespective of the test strategies:

• Prepare the Integration Test Plan

• Design the Test Scenarios, Cases, and Scripts.
• Executing the test Cases followed by reporting the defects.
• Tracking & re-testing the defects.
• Steps 3 and 4 are repeated until the completion of Integration is successfully

Integration Test Plans:

include the following attributes:

• Methods/Approaches to testing (as discussed above).

• Scopes and Out of Scopes Items of Integration Testing.
• Roles and Responsibilities.
• Pre-requisites for Integration testing.
• Testing environment.
• Risk and Mitigation Plans
 Integration Testing Cont…
Integration Testing – Entry and Exit Criteria:

Entry Criteria

• unit tested components/modules

• all high prioritized bugs fixed and closed
• all modules to be code completed and integrated successfully.
• integration test plan, test case, and scenarios to be signed off and
documented.
• required test environment to be set up for Integration testing

Exit Criteria

• successful testing of integrated application.

• executed test cases are documented
• all high prioritized bugs were fixed and closed
• technical documents to be submitted followed by release notes.
 Integration Testing Cont…
Best Practice for Integrating Testing

• First determine the integration test strategy that could be adopted and later
prepare the test cases and test data accordingly.

• Study the architecture design of the application and identify the critical
modules - these need to be tested on priority.

• Obtain the interface designs from the architectural team and create test
cases to verify all of the interfaces in detail. Interface to database/external
hardware/software application must be tested in detail.

• After the test cases, it's the test data that plays a critical role.

• Always have the mock data prepared, prior to executing - do not select test
data while executing the test cases.
System Testing
 System testing

• The testing of a complete and fully integrated software product.

• A series of different tests whose sole purpose is to exercise the full computer-
based system.

• Falls under the black box testing category of software testing.

• White box testing - testing of the internal workings or code of a software

application - black box or system testing is the opposite.

• Need to build detailed test cases and test suites that test each aspect of the
application as seen from the outside without looking at the actual source code.

• Involves the external workings of the software from the user's perspective.
 System testing Cont…

Involves testing the software code for the following:

• testing the fully integrated applications including external peripherals in order to

check how components interact with one another and with the system as a
whole - also called End to End scenario testing.

• verify thorough testing of every input in the application to check for desired
outputs.

• testing of the user's experience with the application.

 System testing Cont…
Software Testing Hierarchy:

Unit testing

testing performed on each module or block of code during development.

normally done by the programmer who writes the code.

Integration testing
testing done before, during and after integration of a new module into the
main software package.
involves testing of each individual code module.

System testing

testing done by a professional testing team on the completed software

product before it is released.

Acceptance testing

beta testing of the product done by the actual end users.

 System testing Cont…
Types of Software Testing:

Usability Testing
mainly focuses on the user's ease to use the application, flexibility in
handling controls and ability of the system to meet its objectives

Load Testing
necessary to know that a software solution will perform under real life
loads.

Regression Testing
involves testing done to make sure none of the changes made over the
course of the development process have caused new bugs.

also makes sure no old bugs appear from the addition of new software
modules over time.

Recovery Testing
done to demonstrate a software solution is reliable, trustworthy and can
successfully recoup from possible crashes.
 System testing Cont…

Migration Testing
done to ensure that the software can be moved from older system
infrastructures to current system infrastructures without any issues.

Functional Testing
also known as functional completeness testing, functional testing involves
trying to think of any possible missing functions.

Hardware/Software Testing
IBM refers to Hardware/Software testing as "HW/SW Testing".

This is when the tester focuses his/her attention on the interactions

between the hardware and software during system testing
Smoke and Sanity Testing
 Smoke and Sanity Testing
Software Build:

• a typical software project consists of hundreds or even thousands of source

code files.
• creating an executable program from these source files is a complicated and
time-consuming task. - need to use "build" software to create an executable
program and the process is called "Software Build“.

Smoke Testing:

• performed after software build to ascertain that the critical functionalities of

the program are working fine.

• objective is not to perform exhaustive testing, but to verify that the critical
functionalities of the system are working fine.

• executed "before" any detailed functional or regression tests are executed

on the software build.

• purpose is to reject a badly broken application so that the QA team does not
waste time installing and testing the software application.
 Smoke and Sanity Testing cont…
For Example:

• Verify that the application launches successfully,

• Or check that the GUI is responsive

Sanity Testing:

• after receiving a software build, with minor changes in code, or functionality,

Sanity testing is performed to ascertain that the bugs have been fixed and no
further issues are introduced due to these changes.

• goal is to determine that the proposed functionality works roughly as

expected.
• if the sanity test fails, the build is rejected to save the time and costs involved
in more rigorous testing.
• objective is "not" to verify thoroughly the new functionality but to determine
that the developer has applied some rationality (sanity) while producing the
software.
Comparison: Smoke vs Sanity Testing
 Smoke and Sanity Testing Cont…
Pointers to Note:

• Both sanity tests and smoke tests are ways to avoid wasting time and effort
by quickly determining whether an application is too flawed to merit any
rigorous testing.

• Sanity testing is also called tester acceptance testing.

• smoke testing performed on a particular build is also known as a build

verification test.

• One of the best industry practices is to conduct a daily build and smoke test in
software projects.

• Both smoke and sanity tests can be executed manually or using an automation
tool. When automated tools are used, the tests are often initiated by the same
process that generates the build itself.

• As per the needs of testing, you may have to execute both Sanity and Smoke
Tests on the software build. In such cases, you will first execute Smoke tests
and then go ahead with Sanity Testing.
Acceptance Testing
 Acceptance Testing

• Also known as user testing

• All units in the system are combined into the final program/application

• Ensure that the system works how the user expects, i.e., it meets the user
specifications for functionality.

Three major objectives of acceptance testing:

• Confirm that the system meets the agreed-upon criteria.

• Identify and resolve discrepancies, if there are any.

• Determine the readiness of the system for cut-over to live operations

Regression Testing
 Regression Testing

• Purpose is to confirm that a recent program or code change has not

adversely affected existing features.

• Is nothing but a full or partial selection of already executed test cases that
are re-executed to ensure existing functionalities work fine.

• this testing is done to make sure that new code changes should not have
side effects on the existing functionalities.
• ensures that old code still works once the new code changes are done

RT is required when there is a:

• change in requirements and code is modified according to the requirement

• new feature is added to the software
• defect fixing
• performance issue fix
 Regression Testing Cont..
Regression Testing Technique:

• Software maintenance includes enhancements, error corrections,

optimization, and deletion of existing features.
• these modifications may cause the system to work incorrectly.
• regression testing becomes necessary.
 Regression Testing Cont..
Retest All

• all the tests in the existing test bucket or suite should be re-executed.
• very expensive as it requires a lot of time and resources.

Regression Test Selection

• instead of re-executing the entire test suite, it is better to select part of a test
suite to be run
• test cases selected can be categorized as:
-Reusable Test Cases - can be used in succeeding regression cycles.
-Obsolete Test Cases - can't be used in succeeding cycles

Prioritization of Test Cases

• prioritize the test cases depending on business impact, critical & frequently
used functionalities.
• selection of test cases based on priority will greatly reduce the regression
test suite.
 Regression Testing Cont..
Difference between Re-testing and Regression Testing

• Retesting means testing the functionality or bug again to ensure the code is
fixed. If it is not fixed, the defect needs to be reopened. If fixed, the defect is
closed.

• Regression testing means testing your software application when it

undergoes a code change to ensure that the new code has not affected
other parts of the software.

Challenges in regression Testing

major testing problems for doing regression testing:

• With successive regression runs, test suites become fairly large - due to time
and budget constraints, the entire regression test suite cannot be executed
• Minimizing the test suite while achieving maximum test coverage remains a
challenge
• Determination of the frequency of Regression Tests, i.e., after every
modification or every build update or after a bunch of bug fixes, is a
challenge.
DevOps
DevOps is a CULTURAL and
OPERATIONAL model that
fosters COLLABORATION to
ENABLE high-performance IT to
ACHIEVE business goals

Source: DevOps Fundamentals® Manual - DASA Limited

Automation in DevOps
 What to Automate?
Automate Provisioning -
Infrastructure as Code
Continuous Integration - Builds
Defined Deployment Pipeline and
Continuous Deployments
Continuous Testing, Automated
tests after each deployment
Proper monitors in place sending
alerts
Roles in DevOps

Source: DevOps Fundamentals® Manual - DASA Limited

DevOps on Testing

Fast feedback on the code quality

CI promotes a shift-left approach in testing
Promotes automated processes like CI/CD that facilitate
establishing stable test environments
Increases the view on non-functional quality characteristics
Automation through a delivery pipeline reduces the need for
repetitive manual testing
The risk in regression is minimized due to the scale and
range of automated regression tests
 ST Key Terminology
Test Case: a set of inputs, execution conditions, and expected results
developed for a particular objective

Test Suite: a collection of test cases, typically related by a testing goal or an

implementation dependency

Test Driver: class or utility program that applies test cases

Test harness: a system of test drivers and other tools that support test
execution

Test Strategy: algorithm or heuristic to create test cases from a representation,

implementation, or a test model

Oracle: means to check the output from a program is correct for the given input

Stub: partial temporary implementation of a component.

CI/CD: Continuous Integration/ Continuous Delivery

• Test Case Development (Learning Unit 3)