ADVANCED DIPLOMA in
BA
SOFTWARE TESTING
SESSION 12
Session 12
1
OVERVIEW
INTRODUCTION
FUNDAMENTALS
TESTING LIFECYCLE
STATIC & DYNAMIC TESTING
TEST DESIGN TECHNIQUES (White Box)
MANUAL TEST CASES
TOOL SUPPORT & TEST MANAGEMENT
AUTOMATED SOFTWARE TESTING
Session 12
2 2
Software Testing
AUTOMATED SOFTWARE TESTING
Session 12
3
Objectives
• Need for automating testing activities
• Testing tools are not silver bullets that will solve all
testing problems
• Static and dynamic testing tools (recap)
• Selection of a testing tool
• Costs incurred in adopting a testing tool
• Guidelines for automating testing
Session 12
4 4
Content (1)
Foundational Concepts:
What is automated testing?
Why automated testing?
When to automate?
Automated Testing Lifecycle
Methods used in Automated Testing
Testing Tools & Frameworks
Session 12
5 5
What is automated software testing?
A process that covers:
• the use of software to control the execution of tests
• the comparison of actual outcomes to predicted
outcomes
• the setting up of test preconditions
• other test control (entering input data) and test
reporting functions.
Session 12
6 6
Why testing is critical?
Session 12
7 7
Why automated testing?
• Time savings translates directly into cost savings
• Improves testing productivity
• Improves accuracy
• Increases test coverage
• Does what manual testing cannot do
Session 12
8 8
Effort and cost
– Let’s assume 6 test cases
– Effort required to run all 6 manually => 10 min
– Effort required to write unit tests for all 6 cases => 10 min
– Effort required to run unit tests for all 6 cases => < 1 min
– Number of testing iterations => 5
– Total manual testing time => 50 min
– Total unit testing time => 10 min
Manual Test
Release Manual Test Auto Test Cumulative
1 10 10 10
2 10 0 20
3 10 0 30
4 10 0 40
5 10 0 50
Session 12
9 9
Size of the testing problem
• Consider inputting one value in a 10 character field
• 26 uppercase, 26 lowercase, 10 numbers
• Gives 6210 combinations
• How long to test all assuming 1 million cases per
second?
• So realistically, we can only test a small portion
Session 12
10 10
Need for automation
• Reduction of testing effort
• Reduces the testers’ involvement in executing tests
• Facilitates regression testing
• Avoids human mistakes
• Reduces overall cost of the software
• Simulated testing
• Internal testing
• Test enablers
• Test case design
Session 12
11 11
Benefits of Automation (1)
• Production of a reliable system
• Improved requirements definition
• Improved performance testing
• Improved load/stress testing
• Improved system development lifecycle
• Quality measurement and test optimization
• Improved partnership with development team
Session 12
12
Benefits of Automation (2)
• Improved quality of test effort
• Improved build verification testing (smoke testing)
• Improved regression testing
• Improved multi-platform compatibility testing
• Improved software compatibility testing
• Execution of tests that manual testing cannot accomplish
• Improved execution of mundane tests
• Improved focus on advanced test issues
• Ability to reproduce software defects
• Enhancement of business expertise
• After-hours testing
Session 12
13
Benefits of Automation (3)
• Reduction of test effort and
minimization of schedule
• Test plan development: > test effort
• Test procedure development: < test effort
• Test execution: < test effort / schedule
• Test results analysis : < test effort / schedule
• Error status/correction monitoring: < test effort / schedule
• Report creation: < test effort / schedule
Session 12
14
Things to consider about automation
• Don’t expect to be more productive over the short term
• Not possible to automate 100% of testing – still need manual testing
• Automation is software development (i.e.
programming)
• Automation need not be only regression--you can
run new tests instead of old ones
• Maintainability is essential
• Design to your requirements
• Set management expectations with care
Session 12
15 15
Automated vs. manual testing
• Not all tests can be automated and most
times is difficult to decide what to automate
and what to test manually
Session 12
16 16
Strengths of each
Automated Testing Manual Testing
• Need to run a set of tests • Test cases that need to be run a
repeatedly small number of times
• Helps performing "compatibility • Allows the tester to perform
testing" (on different more specific tests
configurations and platforms)
•Long term costs are reduced • Short term testing costs are
reduced
• Possible to run regressions on a • The more time tester spends
code that is continuously testing a module the greater
changing and in shorter time chance of finding real bugs
Session 12
17 17
Downfalls & other factors to consider
Automated Testing Manual Testing
• It is more expensive to • Manual tests can be very
automate (bigger initial time consuming
investments)
• You cannot automate • For every release you must
everything, some tests still rerun the same set of tests
have to be done manually which can be tiresome
Other Factors
• The performance of your test tools
• The knowledge level of your testing team
• The continuous growth of software to be tested
Session 12
18 18
An excellent test case
• Reasonable probability of catching an error
• Not redundant with other tests
• Exercise to stress the area of interest
• Minimal use of other areas (independence)
• Neither too simple nor too complex
• Makes failures obvious
• Allows isolation and identification of errors
Session 12
19 19
Good test case design:
Neither too simple nor too complex
– What makes test cases simple or complex?
(A simple test manipulates one variable at a time)
– Advantages of simplicity?
– Advantages of complexity?
– Transition from simple cases to complex cases
(You should increase the power and complexity of tests over time)
– Automation tools can bias your development
toward overly simple or complex tests
Session 12
20 20
Test cases suitable for automation
• Following criteria used to increase the
automation ROI
– High Risk - business critical test cases
– Test cases that are executed repeatedly
– Test cases that are very tedious or difficult to
perform manually
– Test cases that are time consuming
Session 12
21 21
Test cases not suitable for automation
• Test cases that are newly designed and have not
been executed manually at least once
• Test cases for which the requirements are
changing frequently
• Test cases that are executed on ad-hoc basis
• Tests not supported by test tool functionality
Session 12
22 22
Content (2)
Foundational Concepts: What, why, how, …
Automated Testing Lifecycle
Methods used in Automated Testing
Testing Tools & Frameworks
Session 12
23 23
Automated testing life-cycle
Define Scope of Automation
Test Tool Selection
Planning, Design, Development
Test Execution
Test Review & Maintenance
Session 12
24 24
1. Define scope of automation
• Scope of automation is the area of your “System
Under Test” that will be automated.
• Following points help determine scope:
– Features that are important for the business
– Scenarios that have large amounts of data
– Common functionalities across applications
– Technical feasibility
– Extent to which business components are reused
– Complexity of test cases
– Ability to use the same test cases for cross browser
testing
Session 12
25 25
2. Test tool selection
• If the only tool you have is a hammer, you tend
to see every problem as a nail
Abraham Maslow
• Not the case with test automation tools
Session 12
26 26
Test development tool selection
• Largely depends on the technology the
“System under Test” (SUT) is built on
• For example, mobile or web or standalone
applications
• It's a good idea to conduct Proof of
Concept of Tool on SUT
• Other criteria include: cost, skillset
requirements, …
Session 12
27 27
3. Planning, design, development
• Automation strategy & plan is created, containing:
– Automation tools selected
– Framework design and its features
– In- and out-of-scope items of automation defined
– Automation test bed preparation
– Schedule and timeline of scripting and execution
– Deliverables of automation testing
• Design covers number of tests, ways test will be
approached (paths or functions), test conditions
• Test scripts created following test development
standards
Session 12
28 28
4. Test execution
• Automation scripts are executed during this phase.
• Scripts need input test data before there are set to run.
• Scripts can be executed on a single machine or a group
of machines.
• Execution can be done 24/7 to save time.
• Detailed test reports produced by execution of scripts.
• Execution can be performed using the automation tool
directly or through a Test Management tool which will
invoke the automation tool.
• Example: Quality Center is the Test Management tool
which in turn will invoke QTP for execution of automation
scripts.
Session 12
29 29
5. Test review and maintenance
• Metrics evaluated
• Final review and assessment conducted to allow
for process improvement
• As new functionalities are added to the SUT
with successive cycles, automation scripts need
to be added, reviewed and maintained for each
release cycle.
• Maintenance becomes necessary to improve
effectiveness of automation scripts.
Session 12
30 30
Content (3)
Foundational Concepts: What, why, how, …
Automated Testing Lifecycle
Methods used in Automated Testing
Testing Tools & Frameworks
Session 12
31 31
Common types of automated testing
methods
• Monkey testing
• Capture / playback
• Code-based (unit) testing
• Intelligent test automation
Session 12
32 32
Monkey testing
• Randomly selecting inputs from a large
range of values and monitoring if
exceptions are thrown
• For example, a monkey test can enter
random strings into text boxes to ensure
handling of all possible user input
• Applies not only for GUI or web testing, but
also for unit testing
Session 12
33 33
Tool example
Monkey – Android testing tool
The simplest way to use the monkey is with the
following command, which will launch the application
and send 500 pseudo-random events to it.
$ adb shell monkey [options] <event-count>
$ adb shell monkey -v -p [Link] 500
Session 12
34 34
Capture / playback
• Set of software programs that capture user inputs
and store these into a script for later replay
+ Repeated testing can be performed quickly
+ Does not require programming skills
- When the GUI changes, input sequences
previously recorded may no longer be valid
- Difficult to determine location of bugs
Session 12
35 35
Tool example
Autohotkey - Free keyboard macro program
Supports hotkeys for keyboard, mouse
Session 12
36 36
Code-based (unit) testing
• Individual units of source code are tested to
determine if they are fit for use
• Ideally, each test case is independent from the
others
+ Bugs can be found in early development stage
+ Easy to test boundary cases
- Not effective for integrated system testing
Session 12
37 37
Unit test-case generation method
• Random
• Genetic
• Other…
Session 12
38 38
Random
• Input values generated randomly
• Run with generated inputs
• User checks output values (pass/fail) and saves them
for later automatic checking (oracle)
Session 12
39 39
Genetic
This methods approach is to combine JML (JAVA model
language) and genetic algorithms. JUnit is used as a
test execution platform.
Session 12
40 40
Example tools
Parasoft Jtest is an automated Java testing and
static code analysis product
TestGen is a
collection of
open-source
tools that
automatically
generate unit
test cases
Session 12
41 41
Intelligent test automation
• A model-based method for generating tests from a
description of an application’s behavior (UML)
+ Models are simple to modify, so model-based tests are
economical to maintain over the life of a project
- Most of these methods are researched only
theoretically.
- In future likely to see machine learning involved as
well.
Session 12
42 42
OCL + UML
Inputs Constraints
Inputs generator
Tested software UML + OCL
Oracle
Outputs Constraints
pass / fail
Report
Session 12
43 43
Content (4)
Foundational Concepts: What, why, how, …
Automated Testing Lifecycle
Methods used in Automated Testing
Testing Tools & Frameworks
Session 12
44 44
Automation testing tools
• There are tons of Functional and Regression
Testing Tools available in market.
• We look at six of the popular tools that
have been around for a while
Session 12
45 45
Selenium
• It is a software testing tool used for regression testing. It is an open
source testing tool that provides playback and recording facility for
regression testing.
• The Selenium IDE only supports Mozilla Firebox web browser.
• Recorded script can be exported to other languages like Java, Ruby,
RSpec, Python, C#, Junit and TestNG
• It can execute multiple tests at a time
• Autocomplete available for common Selenium commands
• Walkthrough tests
• Identifies elements using id, name , X-path, etc.
• Store tests as Ruby Script, HTML, or any other format
• It provides an option to assert the title for every page
• It supports selenium [Link] file
• It allows inserting comments in the middle of the script for better
understanding and debugging
Session 12
46 46
Katalon Studio
• Based on Selenium and Appium frameworks, it
is an open-source tool for testing mobile and
web applications.
• Tool also supports API testing on different
operating systems.
• It has a user-friendly IDE, supports object spy
and object repository and browser plugins.
• Can be integrated with GIT, Jira and Jenkins.
• The tests are generated automatically by using
built-in keywords.
Session 12
47 47
QTP (HP Unified functional testing)
• Widely used for functional and regression testing, it addresses every
major software application and environment.
• To simplify test creation and maintenance, uses the concept of
keyword driven testing. It allows the tester to build test cases directly
from the application.
• It is easier to use for non-technical person to adapt to and create
working test cases
• It fixes defects faster by thoroughly documenting and replicating
defects for developer
• Collapses test creation and test documentation at a single site
• Parameterization is easier than WinRunner
• QTP supports .NET development environment
• It has better object identification mechanism
• It can enhance existing QTP scripts without “System Under Test" being
available, by using the ActiveScreen
• Requires a licence
Session 12
48 48
IBM Rational Functional Tester
• Requires a licence
• Object-Oriented automated functional testing tool capable of performing
automated functional, regression, data-driven testing and GUI testing.
• It supports a wide range of protocols and applications like Java, HTML, NET,
Windows, SAP, Visual basic, etc.
• It can record and replay the actions on demand
• It integrates well with source control management tools such as Rational Clear
Case and Rational Team Concert integration
• It allows developers to create keyword associated scripts that can be reused
• Eclipse Java Developer Toolkit editor facilitates coding test scripts in Java with
Eclipse
• It supports custom controls through proxy SDK (Java/.Net)
• It supports version control to enable parallel development of test scripts and
concurrent usage by geographically distributed team
Session 12
49 49
WATIR
• Open source testing software for regression testing. It
enables writing tests that are easy to read and maintain.
• Watir supports only Internet Explorer on windows while
Watir webdriver supports Chrome, Firefox, IE, Opera, etc.
Runs on Linux!
• It supports multiple browsers on different platforms
• Rather than using proprietary vendorscript it uses the full
featured modern scripting language Ruby
• It supports all web apps regardless of what they are
developed in
Session 12
50 50
Silk Test
• Silk Test is designed for doing functional and regression testing.
• For e-business application, silk test is the leading functional testing
product designed by Segue Software.
• It is an object oriented language just like C++ -- uses concept of object,
classes, and inheritance.
• It consists of all the source script files
• It converts the script commands into GUI commands. On the same
machine, commands can be run on a remote or host machine
• To identify the movement of mouse along with keystrokes, Silktest can
be executed. It can avail both playback and record method or
descriptive programming methods to get the dialogs
• It identifies all controls and windows of the application under test as
objects and determine all of the attributes and properties of each
window
• Open-source and licenced
Session 12
51 51
Tool examples
Session 12
52 52
Compulsory video
• Selecting test tools – best tool (6 min)
[Link]
1GOo
Session 12
53 53
Video and reading (not compulsory)
• 6 Open Source Test Automation Frameworks
You Need to Know
[Link]
F6qY
Top Automation Testing Frameworks (2025)
[Link]
648
Session 12
54 54
