Load Testing 101:

Essential Tips for Testers

and Developers
 LoadComplete is a desktop tool for load, stress,
scalability testing of websites and web applications.
Designed from the ground-up to help you focus
on the tasks at hand, it does not require advanced
programming skills.

LEARN MORE ABOUT LOADCOMPLETE

Content
Preface 4

Load Testing Basics 6

Goals of Load Testing 8
Preparing your load tests 9
Making it real:
Emulating real life conditions in your load tests 14
The Dos and Donts of Load Testing 17
About LoadComplete 19
By the time any software development and improvement. As websites and web

development project nears applications become modern and complex, load test-
ing becomes challenging. Though it has been around
completion, it likely will have gone
for years, we still see well known mobile and web ap-
through numerous tests, particularly plications getting overwhelmed with peak traffic.
in an Agile environment where
testing and development happen The other parallel change is about the place of testing
concurrently. But no matter how many tests in software development. With shift-left and DevOps
youve run, once your application is nearly motion, software teams are encouraged to test more
complete, theres really only one way to know and test often. As a result, software testing is becom-
whether or not your software can handle the ing a specialized career on par with developers them-
actual demands your army of end users will selves. Testing teams and individual testers are now
soon be placing on it load testing. critical players that need to be valued and better un-
derstood by anyone involved in software.
Load testing is the process of putting simulated demand
on software, an application, or website in a way that We can trace this move to three main trends:
demonstrates its behavior under various conditions. the rising popularity of Agile/DevOps approaches

the interest in collaborative coding as a means of soft-

Over the last decade, the importance of load testing has
ware innovation
skyrocketed. What was once a simple pre-deployment ex-
ercise to ensure a web application could handle the load the increasing use of software in all industries and
of multiple users has become an intrinsic part of software critical domains

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 4

As a result, load testing needs to be done more frequent-
Well cover:
ly, more effectively, and more efficiently. Its also created
Load testing basics
a need to simply train more people in the basics of load
testing. Given that our lives increasingly rest on software How to prepare for load testing
functioning properly whether its in medical devices,
transportation, communications, defense, or entertain- Emulating real life conditions in your load tests
ment software performance has never been more
important. Load testing dos and donts

Thats what this eBook is ultimately about a Load Test-

ing 101 manual to get the new and aspiring testers start-
ed with load testing.

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 5

 Stress testing helps you find your applications weak
Load Testing Basics points, and how it behaves under extreme load condi-
tions.
Load Testing is a type of performance testing. (Other two
types being stress testing and capacity testing). Load
Capacity testing is complementary to load testing and
testing is used to verify your applications behavior un-
determines your servers ultimate failure point, where-
der normal and peak load conditions. This allows you to
as load testing monitors results at various levels of
verify that your application can meet the desired perfor- load and traffic patterns.
mance objectives; which are often specified in a service
level agreement (SLA). Load testing also enables you As just one example, lets say youre developing a
to measure response times, throughput rates, resource new online voting platform, and youd like it to be able
utilization levels, and to identify your applications break- to handle potentially up to 10,000 user submissions
ing point, assuming that breaking point occurs below the per minute during peak load times. While developing
peak load condition. Load testing helps you check your the software, you may have performed unit tests as
web servers performance under a massive load, deter- the code was being written, plus periodic regression
mine its robustness, and estimate its scalability. tests to make sure you didnt break existing func-
tionality with each new modification as development
Stress testing enables you to evaluate your applications progressed, but at what point did you begin testing for
behavior when it is pushed beyond the normal or peak multiple users?
load conditions. The purpose of web server stress testing
is to find the target applications crash point. The crash At what point did you begin testing the program to ac-
point is not always an error message or access violation. cept hundreds or even thousands of overlapping field
It can be a perceptible slowdown in the request process- entries, form submissions, and other commands? This
ing. The goal of stress testing is to unearth application is when load testing comes into the picture.
bugs that surface only under high load conditions.

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 6

 LOAD TESTING USUALLY INVOLVES THE FOLLOWING STEPS:

LITERATE

Identify key Identify Identify Create test Simulate Analyze

scenarios workload Metrics cases load results

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 7

Goals of Load Testing Symptoms and causes of application failure under
stress conditions
The main goals of load testing are to: Weak points in your application
Identify bottlenecks and their causes What is required to support a projected increase
in load
Optimize the application configuration (both the hardware
and software) for maximum performance.
This eBook keeps the abovementioned steps in
Verify the reliability of your application under stress mind and uses our load testing tool LoadComplete
to help you understand basics of Load testing. Well
Load/stress testing helps you identify the following charac- approach load testing with a four-step process:
teristics:

Response time Prepare: to prepare your web application

for load testing
Throughput
Record: How to record load testing scenarios
Maximum concurrent users supported
Test: Creating tests that match real-life
Resource utilization in terms of the amount of CPU, RAM, circumstances
network I/O, and disk I/O resources your application con-
Analyze: Understanding and using load testing
sumes during the test
data
B
ehavior under various workload patterns including nor-
mal load conditions, excessive load conditions, and condi-
tions in between

A
pplication breaking point
LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 8
 different types of tests based on what you hope to
Preparing your load tests find out.

As we touched upon earlier in the eBook, Agile and

DevOps methodologies encourage you to test early and For example:
test often. Testing should be performed at each step in If you hope to discover how your application per-
the development cycle and should continue after the forms with little or no load in order to get a baseline,
application is live. While it can be frustrating that a tes- you will run a single user test.
ters job is never done, its important to take into consid-
eration that with each testing and remediation cycle, the If you hope to determine how your system will
application improves. perform under normal expected load, you will run
a load test.
Applications go down under load for two reasons: either
If you hope to determine the breaking point, the point
developers didnt load test or, worse, they took the time
where your application either stops responding or
to load test but didnt prepare properly. Without ade-
responds so slowly that it is unusable, you need to
quate prep work, a load test cant find all the issues that it
run a stress test.
should.
If you want to know if your application has memory
So, how do you best prepare for a load test? Here are our leaks, you will want to run endurance or soak test.
9 steps to help you prepare:

1. What do you really need to know?

Determine what you want to learn about your application
or system. Each type of test is run differently, and looks at
your application in a different light. So, youll need to run

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 9

2. Decide on a number of users mula should be used to estimate the number of concur-
If you are going to load test, how many virtual users do you rent users required to achieve a specific page-view rate
want to simulate? In order to answer this, you will want to at fully-ramped load. It should not be used to estimate
approximate how many concurrent users may visit your site, the number of pages that will load within a specific time
depending on the time of day. interval.

Dont guess. Instead, talk to your architect or marketing 3. Study your analytics
people, and look at the performance specifications. If you Dont pretend to know how your customers use your ap-
want to know some concrete statistics from a historical per- plication. The only way to truly understand your users is
spective, go directly to your analytics reports. You may even to study history (i..e. analytics). By studying your analytics,
want to ask your engineers how many concurrent users they you will be able to create tests that are representative of
designed the application for, and your product owner for pro- your actual users, as opposed to tests that you think are
jected numbers, based on promotional activities. Plan to test representative of your users. In this regard, analytics are
that number and some percentage above it. a testers best friend.

There are numerous ways you can find the number of virtual 4. Performance is a team goal gather your team
users needed to run a load test. We recommend the follow- You need to involve a number of people in the testing
ing formula to find the number of virtual users needed to run effort, including: developer, network engineer, DBA, and
a load test. business owner. All of these individuals have a vested
interest in making the application successful, and each
Concurrent users = (peak hourly visits * visit duration in
will approach the problem from a different angle.
seconds) / 3600

You can find peak hourly visits and visit duration from an The correct solution will not fall directly into one of these
analytics tool, such as Google Analytics. Note that this for- buckets, but will be a combination of two or more. Make

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 10

sure each is aligned with the performance expectations and Complex scenarios
is available during testing to:
Parameterization
M
onitor their area of expertise
G
enerating load from multiple agents (network/cloud)
P
rovide balanced feedback
6. Be prepared to test your production application
Gain a sense of ownership for the health and perfor-
While it is valuable to test your application when it is in a
mance of the application
staging environment, this can leave some holes in your
5. Prepare your browsers testing. There are a few reasons for this, including:
Use testing software that brings you as close to your actual
Staging environments are not often exact duplicates
users experience as possible. You should be able to record
of production.
your scenario in the browsers of your choice, but you also
need to anticipate the browsers your users will most likely Staging environments are often accessible from
use. Consider the countries and regions where you antici- only inside the firewall.
pate high usage, and research the most used browsers.
There is something to be said for testing the same
Youll need to have these installed on your machine to begin system that you are gathering information about.
testing. Then you need to make sure your load testing soft-
ware emulates as closely as possible actual user behavior. 7. Set aside time to analyze results
This includes: You should be prepared to spend some time analyzing
test results as a group (remember all of those people
Parallel thread processing
that were present during testing?). Results need to be
Think time looked at to ensure bottlenecks/errors/weaknesses are
Multiple concurrent scenarios understood and remediation is effective. Reach out to
everyone involved and schedule adequate time.

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 11

8. Set aside time to make changes bottleneck is uncovered and corrected, another one
Different remediation will have differing costs in terms of rises to take its place. It is important to plan an Agile test-
time. Remediation such as implementing a caching strate- ing methodology, whereby performance testing is baked
gy, refactoring code, database optimization and hardware into each step of the development cycle. Additional
upgrades have a wide range of costs to implement in terms testing should be performed:
of both time and money. As an example, adding more hard-
When code is modified or updated
ware will require time to order the hardware, receive the
shipment, test the new hardware, install software and data, W
hen environment/infrastructure changes are
test, install into the network, and test some more. This can introduced
be weeks or months of work. This is less of a problem if you
When changes are made to the application server
are in the cloud. In which case, it takes less than a day. Many
or DB server
companies are opting to move to cloud infrastructures, offer-
ing an unlimited number of environments without the need When traffic spikes are anticipated
for additional hardware costs and time constraints.
Now that youve taken the time to really prepare, load
However, its always advisable to load test within your firewall testing your application will help you continually improve
as well. Some load testing tools, like LoadComplete, allow your product and your business.
you to do both. LoadComplete offers hybrid load generation
options you can generate load from your local comput- How role playing games can save your app:
ers, from virtual machines, or from the cloud Amazon EC2
instances. Its easy to get swept up in role -playing video games.
Who can resist the temptation to be anyone you want in
9. Plan an Agile testing methodology a fictional world filled with unending excitement? Believe
Once you remediate, it is time to test again. The saying, test- it or not, role playing games also offer valuable lessons
ing is a process, not a destination is very true. Each time a for testing your application. In the next section, we look

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 12

at how you can create scenarios in order to perform load browsing and leaving, and 5% (or less!) actually making
tests, and why this needs to be a major aspect of both de- a purchase.
ploying and improving web applications.

Percentage Role
How to record scenarios
The first thing to do is to determine the roles you will define 95% Browse and Leave
for use in your test. A role is equivalent to a certain type of 5% Browse, add something to the cart and checkout
user that will visit the tested website, and the steps they will
take while visiting.
The combination of these two roles, or scenarios, will
If the tested site is a retail site, for example, you might have represent actual site traffic.
the following roles:
After each scenario is recorded, you need to verify it in-
Browse and leave
dividually. This involves running a single virtual user for a
B
rowse, add something to the cart, and check out single pass through the scenario. This step should never
be forgotten.
If your tested site is a restaurant site, your roles might look
something like this:
Now youre ready to start testing. By recording scenari-
Browse menu and find directions os that imitate actual user traffic, youre setting the stage
to greatly improve customer experience and, if youre
Look at hours of operation and make a reservation
into e-commerce, get that percentage of purchases
above 5%.
Its best to choose at least three of the most common path-
ways through your site, and add a few uncommon routes
as well. Next, you need to break these roles down by per-
centage of traffic. A typical retail site may have 95% of users

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 13

 virtual users at the same time can create artificial bottle-
Making it real: emulating
.
necks in certain parts of the application such as the
real life conditions in your login process. In LoadComplete, you can elect to set a

load tests load profile whereby you start with a small number of
virtual users, and increase that number over time. For
example: you start with a single virtual user and add one
Whether its an elementary school math quiz, a college
virtual user every two seconds until you reach a certain
history exam, or a software development teams load
number of simultaneous users, and then hold that num-
test, we always want our tests to emulate real-life condi-
ber for the duration of the test.
tions. Otherwise, whats the point of testing?

Setting load duration

In this chapter, well discuss ways you can ensure your
In order for you to run a test with large numbers of vir-
load tests match reality. Some of the best perks about our
tual users, you will need to set duration for your test. By
load testing tool, LoadComplete, are the settings that aid
setting duration, each virtual user will execute its scenar-
the process of generating a realistic load test. Of course, io and when it reaches the end it will start over thereby
like all load testing tools, you can specify the number of maintaining the level of load. For example, if your sce-
virtual users to be simulated, but you can also set certain nario takes two minutes to be executed but you run a
conditions that easily create more powerful and reliable test for 10 minutes, the scenario might be executed five
load tests. No matter which load testing tool you choose, times by each virtual user. This function can be found un-
make sure it allows you to set some version of these ba- der the continuous load tab of the load profile in Load-
sic conditions. Complete. Merely check the enable box, and set a time.

Ramping up virtual users Parameterizing tests

When running performance tests, its not desirable (and While recording a scenario, you may need to specify
realistic) to start all virtual users at the same time. Starting all some parameters that will be used for further test runs.

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 14

For example, you can enter some search terms, user Simulating concurrent requests
names and so on in the applications fields. However, it To download a Web page, modern Web browsers send
is not a good practice to play back a test with the same requests to the server using several simultaneous
recorded data for each user as it does not simulate the connections. These parallel requests download imag-
real-life conditions. To solve the problem, LoadComplete es, scripts, CSS files and other resources located on
allows you to parameterize your load test using special the page. Unlike many other load testing tools, Load-
Complete automatically detects and simulates parallel
variables. A variable can store desired data and your re-
requests that is, it sends requests exactly like your
quests can use this data during test runs.
browser did when you recorded your scenario. This
makes the simulation closer to real-world conditions.
Emulating browsers, network connection bandwidth and
browsing speed Most load testing tools collect exhaustive information
Real users visit a website using different browsers, the band- for each load test run. Traditionally in load testing, you
width of users Internet connections can vary significantly, needed to have pretty extensive knowledge in order
and they spend different amounts of time on each page. to interpret the majority of load testing data in any kind
of meaningful way. This, unfortunately, has resulted in
You can configure all these parameters when creating countless load tests that generated tons of useless data.
load tests in LoadComplete. This means you can specify
different browsers and connection bandwidth for differ- Analyzing data in Load testing: What you need to
ent groups of virtual users. You can also specify think know
More often than not, developers and QA managers
time for each tested page (we call this think time, as it
come away from a trial of load testing software with little
simulates the time when a user is viewing the page and
more than the number of users that will crash their sys-
thinking). You can easily randomize the think time for sim-
tem. Unless they have a professional load tester on staff,
ulated scenarios to better emulate real user activity.
most development teams dont have the resources, or

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 15

knowledge to garner all they could from their load tests. ed and at what level of load. This is especially important
That data is wasted because the person running the tests is information to see in chart format. It is important to see
unable to apply it to an applications performance. Luckily, im- which errors and warnings are generated and be able
provements in graphics and UIs have made interpreting data to see how that changes as load increases. A common
much easier if you know what to look for. error at high levels of load is Server Error 500s.

Heres a list of the most important results in load testing Request and response throughput
and how you should be working with them. Its important to see the amount of data going to and
coming from the tested system. This is especially im-
Page load time portant in a case where load is increasing, but band-
You need to know the average page load time for each page width reaches and maintains a plateau. In this case, it
in your scenario. You might have a strict Service Level Agree- becomes apparent that bandwidth is being throttled at
ment (SLA) that mandates how quickly pages must load, or some point in the process, possibly at the firewall.
may just want to know what this number is. It is also import-
ant to know if one page takes longer than others to load Hosts
this indicates a bottleneck in your application. Because so many of todays websites call out to a pleth-
ora of additional hosts for things like content delivery
Response load time networks, ad servers, analytics servers, social media and
Just knowing page load time is not enough. If a page is syndicated content, it is important for these sites to be
slow, you need to know why. Being able to look at average enumerated in your reports. Its equally important to be
response times for each response gives you a detailed look able to view all of the calls to a particular host. If a host is
into where the time is being spent. called from your pages, the response time for those re-
quests will add to the time it takes your pages to render.
Errors and warnings
You need to know which errors and warnings were generat-

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 16

The Dos and Donts of Load This will happen because the recorded traffic will
not reproduce the authentication procedure, and
Testing the tested web server will ignore the simulated
requests.
As you get ready to implement load testing to your per-
Parameterize scenarios to simulate more realis-
formance strategy, there are a few dos and donts youll
tic load on the server. Parameterizing scenarios
want to keep in mind.
involves replacing recorded parameters in the
requests with variable values. This way you can
Dos:
make virtual users send user-specific data to the
Clear browser cache and cookies before re- server. This lets you add dynamic behaviors to
cording traffic. When the cache and cookies are your scenarios as if they were run by a group of
not clean, while recording a user scenario, the web unique human users. Note: Before you parame-
browser uses cached data and cookies to process cli- terize a scenario, carefully explore its requests
ent requests instead of sending data to and receiving and responses to better understand what data is
responses from the server. However, while running being transferred to and received from the server.
the scenario, LoadComplete does not use the cache.
It actually sends requests to and receives responses Verify user scenarios. Before creating tests on
from the server. This may cause a difference between the basis of a recorded scenario, make sure that
the recorded and actual response codes. LoadCom- the scenario is executed successfully for one
plete reports this difference as warnings or errors, virtual user. This can help you identify bottlenecks
and the load test is treated as failed. of the scenario and eliminate problems which
are not related to the number of virtual users and
Start recording a new scenario from the web additional testing conditions.
browsers start. If you start recording a scenario
after you connect to the tested web server and open Arrange user scenarios in your tests so that criti-
a few web pages, the playback of the scenario will fail. cal functionality is tested first.

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 17

 During test runs, monitor the tested server metrics Do not overly stress the client test systems.
as well as resource usage on the client computer
Do not use zero think time. Make sure that think
where LoadComplete is running. You do not have to
time in your test is based on real-life conditions.
overly stress the client machine. Try to control the
Using zero think time does not provide realistic
processor and memory usage metrics on the client
user simulation and puts an abnormal load on the
computer and keep them below the threshold limit.
tested server. However, omitting think time can
Otherwise, the client computer may fail during your
help you determine bottleneck issues.
testing.

Donts:
Do not run tests in real environments*. A real
environment can have other network traffic, and this
may affect test results. To avoid excess data transfer,
use a test environment that behaves in the same way
as the real environment except that there is no other
traffic usage. (Refer details on prepare stage to per-
form load test in production).

Do not try to crash the tested server. The goal of

web server performance testing is not to break the
server, but to identify web application performance
bottlenecks under various loads.

Do not browse web pages during test runs. Param-

eterizing This may affect the traffic simulation and
make the test fail.

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 18

 About LoadComplete
LoadComplete is an affordable and easy to use load testing software tool designed to test your entire Web application
so you can identify and pinpoint serious bottlenecks early, inside or outside your network, at both the application and
server levels. Amazon EC2 integration lets you generate load from the cloud so that you can cost effectively run and
manage both on-premise tests and large-scale cloud tests from within the same load testing tool.

Key features of LoadComplete

Automated recording of browser-based user scenarios without scripts

Correlation wizard for dynamic variables and visual programming

Custom load shaping to create detailed visitor profiles to follow

Dynamic simulation of visitor data and traffic patterns via data wizard

Configurable think times mimic human pausing during page views

Automatic or custom test data generation instantly diversifies test cases

Smart server side monitors capture infrastructure statistics

Detailed reports provide in depth analysis of test failures

LEARN MORE ABOUT LOADCOMPLETE

LOAD TESTING 101: ESSENTIAL TIPS FOR TESTERS AND DEVELOPERS 19

LoadComplete allows you to generate load from
virtual machines, on-premise computers, or even
the cloud. Customized reports in LoadComplete
help you get end-to-end visibility and find
problems at both client and server side.

TRY IT FOR FREE

 About SmartBear Software
As the leader in software quality tools for the connected world, SmartBear
supports more than four million software professionals and over 25,000 orga-
nizations in 194 countries that use its products to build and deliver the worlds
greatest applications.

With todays applications deploying on mobile, Web, desktop, Internet of

Things (IoT) or even embedded computing platforms, the connected nature
of these applications through public and private APIs presents a unique set of
challenges for developers, testers and operations teams.

SmartBears software quality tools assist with code review, functional and
load testing, API readiness as well as performance monitoring of these mod-
ern applications.

LEARN MORE ABOUT SMARTBEAR

Visit SmartBear community:

LOAD TESTINGWISHT
10 THINGS DEVELOPERS 101: THEIR
ESSENTIAL TIPSUNDERSTOOD
BOSSES FOR TESTERSABOUT
AND DEVELOPERS
CODE REVIEW 21
21
 Over 4 million software professionals and
25,000 organizations across 194 countries
use SmartBear tool

4M+ 25K+ 194

users organizations countries

See Some Succesful Customers >>

API TESTING PERFORMANCE CODE

RE AD I N E S S MONITOR I N G COL L ABORAT I ON

Functional testing through Functional testing, Synthetic monitoring for API, Peer code and documentation
performance monitoring performance testing and test web, mobile, SaaS, and review
management Infrastructure

SEE API READINESS SEE TESTING SEE MONITORING SEE COLLABORATION

PRODUCTS PRODUCTS PRODUCTS PRODUCTS