A Cross-browser Web Application Testing Tool1

Shauvik Roy Choudhary, Husayn Versee, Alessandro Orso

Georgia Institute of Technology
shauvik@[Link], hversee3@[Link], orso@[Link]

Abstract—Web applications have gained increased popularity to limit the number of officially supported browsers, which
in the past decade due to the ubiquity of the web browser can also negatively affect the user experience [5], [6].
across platforms. With the rapid evolution of web technologies,
the complexity of web applications has also grown, making
maintenance tasks harder. In particular, maintaining cross- A. State of the Art
browser compliance is a challenging task for web developers,
as they must test their application on a variety of browsers and Currently, for detecting cross-browser issues in a given web
platforms. Existing tools provide some support for this kind of page, developers must render the page in different browsers
test, but developers are still required to identify and fix cross- and manually inspect the appearance and behavior of the page.
browser issues mainly through manual inspection. Our W EB D IFF
Commercial tools, such as Microsoft Expression Web [7] and
tool addresses the limitations of existing tools by (1) automatically
comparing the structural and visual characteristics of web pages Adobe’s Browser Lab [8], can assist manual inspection by
when they are rendered in different browsers, and (2) reporting presenting a side-by-side differential rendering of the web
potential differences to developers. When used on nine real application in two browsers. However, these tools still require
web pages, W EB D IFF automatically identified 121 issues, out of the developer to spend a considerable amount of effort in
which 100 were actual problems. In this demo, we will present
manually identifying, understanding, and fixing the actual
W EB D IFF, its underlying technology, and several examples of its
use on real applications. cross-browser issues. Moreover, some functionality related
issues are difficult to spot by manual inspection, as they may
I. I NTRODUCTION not result in readily observable differences.
Web applications follow a traditional client-server comput- There are only a few research tools that target cross-browser
ing model: server-side components receive requests from a issues. The compliance assessment technique presented by
client and generate and return responses to such requests. Eaton and Memon [9] requires developers to manually provide
These responses typically consist of HTML documents that examples of correct and faulty (e.g., containing layout issues)
contain data, CSS style rules, and client-side scripting compo- pages, which are then used to compute the probability of a
nents in the form of JavaScript (standardized as ECMAScript) new HTML tag to be faulty. Although, this technique can be
or VBScript. Over the past decade, web applications have useful for simple cases, it requires a considerable amount of
evolved from having extremely simple and thin clients to manual effort for tagging html pages. Moreover, it does not
having increasingly rich and complex client-side components. consider client-side script and CSS style components. More
recently, Tamm developed a research tool that leverages both
With this increased complexity of web applications, the
DOM (Document Object Model — [Link]
maintenance effort required by developers has also increased
and visual information for finding layout bugs in a particular
proportionally. In particular, because the web applications can
browser [10]. Tamm’s tool is focused mainly on the text
be run on a variety of web browsers and platforms on the client
portion of a page and identifies web page elements by hiding
side, cross-browser compatibility issues are prevalent in such
and showing different elements at different times. In our prior
applications. (Current browser statistics report usage of seven
experience with a similar approach, we have discovered that
popular web browsers across different platforms [2].) Cross-
this approach is typically very expensive, as it must render
browser compatibility issues range from simple cosmetic prob-
a page a potentially large number of times. Moreover, the
lems in the user interface to critical functionality failures.
approach cannot detect many relevant issues, as it focuses
According to the latest statistics of the Mozilla’s Broken
mostly on text elements. The VIPS algorithm [11] infers the
Website Reporter [3], users have reported cross-browser issues
hierarchy of a web page from the page’s visual layout, rather
on 1,767,900 websites on 463,314 hosts. These figures include
than from the DOM, by segmenting its screenshot into visual
12,072 web sites reported in the past week and 1,568 web
blocks. A limitation of that approach is that it assumes that the
sites reported just in the last 24 hours. Because cross-browser
page follows a specific layout and groups regions of the page
issues are observed on the client side (i.e., in the browser)
accordingly. However, modern web pages have often complex
and directly affect the users, there is an increased interest
and creative layouts, and that technique is unlikely to work
in identifying such issues during in-house testing, before the
in these cases. Moreover, disregarding the structural data in
software is released [4]. In addition, organizations are trying
the DOM prevents from reporting the specific location in the
1 This demo illustrates the implementation of a technique presented in a HTML document that is responsible for an issue, which makes
paper accepted for publication at ICSM 2010 [1]. it more difficult for developers to fix the issue.

26th IEEE International Conference on Software Maintenance in Timișoara, Romania

978-1-4244-8628-1/10/$26.00 ©2010 IEEE
B. Proposed Solution information of the web page and captures a screenshot for each
To address the limitations of the existing techniques, we browser window. Specifically, the DOM information captured
developed W EB D IFF, a technique and tool for (1) detect- by W EB D IFF for every DOM node consists of the following
ing cross-browser issues automatically and (2) reporting the properties:
locations of such issues in the corresponding web page to • tagname: Name of the tag associated with the DOM element.
help developers fix them. W EB D IFF is based on the concept • id: Unique identifier of the DOM node, if defined.
of differential testing [12]. It loads a web page in different • xpath: X-Path of the node in the DOM structure.
environments and compares the behaviors of the page in such • coord: Absolute screen position of the DOM element.
environments; intuitively, a difference in behavior indicates a • clickable: True if the DOM element has a click handler.
potential problem. • visible: True if the DOM element is visible.
Our tool can be used in isolation or in combination with • zindex: DOM element’s screen stack order, if defined.
existing tools for functional testing, such as Selenium [13], and • hash: Checksum of the node’s textual content, if any.
existing test-input generation techniques for web applications W EB D IFF captures the screenshot of the web page by taking
(e.g., [14]–[18]). These tools can be used to generate and run a snapshot of the browser window and extracting just the
inputs for the web applications under test, while W EB D IFF content in the viewport (i.e., the section of the browser where
can compare the web pages generated using such inputs on the web page is rendered).
multiple browsers and platforms.
B. Detection of Variable Elements
To assess the effectiveness of W EB D IFF, we performed an
empirical evaluation in which we used our tool on nine real Web Pages often have generated elements that differ across
web pages. Our results are promising: W EB D IFF automatically executions, such as advertisements and statistics about the
identified 121 issues, out of which 100 were actual problems. page. Such variable elements must be ignored during com-
In this tool demonstration, we will present W EB D IFF, its parison, as they are highly likely to result in false positives.
underlying technology, and several examples of its use on real W EB D IFF detects variable elements by loading the web page
applications. In the rest of the paper, we describe in greater in the reference browser twice and comparing the DOM and
detail our technique, results, and proposed demonstration. screenshot information obtained in the two cases. The intuition
is that the variable elements will likely vary in subsequent
II. O UR T ECHNIQUE requests. All DOM nodes that reveal either a structural or a
As discussed above, our technique finds dissimilarities be- visual difference in this analysis are marked as variable and
tween the corresponding elements of a web page rendered ignored in the subsequent steps.
in different browsers, one of which is considered to be a
C. Cross-browser Comparison — Structural Analysis
“reference browser”. Having a reference browser is important
to keep the number of comparisons low. Moreover, it mimics The goal of this phase is to match the DOM nodes obtained
a normal scenario where developers focus on making sure from different browsers for the same page. W EB D IFF traverses
that the web application behaves well in their browser of the DOM trees for two browsers in parallel while searching for
choice and then check whether it behaves consistently in the best matching nodes. While analyzing a pair of nodes, the
other browsers. More precisely, the technique operates as technique computes a match index, a number between zero and
follows. Given a web page URL, it first opens the page in all one that represents how similar the two nodes are. The match
considered browsers. Second, it extracts from each browser the index is computed using the DOM properties that are recorded
DOM information and a screenshot of the rendered web page. in the previous step (see Section II-A). More precisely, two
Third, it identifies variable elements on the reference browser’s nodes with the same id are marked as a perfect match, and so
DOM and eliminates them from further consideration. Fourth, are two nodes with the same xpath. If none of these conditions
it matches the nodes in the DOM tree hierarchy amongst is satisfied the match index for the two nodes is proportional
different browsers. Finally, for each pair of matched nodes, it to the similarity of the xpaths for the nodes and the number of
compares the attributes of the two nodes and the regions of the other properties that have identical values. When all possible
screenshots corresponding to the nodes to find inconsistencies. node pairs have been considered, the matching node for a given
We explain each of these steps concisely in the following node n is the node m such that there is no other pair of nodes
sections. More details on the technique can be found in [1]. that (1) contains n and (2) has a match index greater than the
match index for the pair (n, m). At the end of this step, each
A. Data Collection node of the reference browser’s DOM is mapped to a node in
W EB D IFF launches the web page under test in all browsers the DOMs of each other browser considered.
considered and adjusts the browsers’ window size such that
they are all equal. The browser size is obtained inside the D. Cross-browser Comparison — Visual Analysis
web application by a simple script and is communicated to The DOM does not have information on how web page ele-
the tool that resizes the browser accordingly. More details ments exactly appear on screen. Therefore, our technique also
about the engineering of this step can be found in Section performs a visual analysis of the page. In this step, W EB D IFF
IV. After the browsers are ready, W EB D IFF extracts the DOM leverages the structural information computed in the previous
step to (1) identify corresponding elements in the browser The main controller runs in the host machine, and the web
screenshots and (2) perform a graphical matching of such browsers are installed in a virtual machine connected to it. This
elements. As a preliminary step of the graphical matching, setup allows the host to connect to multiple virtual machines,
W EB D IFF grays out the areas corresponding to variable nodes which in turn allows browsers on multiple platforms to be
in the captured screenshots, so as to eliminate possible false used for testing. Different conflicting versions of the same
positives caused by them. It then checks for four classes of browser can also be used with such a setup. For instance,
issues: positional, size, visibility, and appearance differences. currently three different versions of Internet Explorer are
Positional differences are found by checking the parent (or popularly used for web browsing and should therefore be
container) nodes of matching HTML elements and the relative used for compatibility testing of web applications. The virtual
position of the elements with respect to such container nodes. machine contains several browsers, controlled by the Data
Visibility and size differences are identified by comparing the Collection Engine. The Data Collection Engine consists of
properties of two matching nodes. Finally, W EB D IFF identifies three main components: a GUI Automation component, which
appearance differences by graphically comparing the regions automates the tasks of launching browsers, opening web pages,
of a web page that correspond to matching HTML elements. and performing actions on the web page; a DOM Extraction
The graphical comparison uses a histogram based technique component, which is a JavaScript tool to extract the tree based
that can compute the distance between two images [19]. If DOM of the web page; and a Screen Capturing component,
such distance is larger than a given threshold, the two elements which captures and saves the screenshot of a web page. The
are reported as different in the two browsers considered. host machine consists of the W EB D IFF Controller, which
is responsible for starting the Web Proxy and then remotely
III. U SAGE S CENARIOS
starting the capture process on all virtual machines. The Web
Table I lists some scenarios in which the W EB D IFF tool can Proxy is necessary to overcome the browser’s cross domain
be useful. In the first scenario, the tool assists the developers in restrictions. The DOM Extraction script submits the data back
supporting a new web browser. The second scenario describes to the location where the web page originated from. The
the use of the tool for regression testing. In the final scenario, Web Proxy identifies the HTTP request as a data-capturing
W EB D IFF allows developers to reproduce an error that was request and redirects it to a local server side script that
reported to them, thereby allowing them to investigate the saves the data in the Data Store on the host machine. The
issue for resolving it. In general, W EB D IFF can be used in Web Proxy also passes values between the JavaScript DOM
a variety of scenarios for supporting software development Extraction script inside the web page and the GUI Automation
and maintenance tasks. tool. Similar to the data-capturing requests, the Web Proxy
TABLE I handles the communication messages in a special manner.
S CENARIOS OF USE FOR W EB D IFF . These messages are essentially set and get requests to the
web service, such that one of them can retrieve the data set
Steps Expected Output by the other. Specifically, the GUI Automation tool reads the
A new version of a browser is released. scrollWidth and scrollHeight of the web page and
Add the new browser ver- List of issues in the web decides how much more the web page needs to be resized.
sion to W EB D IFF and run pages of the web appli- After the DOM and visual information have been collected
it on the web application. cation related to the new from the target web page, the Analysis Engine reads this
browser version are re- information and performs the steps explained in Section II:
ported. variable element identification and cross-browser comparison.
A new version of the application is released.
Exercise W EB D IFF on the List of issues introduced B. Technologies Used
new version of the applica- in the new version are re- The components of W EB D IFF use a varied set of plat-
tion on all the considered ported to the developers. forms and libraries. Communication between the machines
browsers. is performed using the Software Testing and Automation
A cross-browser issue needs to be reproduced. Framework (STAF),2 which is a P2P based cross-platform
Run W EB D IFF on the af- The relevant issue gets automation platform. The security settings in STAF are con-
fected web page of the identified by W EB D IFF, figured to allow the host to start the remote Data Collection
application on the specific allowing the developer to Engine process on each virtual machine. The GUI Automation
browsers. investigate it. component in the Data Collection Engine process uses the
win32api3 to access and perform activities on the web
browser window inside the Microsoft Windows virtual ma-
IV. T HE T OOL chine. The Screen Capturing tool uses the win32api to
A. Architecture obtain the browser window handle and the python image
Figure 1 shows a high level architecture of the tool. As 2 [Link]

shown in the figure, the tool operates in a distributed manner. 3 [Link]

 Fig. 1. Architecture of W EB D IFF.

Fig. 2. Report generated by W EB D IFF.

library4 to obtain and save the screen captures of a web C. Reporting of Issues Found
page. On the host machine, the W EB D IFF Controller and Web
Proxy are written in the python programming language, and
the latter uses the twisted 5 networking library for making Figure 2 shows a sample report generated by W EB D IFF.
network connections. Inside the Analysis Engine, the Variable The report is an HTML page that consists of two major
Element Identification and DOM tree mapping components are sections: visual section and data. The top visual section shows
written in Java. The Cross-Browser Comparison component a screenshot of the web page in the reference browser, along
is partly written in Java (for the part comparing matched with the same in another browser. This visual section can be
DOM locations) and partly in C using the Open Computer used by developers to quickly inspect the screenshots side by
Vision library6 (for the part performing image processing for side. The lower data section contains the list of issues found in
screenshot comparison). tabular format. The list contains the type of issue, the browser
on which it was found, and the corresponding screen and DOM
4 [Link] locations of the element that manifested the issue. As the figure
5 [Link] shows, the W EB D IFF report is intuitive and can be useful to
6 [Link] find and fix cross-browser issues.
 Fig. 4. Issues with Georgia Tech’s web site: shifted elements in IE.

Fig. 3. Georgia Tech web site.

that were manually confirmed as true positives: positional

V. E VALUATION differences (Pos.), changes in size (Size), visual differences
(Vis.), and general appearance differences (Gen.). In addition,
For evaluating our tool, we selected three widely used web the table shows the total number of true positives (Tot.) and
browsers: Mozilla Firefox7 (Version 3.6), Google Chrome8 the total number of false positive (FP) reported.
(Version 4.1), and Internet Explorer9 (Version 8.0). We in-
stalled the browsers in a Microsoft Windows XP virtual As the results in Table III show, W EB D IFF was able to
machine. For the evaluation, we selected nine web pages as automatically discover and report a large number of cross-
subjects. The first subject is the front page of our institutional browser issues: 121 issues of different types overall. In par-
web site, [Link] shown in Figure 3. As it can ticular, W EB D IFF was able to identify the known issues in
be seen in the figure, the web site uses a professional template the Georgia Tech web site. W EB D IFF reported a total of 21
and makes use of a variety of HTML features. This page was false positives for the nine subjects considered in the study.
selected due to our prior knowledge of its cross-browser issues, This corresponds to a 17% false positive ratio, which can be
so that we could check whether W EB D IFF identified them. considered acceptable given that W EB D IFF is the first tool for
Some of these issues are clearly visible in Figure 4: in the automatically detecting such problems. Moreover, in six out
web page, some elements appear shifted to the right when of nine subjects (a majority), the false positive rate is even
compared to the page shown in Figure 3. lower. Manual analysis of the results showed that the false
The remaining eight pages that we used for the study were positives reported by the tool were due to (1) minor differences
chosen randomly. To perform a fair sampling of subjects, we in some elements containing text and background images that
picked them using a random link generator service provided by are mostly unnoticeable for the human eye and (2) presence of
Yahoo! ([Link] The complete list some variable elements that W EB D IFF failed to identify. After
of web pages considered is shown in Table II. As the table studying the issues, we believe that they can be eliminated by
shows, the list includes a wide range of web applications, careful engineering of the tool.
ranging from static informational web pages to commercial We also performed a manual checking of the web pages
web sites. considered in the study, and the checking did not reveal any
The results of our study are shown in Table III. The numbers additional cross-browser issue that our tool had not revealed.
listed in the table are the total number of issues found while In other words, to the best of our knowledge, the tool generated
comparing the web pages rendered in Firefox (considered as no false negatives.
the reference browser) with pages rendered in Chrome and
One last point that is worth discussing is the cost of our
Internet Explorer. The table reports, for each subject, the
approach. The analysis time for each of the nine web pages
different types of cross-browser issues identified by W EB D IFF
was less than five minutes. Because W EB D IFF can be run
7 [Link] overnight, this time is definitely acceptable. We therefore did
8 [Link] not perform any more detailed study of the performance of
9 [Link] W EB D IFF.
 TABLE II
S UBJECTS USED FOR THE STUDY. T OOL AVAILABILITY
The tool has been released under MIT license, for public
Subject URL Type
download and use. More information about the tool can be
GATECH [Link] university
BECKER [Link] company found at the following location: [Link]
CHESNUT [Link] lodge ∼shauvik/[Link]
CRSTI [Link] hospital
DUICTRL [Link] laywer ACKNOWLEDGMENTS
JTWED [Link] photography
ORTHO [Link] informational
This work was supported in part by the NSF awards CCF-
PROTOOLS [Link] company 0916605 and CCF-0725202 to Georgia Tech.
SPEED [Link] e-commerce
R EFERENCES
TABLE III [1] S. Roy Choudhary and A. Orso, “Webdiff: Automated identification of
N UMBER OF CROSS - BROWSER ISSUES IDENTIFIED . cross-browser issues in web applications,” in ICSM ’10: Proceedings
of the International Conference on Software Maintenance. IEEE,
# Faults identified September 2010.
Subject Pos. Size Vis. Gen. Tot. FP [2] [Link], “Browser statistics month by month,” [Link]
GATECH 2 3 0 1 6 0 [Link]/browsers/browsers [Link], May 2010.
BECKER 2 12 0 2 16 1 [3] Mozilla, “Firefox broken website reporter,” [Link]
CHESNUT 8 4 0 2 14 2 app/stats/, July 2010.
CRSTI 4 4 0 2 9 0 [4] Cambridge Network, “Estate agents must update web browser compati-
DUICTRL 9 8 0 2 19 4 bility ahead of microsoft announcement,” [Link]
JTWED 3 9 0 1 14 0 [Link]/news/article/[Link]?objid=69332, March 2010.
ORTHO 0 0 0 2 2 2 [5] The Korea Times, “Korea sticking to aging browser,” http://
[Link]/www/news/biz/2010/02/123 [Link], Febru-
PROTOOLS 4 5 0 2 11 9
ary 2010.
SPEED 23 5 0 2 30 3
[6] N. Gohring (IDG News), “Google to end support for ie6,” [Link]
Total 55 50 0 16 121 21 [Link]/article/188190/google to end support for [Link], Jan-
uary 2010.
[7] Microsoft, “Expression web,” [Link]
VI. C ONCLUSION products/Web [Link], May 2010.
Cross-browser issues are increasingly prevalent, and achiev- [8] Adobe, “Browser lab,” [Link] May 2010.
[9] C. Eaton and A. M. Memon, “An empirical approach to evaluating web
ing consistent behavior across all major browsers has been application compliance across diverse client platform configurations,”
a serious concern for web application developers. Current Int. J. Web Eng. Technol., vol. 3, no. 3, pp. 227–253, 2007.
techniques for detecting and fixing such issues are immature [10] M. Tamm, “Fighting layout bugs,” [Link]
fighting-layout-bugs/, October 2009.
and require a considerable amount of manual work from the [11] D. Cai, S. Yu, J.-R. Wen, and W.-Y. Ma, “Vips: a vision-based page
developers. Our tool, W EB D IFF, addresses the limitations of segmentation algorithm,” Microsoft Research, Tech. Rep., November
existing techniques by automatically identifying cross-browser 2003.
[12] W. M. McKeeman, “Differential testing for software,” Digital Technical
issues. To do so, W EB D IFF compares web pages by leveraging Journal, vol. 10(1), pp. 100–107, 1998.
both their structural (web page’s DOM) and visual (web page’s [13] OpenQA, “Selenium web application testing system,” [Link]
snapshots) characteristics. We evaluated W EB D IFF on nine org/, May 2010.
[14] D. Roest, A. Mesbah, and A. v. Deursen, “Regression testing ajax
real world web applications, and the results of the evaluation applications: Coping with dynamism,” in Software Testing, Verification
show that the tool is effective in identifying cross-browser and Validation (ICST), 2010 Third International Conference on, 6-10
issues while keeping the false positive rate low. 2010, pp. 127 –136.
[15] S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D.
This demo paper presented our tool, its underlying tech- Ernst, “Finding bugs in dynamic web applications,” in ISSTA ’08:
nique, and some of its implementation details. The paper Proceedings of the 2008 international symposium on Software testing
also presented some possible scenarios of usage for the tool and analysis, 2008, pp. 261–272.
[16] F. Ricca and P. Tonella, “Analysis and testing of web applications,” in
and our initial empirical evaluation. In future work, we will ICSE ’01: Proceedings of the 23rd International Conference on Software
investigate ways to further lower the number of false positives Engineering. Washington, DC, USA: IEEE Computer Society, 2001,
(e.g., by better detecting variable elements and improving pp. 25–34.
[17] X. Jia and H. Liu, “Rigorous and automatic testing of web applications,”
the accuracy of our graphical matching) and to improve in In 6th IASTED International Conference on Software Engineering and
the tool based on feedback from our users. We will also Applications (SEA 2002, 2002, pp. 280–285.
perform more experiments on additional subjects to confirm [18] W. G. J. Halfond and A. Orso, “Improving test case generation for
web applications using automated interface discovery,” in ESEC-FSE
our initial results. In the longer term, we will investigate the ’07: Proceedings of the 6th joint meeting of the European software
use of our approach to evaluate behavioral equivalence of web engineering conference and the ACM SIGSOFT symposium on The
applications that can run not only in different browsers, but foundations of software engineering. ACM, 2007, pp. 145–154.
[19] Y. Rubner, C. Tomasi, and L. J. Guibas, “The earth mover’s distance as
also on completely different platforms (e.g., web applications a metric for image retrieval,” International Journal of Computer Vision,
running on both desktop and mobile platforms). vol. 40, pp. 99–121, 2000.