Securing Oracle Application Server PDF
Securing Oracle Application Server PDF
Table of Contents
Introduction 3
Network Security 42
Conclusion 45
Contact Information 51
Appendix B: Bibliography 55
Introduction
Recent software development and deployment trends have caused a shift
from the classic client-server two-tier architecture to a three-tier model. The
software that occupies this new middle tier is commonly referred to as
middleware, precisely because that is what level it occupies in the hierarchy.
This transition was predominantly caused by the explosion in popularity of
the Web, and the increasing dominance of the Web browser in the client tier.
The programming logic, previously located in the client tier in what is now
known as a fat client, was instead moved onto the server. Initially, the
functionality of the middle tier was restricted to that of the HTTP server and
the scripting engine. However, over time the complexity of the middle tier
has increased to encompass a wide variety of components commonly used
for enterprise application deployment.
The end result should be useful in securing a brand new installation of Oracle
Application Server, or to assess the security of an existing installation. The
focus of this document is a minimal installation of Oracle Application Server.
As will be readily apparent, the minimal installation is a big enough challenge
on its own. We will be focusing on the Oracle Application Server 10g R2
(10.1.2.0.2) release with the following major components installed:
We used two installations of Oracle Application Server for our tests, one
running on Windows (Windows Server 2003) and the other running on Unix
(Red Hat Enterprise Linux AS 4). We wanted to make our guidelines equally
useful to both major platforms. At the time of writing Oracle has made the
preview of the next-generation release, R3 (10.1.3), available to the general
public. Although the new release brings many improvements over R2, the
core functionality (which is our primary focus here) remains stable. Thus we
expect the document to apply equally well to Oracle Application Server 10g
R3 when it becomes officially available. Either way, our approach is to
provide you with a step-by-step guide. You should be able to apply the same
steps to any future version of Oracle Application Server and secure the
installation.
The core component of Oracle Application Server is, of course, Oracle HTTP
Server. Earlier versions of Oracle HTTP Server were entirely proprietary, but
Oracle has decided to base the newer versions on the Apache web server.
This has proven to be a good move. Not only is Oracle HTTP Server now
based on a popular web server of proven quality, it is also compatible with a
great number of third-party modules. It also benefits from widely available
Apache expertise.
A deeper look into Oracle HTTP Server reveals support for most mainstream
technologies:
CGI scripts
Persistent CGI scripts through FastCGI
Compatibility with third-party Apache modules
J2EE container (also known as OC4J)
mod_perl
mod_php (based on 4.3.9 in 10g R2)
PL/SQL Server Pages
SOAP/Web Services (through OC4J)
XML (through OC4J)
SSL
WebDAV
Oracle HTTP Server utilizes two programming models, one on Windows, and
the other on Unix platforms. Understanding the limitations of the
programming model are important when implementing defenses against
then one child process for every request being processed. This is quite a
robust programming model; if the web server crashes while it is processing a
request the crash will affect one process only (the one that crashed). The
parent process will simply create a new child process to replace it and carry
on as usual. The requests processed at that time will remain unaffected.
Consequently, the impact of certain types of vulnerabilities, typically the ones
that can be used to cause Denial of Service attacks by crashing the web
server processes, is lessened. On Windows, a threaded programming model
is used. The one parent process is still there, but in this case there is only
one child process which runs with multiple threads. This programming model
is better suited to Windows, but a crash in the child process will also
terminate all requests that are being processed at that time.
The other important detail to note from the diagram is the relationship
between the web server and the components that provide the additional
services. The core of the web server, on its own, only knows how to serve
static files. A separate web server module needs to be used for all other
required functionality. Modules are thus an essential and integral part of the
web server. It is entirely up to the module to decide how the request will be
processed. Some modules, such as mod_perl, do all of the work from within
the web server. Other modules, such as mod_oc4j, will relay the request to
some other process for processing. Having requests processed outside the
web server is much better for security because if the external process is
compromised it will not affect the work of the web server.
Before Installation
Time invested reading the Oracle Application Server documentation before
installation is time well spent. The more you know about the software the
easier it will be to make an installation plan and configure the necessary
components. If you haven't already secured the operating system, go ahead
and do that first. There is no point in building on top of something you arent
certain is secure. Make sure the computer you are using for the installation
cannot be accessed from the outside. If you need to download any patches,
do it before the installation, and then simply unplug the network connection
until you finish.
At this point it may be a good idea to record the state the system is in. After
the installation, you will be able to use this information as a baseline for
comparison to determine what changes were introduced by the installation
process. If you are using Unix more than likely you already have the tools
you need for this task. On Windows, I recommend the free PsTools
# netstat -lnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 :::6000 :::* LISTEN
tcp 0 0 :::22 :::* LISTEN
You should also record the running processes. On Unix, record the output of
ps auwwxf. On Windows, execute pslist -t. The output is typically very
long and for that reason we will not reproduce it here. You are now ready to
proceed with the installation.
Installation
On Unix, first create the main Oracle Application Server account. Most people
choose to go with oracle. This is what we will assume here for sake of
simplicity. However, you should use another non-obvious name on a
production system. Well-known account names are sometimes targeted by
the attackers in brute force attacks against passwords. Choosing a different
name could preempt such efforts.
# groupadd oinstall
# useradd oracle -g oinstall
It is usually a good idea to lock this account, especially if there will be more
than one person performing the maintenance. If they all connect to the
server as oracle it will be impossible to track who has done what, and
when. You then need to start the installation process as the new user. The
process is quite straightforward. You will need to execute certain scripts as
root to allow OAS to configure privileges correctly.
On Windows, create a new local user account specifically for this purpose,
add it to the Administrator group, and install OAS while running as that user.
After the installation, download the latest patches for the installed
components from Metalink (if you haven't already done so), and apply them.
As you can see, the installation has created a large number of new
processes. We can make two conclusions from looking at the list of open
ports. First, some of the processes are only listening on the loopback address
(127.0.0.1), meaning they are only accessible from the server itself. Thus it
is reasonable to assume these ports are used for the internal components of
the installation. Of all the listening ports only one needs to be accessible
publicly, which by default is Web Cache port 7777 (although you are likely to
change it to port 80 later on). This means you need to protect all other ports
from the potential attackers with a firewall. Keep this thought in mind for
later.
Now you can generate a list of newly created processes. You can do this
either by comparing the process lists before and after the installation. But
even if you do not have a process list from before you can take advantage of
the fact that the new processes are all still running as the main installation
user. You can list all processes belonging to that user with ps -o
pid,command -u oracle.
The first two processes belong to the Oracle Process Manager and Notifier
(OPMN), which is the main controlling component of Oracle Application
Server. In the case of Oracle Application Server, OPMN controls all other
processes except those belonging to the Enterprise Manager.
6155 /opt/oas/opmn/bin/opmn -d
6157 /opt/oas/opmn/bin/opmn -d
There are two Web Cache processes. If you look at the open port list and
compare the process IDs you will find that one process is used to process
HTTP requests (including invalidation and statistics), while the other is used
for the control operations and to run the Web Cache Manager.
There is one Java process in which OC4J runs. (It is very unfortunate that
Java processes do not have meaningful names; the process list would be
much nicer to look at.) In an effort to make the process list useful we have
removed most of the content from the command line, leaving just the
important part - the value of the -jar parameter. This parameter contains
the equivalent of process name in Java.
A number of processes are used by the web server. The first process is the
main web server process (the "Parent Process" in Figure 2). It is followed by
two processes that are in charge of log rotation, one for the access log and
the error log, respectively. The fourth process is the FastCGI process
manager, which, if FastCGI is used, dynamically creates and destroys
To configure Web Cache to run as root you first need to stop it, and then
execute the following commands:.
# cd $ORACLE_HOME/webcache/bin
# ./webcache_setuser.sh setroot oracle
# cd $ORACLE_HOME/Apache/Apache/bin
# chown root .apachectl
# chmod 6750 .apachectl
Because the apachectl script simply calls the apachectl script it is a good
idea to change the ownership of the apachectl script to root to prevent
anyone from modifying it. Otherwise you would be allowing anyone with
write privileges over apachectl to execute any comand as root!
references to other files. You will need a good knowledge of the Apache web
server together with access to Apache documentation in one browser window
and Oracle HTTP Server documentation in another. If you encounter
something you don't need simply comment it out. There is a slight chance
you will remove something useful, but if you perform this step right after the
installation (as opposed to right before production) any problems you
introduce now should be discovered in the development and testing phase.
Oracle Application Server comes with many examples that must be removed
because they do contain vulnerable code. The following is a complete list that
applies to Oracle Application Server 10g R2 (10.1.2.0.2):
After removing the files you may also want to remove the following aliases (if
you won't be using them, that is):
/perl/
/fcgi-bin/
/cgi-bin/
You will need to change the aliases in the configuration files (directives
Alias, ScriptAlias, Oc4jMount).
It is a good idea to avoid adding content to the default location, and instead
creating a new virtual host. You may think if you create a new virtual host for
your application you won't need to go through the boring process of
removing all those examples. You may be surprised to learn that even in that
case the examples would follow you and appear on the new virtual host, too!
This is because the examples are defined in the body of the main server. It is
the configuration of the main server that is used as a template for virtual
hosts; virtual hosts will simply inherit the configuration. In practice this
means that unless you remove them, all the aliases from the main server will
be present in the virtual hosts. This will be true even for the OC4J aliases
because the value of the Oc4jMountCopy directive is On by default.
directory contents, but there is room for two more improvements. First, you
want to remove the web server information from the error pages by changing
the value of the ServerSignature directive (in httpd.conf):
ServerSignature Off
There is another place where web servers usually put their make and model,
in the Server header of every response. To minimize the amount of
information disclosed there, add the following line to httpd.conf:
ServerTokens ProductOnly
The name of the web server will remain visible but the version number will
be protected.
expose_php = Off
The main candidates for removal are, of course, the ones that contain large
chunks of functionality. Remember, the more complex the module the more
likely is to contain an undiscovered problem of some kind. Suggested
modules to remove include:
Proxy (mod_proxy)
Perl (mod_perl)
CGI (mod_cgi)
FastCGI (mod_fastcgi)
PHP (libphp4)
PL/SQL (mod_plsql)
WebDAV (mod_oradav)
SSL (mod_ossl)
To remove a module you need to find and comment out the corresponding
LoadModule directive that loads it from a shared library, and then look for a
similar AddModule directive (which is sometimes not used). If the
There is little to gain from removing some of the other default modules
because they don't contain much code. Some of them are also quite useful so
you may happen to use them in the future after all. Still, if you are very
paranoid you may want to consider also removing the following modules:
mod_log_agent
mod_cern_meta
mod_log_referer
mod_usertrack
mod_speling (that is the correct name)
mod_imap
mod_info
mod_vhost_alias
Configure limits
Properly configured web server limits are essential for the stability of the
system. You do not have to be attacked to experience problems. Problems
with limits can happen for a variety of reasons: too many users (or a few
very demanding users), badly written applications (use too much of your
resources to process one request), traffic spikes, and so on. Ultimately, if you
allow too many requests to be processed at the same time the web server
may attempt to allocate too much of the operating memory, which will cause
the machine to start swapping, and generally make the machine
unresponsive. It may even crash completely. When you do get attacked,
properly configured limits will make the server more resistant. In the worst
case the web server may become unusable, but the machine will survive,
allowing you to access it and implement manual defense systems.
Timeout 60
This setting is used in various places in the Apache code, and even in the
third-party modules. The default value (300 seconds) is far more than
necessary in most situations. You are advised to reduce the value to at least
60 seconds. If you experience problems (unlikely, but that depends on your
circumstances), you can always increase it to a larger value.
There must be a upper limit on the number of Apache instances active at any
one time. This is because each instance consumes some amount of resources
on the server. The upper limit must be put in place to protect the server from
running out of the resources. This means that if there is a maximum limit of
150 instances at any one time, and all of them are engaged, the next
connection request that arrives will simply not be processed until one of the
instances closes a connection. In most cases the problem is manifested by a
long wait experienced by the users. In extreme cases the connection
requests will simply be lost.
The other problem is related to the Keep-Alive feature of the HTTP protocol.
In the early days of HTTP each request was processed on its own TCP/IP
connection. This wasteful approach placed increased performance
requirements on the servers, and increased latency. The Keep-Alive feature,
introduced in HTTP/1.1, allows a client to maintain a connection with the
server across many requests. It sounds like a good idea, but it does not work
well in combination with the Apache processing model which was described
earlier. Remember how there is one instance of Apache dedicated to each
connection? A typical client will not simply send its request one after another
KeepAlive On
KeepAliveTimeout 5
The exact method you will use to control the maximum number of requests
processed at any one time depends on the processing model. The
MaxClients directive is used with the prefork model:
MaxClients 150
ThreadsPerChild 50
There is no easy way to determine the correct value for the maximum
number of instances. You must choose one using your best estimation and
basing that on the information you have from testing and production. Ideally,
you would minimize guessing by performing a proper testing phase and
simulating a varying number of clients accessing your system. You will arrive
at the correct value by monitoring the system parameters (memory
consumption, CPU utilization) and choosing a value that yields maximum
performance but which also keeps the server stable. As a rule of thumb, if
you don't have time to do proper testing, choose a lower value and increase
it if you encounter the limit.
To lower limits you will use the four Limit directives. Again, it is difficult to
choose a one-size-fits-all value here, although we have tried to provide
decent recommended values below:
Your request body sizes are likely to be larger than 65536 bytes if you plan
to support file uploads. In that case, increase the value accordingly. (But
whichever value you choose it is likely to be better than the default value of
unlimited.)
Enhance logging
By default Oracle HTTP Server will produce the access logs in the so-called
common format. At the very least you should change the logging format to
combined, which starts as common buts adds the referrer and the user agent
information at the end. In the interest of security, also consider talking to
your developers and working out a way for the web server to log the session
ID to the logs somehow. This will help you immensely if you ever find
yourself investigating a potential breach of security through the web
application.
As for the error log, by default levels warn and higher are logged. You may
want to change this to info, as that level provides much more information.
LogLevel info
Note:
Oracle Application Server comes with Log Loader, an optional component
used for log storage. Log Loader reads the error messages produced by other
components and stores them into its own repository. However, Log Loader
will not take care of the access logs. This means you need to establish your
access logs retention policy as part of the installation process.
SSL
If you are deploying SSL on the web server level (as opposed to terminating
SSL at the Web Cache) you may want to disable certain protocols and
ciphers. When we speak of SSL we usually refer to versions 2 and 3, and to
the TLS 1.0, which is commonly accepted to be SSL v3.1. However, we now
know that version 2 is not very secure. Unless you have really old clients that
you must support (and they don't understand SSLv3 or better) you can
safely disable SSL v2.
You may also consider looking at disabling certain low-strength ciphers too,
as illustrated in the following example:
SSLCipherSuite All:!EXP:!NULL:!ADH:!LOW
UseWebCacheIP On
Since Web Cache "understands" this problem it always sends the real client
IP address in the headers of the requests sent to the web servers. All
UseWebCacheIP directive does is makes use of this information and "tricks"
the web server into thinking the client is behind the IP address provided in
the headers.
You need to choose the correct setting for UseWebCacheIP after the
installation. It is set to Off by default, which is correct (assuming the web
server receives requests directly). However, if you install the web server
together with Web Cache and leave the value at Off the access controls that
are configured on the web server will no longer work correctly.
As an example, if you have installed both the web server and Web Cache on
the same machine, try accessing the server status page
http://servername.example.com/server-status/ (replace
servername.example.com with your own URI or IP address) from any
computer other than the server itself. You should be allowed access to the
inner workings of the web server, but this is not what you would expect. The
server status page is configured with:
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from localhost SERVERNAME SERVERNAME.EXAMPLE.COM
</Location>
Do you see the problem? Because of Web Cache being on the same machine,
the web server will always use the local address for its access control! The
server status is not the only vulnerable example. There are other OAS
components that expect some requests to come only from the localhost.
Changing UseWebCacheIP to On solves this problem.
<Location /internalStuff/>
Order deny,allow
Deny from all
Allow from LOCALNETWORK
</Location>
If it happens that the first request for the resource comes from the internal
network (and assuming UseWebCacheIP is set to On), the web server will look
at the IP address and allow the request to go through. Since the response
travels through Web Cache it will observe and cache it, seeing no reason not
to. The resource in question will be delivered straight from the cache for all
subsequent requests until it expires from cache.
You can resolve this problem simply by not relying on network access
controls on the web server level. Use passwords instead. Or, if you really
must use network access controls, use mod_headers to tell Web Cache the
content is not cacheable:
<Location /internalStuff/>
Order deny,allow
Deny from all
Allow from LOCALNETWORK
Header set Cache-Control private
</Location>
Limits
The following aspects of Web Cache functionality can be limited:
SSL
Disabling SSLv2 in Web Cache is not as straightforward as it is for the web
server. Although it is supported in the configuration, the web-based GUIs
(neither the Enterprise Manager nor the Web Cache Manager) do not allow it.
You could edit the file $ORACLE_HOME/webcache/webcache.xml directly, and
find all lines like the following (there will be one for each SSL-enabled port):
Then change SSL to SSLV3_V2H (possible values are NONE, SSL, SSLV2, SSLV3,
and SSLV3_V2H):
Unfortunately, any changes you make this way will be overwritten the next
time you choose to configure the ports through a GUI. Consequently, you
should only spend time doing this for cases where very high security levels
are required.
Advanced Hardening
Before you complete the installation and the configuration process, you have
one final opportunity to decide whether you want various Oracle Application
Server components to run under different user accounts. This discussion only
applies when you have all the components on the same machine. If you have
multiple machines to play with, skip this section and read the "Network
Security" section instead. We will also warn you in advance that running as
separate user accounts, although possible, is not at all straightforward.
Things you will need to fix will break in the installation. Ultimately you will
need to make up your own mind whether the end result justifies the effort.
But at the very least the discussion below may save you a lot of time trying
to get it right. Therefore consider it as a set of guidelines and be prepared to
improvise.
There is a conflict between the fact that that you want components to run
under different user accounts, but the main account (the one used to install
Oracle Application Server) still needs to maintain control over the files. This
creates all sorts of problems. For example, if you change the default
permissions, when the time for an upgrade comes you may need to
(temporarily) change the permissions back to the original values.
If you really want to benefit from the separation of components and you
want them to run on the same machine, it may be a better (and cleaner)
solution to simply install standalone versions of each component and
maintain completely separate installations. This approach will produce the
least number of problems. On the other hand, it does make it more difficult
to connect the components together and to control them from only one GUI.
Note:
Create a separate account for every component. Do not use the nobody
account. If you use the nobody account for two or more services and one of
them gets compromised, the attacker will be able to take control of the
remaining services running using the same account. Keep in mind the new
accounts must at least have read access to the installation directory.
to
After Installation
If you have made changes to the process list in the hardening phase you now
have a good opportunity to record the running processes and the listening
ports. This information should become part of the installation report. Other
things you may consider doing:
Maintenance
Maintenance is often a neglected part of the deployment process, probably
because it is a very unattractive activity. Still, it is a vital activity to maintain
the required security levels. There are three general rules of thumb to follow
in this section:
Limit user access. Since OAS is a powerful beast you are not likely to install it
on a multi-purpose machine but instead only allow access to those who need
it for administrative work. Lock down all other accounts. You should not allow
many users to use the same account. Configure the machine to force each
administrator to log in with a separate account. Once on the machine they
can change to the account they need to perform the work.
Deploy minimal services. Each service you leave running is a potential entry
point for the attackers. Shut down the services that are not needed.
Patch regularly. Buffer overflows and other vulnerabilities are still a reality. If
you patch regularly you keep the window of opportunity for an attacker down
to a minimum.
Configure user limits for user accounts. If you do this, misbehaving processes
won't be allowed to take down the whole machine.
Deploy a host firewall. Not only will this protect against misconfiguration at
the central firewall, but will protect the machine from the hostile traffic on
the internal network. Deny everything by default, then allow only the ports
that are really needed (e.g. 80 and 443 from the public, 22 and 3389 from
the administrative stations). Although this step may take a bit longer to
complete it will substantially improve the security of the host and the
network as a whole.
Deploy file integrity monitoring. This task, too, can require a significant
amount of time, but it is the only way to know for sure if you have been
infiltrated or not.
Network Security
Taking another step back from Oracle Application Server leads us to network
security. Network security is a vast subject - we will not attempt to cover it,
not even in bullet points. Instead, we will look at the mechanisms supported
by Oracle Application Server to increase security by breaking up the system
into standalone components. This is only a brief overview of what is well-
explained in the "Recommended Deployment Topologies" of the Oracle
Application Server Security Guide.
Although all components are installed onto the same machine by default, you
dont have to accept this as your deployment architecture. Most Oracle
Application Server components can be installed in a standalone format from
the Application Server Companion CD. For reasons of scalability and security
you should consider splitting the various components onto completely
separate machines, and even completely separate network segments. If your
installation size and budget do not allow for separate machines, consider
using virtualization technologies to maintain separation while saving on
hardware costs.
Use the port tunneling Do this to reduce the number of ports that must
tool distributed with OAS be left open. See "Understanding Port Tunneling"
in the Oracle HTTP Server Administrator's Guide
for more information.
Conclusion
With the exception of the code examples, the default configuration of Oracle
Application Server 10g R2 is generally secure. Working against OAS,
however, is the complexity of the product and the large number of
components that are installed. To maximize security, administrators need to
carefully review each installation before production, and make the right
changes to suit their unique circumstances. The key to success is to gain a
full understanding of the software stack and the interactions between each of
the various components. This document contains step-by-step instructions
that point to the areas of interest and provides a framework to secure Oracle
Application Server.
There are some areas we would be happy to see improve in the future. Doing
so would increase security and make maintenance of Oracle Application
Server easier. Suggestions include:
Whether a security breach is made public or confined internally, the fact that
a hacker has accessed your sensitive data should be a huge concern to your
company, your shareholders and, most importantly, your customers. S.P.I.
Dynamics has found that the majority of companies that are vigilant and
proactive in their approach to application security are better protected. In the
long run, these companies enjoy a higher return on investment for their e-
business ventures. Delivering secure web applications can bolster an
organizations bottom line by:
Mitigating risks: What would be the financial cost of a successful attack upon
your web application? Of application downtime? Of fines levied because of a
breach of private consumer information? From delays in shipping and billing
SPI Labs continuously maintains the worlds largest database of more than
5,000 application layer vulnerabilities and attack methodologies. This direct
research is utilized to provide daily updates to S.P.I. Dynamics suite of
security assessment and testing software products. SPI Labs engineers
comply with the standards proposed by the Internet Engineering Task Force
(IETF) for responsible security vulnerability disclosure. Information regarding
SPI Labs policies and procedures for disclosure are outlined on the S.P.I.
Dynamics Web site at: http://www.spidynamics.com/spilabs.html.
ISSA, one of the founding visionaries of the AVDL standard within OASIS,
and a founding member of the Web Application Security Consortium (WASC).
Contact Information
Pre-Installation
Understand the components and the architecture of
Oracle Application Server.
Configure limits
o Timeout
o Keep-Alive Timeout
Enhance logging
o Access log
o Error log
Disable SSLv2
o Keep-Alive timeout
Patch regularly
Appendix B: Bibliography
Apache httpd 1.3 documentation (http://httpd.apache.org/docs/1.3/).
Apache Security by Ivan Ristic (O'Reilly). Two chapters relevant for this whitepaper
are available for free download from http://www.apachesecurity.net.
Oracle Application Server 10g Essentials by Donald Bales, Rick Greenwald, and Robert
Stackowiak (O'Reilly).
Special OPS: Host and Network Security for Microsoft, UNIX, and Oracle by Erik Pace
Birkholz et al. (Syngress).