Data Center Threats and Vulnerabilities
Data Center Threats and Vulnerabilities
Jonathan A. Zdziarski
jonathan@zdziarski.com
Abstract
Data center facilities are at the heart of today's electronic infrastructure, giving life to a significant
percentage of online commerce. Due to their planned construction on critical infrastructure, such as
converging power grids and dense telecom networks, they are also, however largely unguarded potential
targets for terrorists. Data centers are among the only facilities located globally within proximity to mission
critical infrastructure, where even a lightly funded individual is clear to install heavy equipment virtually
unchecked and undisturbed for long periods of time, with access to a remote high-QoS network (the
Internet) as a detonator. Because private data centers are largely ubiquitous in their procedures, which
typically do not include hardware inspection, this would allow a terrorist group to launch a national,
distributed, coordinated attack taking out many large cities' infrastructures simultaneously, and without
prior detection.
1. Introduction
Data center facilities are commonplace in today’s vast telecommunications infrastructure
as an inexpensive solution for high speed Internet access. These and other such hosting
facilities provide cost effective connectivity for computer equipment in a remote, climate-
controlled environment, usually with several redundant high-speed connections to the
Internet. These facilities are responsible for a significant percentage of electronic
business performed in the United States and other countries and provide hosting for
organizations ranging from financial institutions to adult websites and spammers.
Prior to September 11, 2001, the notion of an attack using data center facilities seemed
infeasible to most of the private sector, however this viewpoint has changed, and these
facilities now appear as a logical target for terrorists, allowing a group to potentially
launch a successful, distributed attack against surrounding critical infrastructure. Many
data center facilities are strategically placed in areas where significant commerce is
occurring in order to take advantage of the abundant infrastructure in the area. Along
with these come major Internet exchanges (peering points), large corporate
concentrations, and many possibly high-profile targets such as financial institutions, news
buildings, and political targets. Some facilities are within immediate proximity to targets
such as the New York Stock Exchange, the CNN building, and the public and private
networks that are responsible for the Internet as well as military and public service
networks. Due to the placement of such facilities, they are more than likely targets.
Data center facilities are among the only facilities located globally within proximity to
mission critical infrastructure where even a lightly funded individual is able to install
heavy equipment virtually unchecked and undisturbed for long periods of time with a
remote, high quality of service (QoS) network. Too, this network can be synchronized to
launch attacks involving multiple simultaneous targets can be coordinated from anywhere
in the world or made to operate autonomously based on specific predetermined events,
including for example:
• A daily news search for a specific event, and a match threshold
• The price of a certain share of stock
• Extended dark periods in reaching a specific Internet Protocol (IP) address
• Simple date and time of the machine
To summarize, data centers are an ideal target for terrorists to use because it provides all
of the important ingredients for a distributed, coordinated attack:
Such an attack in one single area would easily leave thousands without electricity,
emergency services, and other such critical services. Historically, these conditions
frequently lead to a significant loss of commerce, high rate of crime and possible loss of
life.
2.1 Components
A data center is typically a large, spacious facility located either in a dedicated building
(also referred to as a “server farm”) or leased space within an office. Slices of space are
leased to the data center’s customers, who are responsible for moving their equipment
into the facility, setting it up on racks or in cages, and connecting it to (usually backbone)
networks provided by the facility. Most facilities are comprised of the following
components:
Most facilities are unmanned, and remotely managed from a network operations center
(NOC). This NOC may be located in an adjacent room, on a different floor, or in a
nearby building. The NOC is usually responsible for the electrical and environmental
state of the entire facility including its network, hosted equipment, and etcetera.
1. Telco and equipment technicians, fire marshals, building personnel and other
authorized technical or building staff.
2.2.1 Maintenance
Maintenance personnel seeking access to the facility will generally be required to provide
an identification card and/or information about the service call. Because the maintenance
may be on behalf of a specific customer, an access ticket may be required to permit the
individual into the customer’s cage or cabinet. An individual performing maintenance on
the facility may be from one of many local carriers with hardware in the facility, an
electrician configuring a rack for a customer, fire marshal (who frequently inspect
facilities), or a technician from a customer sent out to service a system. Guest cards are
usually granted to the technician upon arrival, or the technician is escorted into the
facility and left to work.
2.2.2 Customers
The standard procedure to procure rack space in a data center is to sign and pay on a
contract (usually annual) through the company's sales representative. The customer's
network access will then be provisioned and rack space assigned. Access control for
customers is typically formal. The customer may be granted access cards, a key, or other
means to access the facility at which point it will be their responsibility to install their
equipment and bring it online. Biometrics may be in use as well to prevent access card
theft.
Because a majority of data center facilities are unmanned (perhaps with the exception of
a security guard), it is the customer or technician's responsibility to conduct themselves in
a professional manner by not stealing or sabotaging the equipment of another customer or
vendor. Frequently, this is left to physical security by means of caged equipment or
locked cabinets. Should an individual visit the facility during a lull time (the evening, for
example), one may be virtually alone. The individual will usually bring the necessary
equipment in on a cart, swipe a card, possibly sign in, and be cleared for access to the
facility. Due to confidentiality concerns, customers are typically not monitored unless
there is a pressing need.
2.3 Presupposed Innocence
Security staff and procedures are in place to ensure that unauthorized individuals are not
permitted into the facility – to discourage an individual from an overt attack such as
walking in with a baseball bat and smashing equipment. Security staff is not by any
means equipped with the ability to determine if a chassis is armed with explosives or
sometimes even if the equipment a customer is working on belongs to them. The
perception to staff working in a data center facility on a daily basis is typically limited to
access only and not to equipment. Customers and their equipment is presupposed to be
innocuous, even by engineers in the facility - and is therefore rarely ever challenged.
Authorized guests of the facility will occasionally introduce large pieces of heavy
equipment into the facility on behalf of a customer or telecommunications provider.
These can include enterprise-class servers or mainframes, network or telecom equipment,
large batteries, and possibly even customer-owned hardware closets. It is not uncommon
to see computer systems five or six feet high with large locking doors for disk storage or
processor blades nor is it uncommon to see a group of deep-cycle batteries with bare
terminals connected to equipment. Other equipment involved usually includes copper and
fiber optic cabling, miscellaneous small devices, and possibly even the customer's own
remote access devices. All forms of this equipment are considered innocuous due to its
necessity in the telecommunications space.
• Names of some customers and possibly (via labels) IP addresses and hostnames of
some of their equipment.
• A relationship with the sales representative, who may also introduce him to
facility staff, establishing a level of familiarity.
• Location of most or all security cameras, if and how they are recorded, and the
frame-rate and size of the capture.
• Understanding of the general security in the building, entry practices, hardware
policies (if any), and etcetera.
• Knowledge of any businesses in the facility that would make good targets
• Knowledge of the part of the city they are located in and what targets are nearby,
what power grids and telecommunications networks support the facility, etcetera
• How much space the attacker will have to place their equipment, and where
• Typical hours of operation, staff rotation, and other internal operational data
Once the attacker has this information, they can make an even better judgment about
whether or not the facility is a suitable point of attack and select a method of insertion.
4. Exploiting Facility Weaknesses
In this section, we will discuss the various weaknesses in many data center policies and
illustrate some possible scenarios of how a malicious group could possibly exploit them.
Because many visits (particularly emergency visits) can go unannounced at the last
minute, there is a risk of error in procedures to verify the individual accessing the facility.
Proper procedure should involve:
There are some significant weaknesses, which can be exploited to allow an unauthorized
individual to gain access to a facility and introduce new equipment, as we’ll illustrate
below. Facilities enforcing the proper policies in place bear a lower risk to such an attack.
The typical social engineering attack begins with introduction and can be followed-up by
a trust-building exercise, where the malicious actor builds a presumed trust relationship
with the target, ending with a clandestine and undetected attack. Some examples of
potentially effective scenarios include:
In this scenario, the actor poses as an engineer for a company providing backbone
Internet services to the facility. Data centers typically use multiple telecoms to provide
bandwidth to its customers, or customers can alternatively choose their own. It is
typically easy to identify which telecoms a data center is using by reading their website
or contacting a sales executive. Telecoms also place premise equipment within the data
center to terminate the customer side of their connections, and may even stencil their
name on conduit covers outside the building. Posing as such an engineer, an initial phone
call will be made from a mobile phone or possibly even from a telecom-owned pay phone
(where the caller-id will appear as the telecom’s), or from any other location using a
caller-id spoofer. The data center will be informed that there is an open repair ticket
pending and may make up a reason, such as an alarm code sent from their equipment.
Because telecoms are so large in size, follow-up phone calls may be hit-or-miss, and are
frequently overlooked. Tight procedure would require a phone number on file be used for
follow-up, but in many cases this information is either not stored on file or the attacker
convinces the data center staff to use an alternate number. Once granted access to the
facility (using a phony identification with a logo from the telecom’s website), the actor is
free to plant equipment in the telecom rack and possibly set up network or out-of-band
access.
In this scenario, the actor is dressed professionally in a dark blazer with a walkie-talkie
can easily pose as security or building personnel at some non-dedicated facilities located
within office buildings. Building management for office buildings are, in most cases,
required to have full access to all rooms. Because a building is responsible for electrical,
fire alarms, and sometimes ventilation, individuals looking to socially engineer their way
into a particular office (including a data center) might take advantage of the obscurity
between building management and the data center. Using ploys such as fire alarms
malfunctioning or electrical problems, the malicious individual may convince staff to
grant him a guest card (for working through the problem throughout the evening) or even
prop the door open. While building management typically isn’t expected to introduce
new equipment, a device could be concealed inside a toolbox introduced with a
secondary attack (e.g. escorting an electrician with the now-trusted building manager). A
smaller device could be introduced as well, possibly to simply plant on a door lock for
future entries.
Because expensive equipment is typically stored within a data center, many customers
purchase third-party service contracts to perform repairs of defective servers, networking
gear, and other equipment. In order for the actor to pose as a phony service technician, a
one would need to manufacture a false identification. This can be done with a laminator
and logo from the company’s website. Because most procedures do not require the
customer to list all possible vendors for their facility’s equipment, the attacker could pose
as a technician from any high-profile company such as Sun Microsystems, Cisco, IBM,
or others. This attack begins with a proactive telephone call posing as a customer, and
requires knowledge of which customers are hosted in the data center facility. This
information can usually be gained through discussion with one of a data center’s account
executives during reconnaissance, who, in order to close a sale, might drop a few names
of customers hosted in the facility. If procedure is not followed and a phone number
already on file (or a challenge) is not used to verify, the actor can set up access for a
second actor, posing as the vendor, who will follow. This actor will be granted access to
the facility and the customer’s cage, where they may introduce new equipment or
sabotage the customer.
Data centers are a corporate business, and it is both competitive and financially beneficial
to expedite the initial installation time for new customers. Typically, little or no
requirements, other than financial and contractual obligations, are required in order to
procure rack space in such a facility. Because terrorists are organized and funded,
purchasing facility space is not be a problem. Many facilities will lease space for as little
as $500 per month. The corporate strategy behind these facilities also caters to new
business, which requires these facilities’ services immediately. While some facilities have
basic safeguards in place, such as refusing cash payments or signing in new equipment at
the door, it is still quite simple to introduce concealed explosives or other devices into a
facility as a customer. After an initial reconnaissance evaluation, the attacker may
become a customer of the facility at which point a contract will be filled out and initial
payment made via check or credit card. Once this is complete, the provisioning process
begins and the rack space is assigned. In many cases, this entire process can take less than
24 hours.
NOTE:
Pre-Paid Credit Cards are now available, leaving very little if any
trail. Organized terrorists may not even require such cards as they
may be in possession of laundered card, appearing legitimate, but
be suspicious of prepaid cards. Most data centers do not have
procedures in place to identify or refuse prepaid credit cards.
NOTE:
I recently received an email informing me that the gamma
radiation from a nuclear device will cause fiber to turn opaque.
Although this can be avoided with the proper shielding, this can be
a detection approach taught to staff, as most facilities have some
form of exposed fiber connected to equipment. Small radiation
detection devices can also be purchased to detect radiation in a
room.
Due to the large and heavy nature of most enterprise class computer equipment, plenty of
free space is available in any large computer chassis to accommodate such devices as
well as keep them cool, concealed, and stable. The Sun E450 is an ideal example of such
a chassis, as a significant amount of free space is available for drive arrays and would
require little or no modification to hide such a device. Other choices may include a Cisco
12000 series router, most RAID chasses, or any large "stackable" type computer systems.
Using the serial cable or other interfaces, these devices can be connected to a computer or
directly connected to one or more of the data center’s Internet carriers. Some of the
possible ways such devices could be smuggled in include:
In most cases, the device can be hidden in plain site. Once the attacker has introduced
their equipment, it will generally go undisturbed by staff. A majority of facilities have no
hardware inspection policies or procedures and most facility staff is forbidden to handle
customer equipment except in an emergency (e.g. fire), to avoid liability. Some facilities
do have very strict policies regarding exit hardware, however these do not generally
extend to entrance hardware.
5. Detonation of Equipment
Data centers render timers and RF detonators obsolete. The key commodities in a data
center are space and high speed Internet access. As the business involves Internet
connectivity, very high-speed connections are provided to the customers. Connecting this
to the computer that has been introduced provides a remote means of detonation from
anywhere in the world over any layer of encryption the attacker finds necessary. Should a
small nuclear device have been smuggled in, detonating the equipment from inside the
building could easily take out the city block or more. This can also be used to launch a
collaborative attack involving multiple large cities as targets. Ticking bombs with daily
or weekly reset switches may also be used, ensuring that isolation will not prevent
detonation.
A more complex attack may include an automated scan of news sites for a threshold of
matches to particular key words such as "Bin Laden Captured". Once the system has been
connected to the Internet, it can be trained to detonate on any event ranging from the
system date to the score at the last Yankee's game. There is virtually no way to detect
such monitoring tools.
6. Possible Solutions
Now that we have discussed various vulnerabilities, let's take a look at some of the ways
security can be improved in these co-location facilities. Below are some suggestions,
which can help as both deterrents and detection approaches.
All facilities should make every attempt to know who their customers are and where
they've come from. Having background checks part of the standard contractual obligation
reduces the chances that anyone will become offended. Visiting their place of business
and learning as much about the customer are also ways to insure that you're not doing
business with the wrong group. This will force attackers to build long term relationships
with their targets, establishing a corporate presence, draining their finances, and making
it all the more possible for them to become exposed.
New customers should provide more specific information about their business. Are they a
web hosting company or an online trading company? What do they intend to use the
hardware in the rack space for? If the customer appears to be unfamiliar with their own
business, this can help set off an alarm to investigate the customer more thoroughly.
Some other questions to ask include:
Some information can be discerned from common interaction with the customer. If the
company has been around for several years, have offices, and a staff then it is most likely
a legitimate business (or an elaborate front). On the other hand, if the company is a new
virtual company with a residential address, there is significant reason to be concerned.
Data center facilities already have the liberty to discriminate based on many other criteria
such as whether or not the company sends spam, broadcasts pornography, etcetera.
Ensuring the customer has a justifiable business is certainly a responsible approach to
accepting new customers.
Facility contracts should be worded in such a way to allow for a one or two-week
provisioning period. During this period, all hardware inspections, background checks,
and other such checks can be performed.
Explosive detection tools are available as well to scan new hardware. K-9 officers can be
trained to detect up to 11 distinct odors of explosives. An occasional pass-through of the
facility by a trained K-9 officer can provide a non-intrusive way to check out new
customers without the need to even touch their hardware. Such inspections may even be
provided at no charge as a public service by the police department depending on the area.
Other actions such as checking underneath the raised floor for any devices, closely
monitoring a customer's actions and behavior, and scanning the room with a thermal
camera are all less intrusive ways to keep an eye out for suspicious activity. Finally, swab
tests can be performed in almost any setting to detect explosive residue.
Paying attention to the kind of hardware the customer brings in is another good way to
identify suspicious activity once the account executive builds a relationship with them
and knows their business. Is the user a dialup Internet provider? Why do they not have
any dialup access equipment such as modem banks? Are they a web hosting company?
Why do they have only one large server instead of several small ones? Is the traffic they
are pushing commensurate to the business they are running and the amount of bandwidth
they purchased? Are they pushing any traffic at all? As small as they are, these
inconsistencies can pile up and help to identify a customer who is not really who they say
they are (even if one is just looking for spammers rather than terrorists).
NOTE:
In some states, such as New York, laws exist requiring that
rooftops to any Federal builds be secured.
7. Conclusion
Data centers are an ideal terrorist target. It is of the utmost importance to take the
necessary steps to protect these facilities from an attack. Detecting the vulnerabilities in
these facilities is the first step. Once they are exposed, finding an effective plan to fix
procedure and inspection will help make the facility a secure place of commerce rather
than the next target of attack.
Security for these facilities is ultimately left up to the facility maintainer. The FBI may be
able to help improve the security of such locations by cooperating to assist in detection
and defining policy and procedure.