Database Middleware and Web Services for Data

Distribution and Integration in Distributed

Heterogeneous Database Systems
Han-Chieh Wei Travis Godfrey
Computer Science Department Computer Science Department
University of Central Arkansas University of Central Arkansas
Conway AR, USA Conway AR, USA

federated databases may consist of not only multi-vendor

Abstract - As the database environment changing,
relational databases, but also non-relational databases
database is no longer accessed only by using SQL to
such as object-oriented databases, ASCII data organized
communicate with DBMS as in the two-tier client-server
in flat files, or even XML databases. Such autonomous
architecture. In modern enterprise distributed systems,
data sources have no ability to relate data from these
applications are developed and deployed as components
heterogeneous data sources within the organization. The
in n-tier architecture. Database system is integrated with
difficulties result from the diverge of data types, data
related applications and can only be accessed via
representations, data manipulation languages (DML),
interfaces provided by the application server or
data definition languages (DDL), transaction
middleware. Moreover, in most of the middleware
management, and security. These difficulties cause
systems such as CORBA, J2EE, and RPC, clients and
serious problems for data distribution and integration in
servers are tightly-coupled and client applications can
enterprise applications.
only invoke methods by using proprietary communication
protocols. As a consequence, data integration is no
Middleware is a layer of software whose purpose is
longer achievable by applying existing database
to mask heterogeneity and to provide a convenient
middleware systems. What we dealing now are not just
programming model to application programmers.
different DBMS vendors, dialects of SQL, or variants of
Middleware is represented by processes or services in a
data models, but also heterogeneous interfaces and
set of computers that interact with each other to
communication protocols defined in various distributed
implement communication and resource sharing for
paradigms. In this paper, we propose ADIM, an
distributed applications. With the same purpose, database
autonomous data integration middleware, with
middleware systems are used to integrate heterogeneous
WebService for data integration in heterogeneous
data sources distributed over computer networks. Client
environments.
applications serviced by the database middleware are
provided with uniform and transparent view and access
Keywords: Database Middleware, Data Integration,
interfaces to the distributed data. Whenever an
Distributed Databases.
application wants to access data managed in a
1 Introduction heterogeneous distributed database, instead of writing
code to establish connections to each local and remote
Enterprises have been very interested in the data source and writing proprietary APIs for each data
decentralization of processing while achieving an source, the client applications use local API’s for data
integration of the information resources within the access provided by the database middleware. There is no
geographically distributed systems of databases. A need to modify or re-write the applications if there is
distributed database is a collection of multiple logically replacement or insertion of new data sources.
interrelated database distributed over a computer
network. Compared with centralized databases, Currently there are two main categories of database
distributed databases have the advantages of increased middleware systems: database mediator and database
reliability and availability, better performance, local gateway. The database mediator establishes connections
autonomy, expandability, and sharability [12]. Nowadays, to data sources and use wrappers to access and translate
distributed databases in organizations commonly are the information from the data sources into a global data
heterogeneous, i.e., data are organized and managed by a model. The database gateway establishes a point-to-point
mix of different data management systems from different connection to one remote database and importing data
vendors and different operating systems that use different into the local DBMS (Oracle, Informix, Sybase). Both
network protocols. In essence, the data source in such solutions have been adopted in commercial database
systems to solve the heterogeneous distributed DBMS distribution. For local autonomy, if there is no provision
problem. for the local site to function as a stand-alone DBMS, then
the system has no local autonomy. On the other hand, the
However, as the database environment changing, system is fully autonomous if the individual database
database is no longer accessed only by using SQL to systems are stand-alone DBMSs, which know neither the
communicate with DBMS as in the two-tier client-server existence of other DBMSs nor how to communicate with
architecture. In modern enterprise distributed systems, them. What exists in between is the semi-autonomous
applications are developed and deployed in 3-tier or n-tier system in which each DBMS operates independently and
architecture [2,3]. Database system is integrated with participates in a federation to share its local data. This
related applications and can only be accessed via type of distributed database systems is commonly
interfaces provided by the application server or categorized as federated DBMS [12] where there is a
middleware. Moreover, in most of the middleware global view or schema of the federations of databases that
systems such as CORBA, J2EE, and RPC, clients and is shared by applications and users at each site. For
servers are tightly-coupled and client applications can heterogeneity, it may occur in various forms in
only invoke methods by using proprietary communication distributed database systems, ranging from operating
protocols. Figure 1 illustrates an example of distributed systems, programming languages, and communication
environment. As a consequence, data integration is no protocols to variations of data models, query languages,
longer achievable by applying existing database and transaction management protocols. If all participating
middleware systems. What we dealing now are not just data servers follow the same form, the DDBS is called
different DBMS vendors, dialect of SQL, or different homogeneous; otherwise it is called heterogeneous. For
data models, but also interfaces and communication data distribution, the data is either stored at one site or
protocols defined in various distributed paradigms. physically distributed over multiple sites and
communicate each other over the networks.

Our proposed middleware approach is targeting at

the federated heterogeneous distributed database systems.
The data is distributed over multiple sites. Based on the
data model, data is managed by different data servers
which can be a full-fledged database server, a application
server, or even a file server providing access to flat files.
These servers may provide different access interfaces and
communicate with different protocols. Before presenting
the proposed middleware architecture, we first investigate
the requirements for a middleware to process data
distribution and integration in a distributed heterogeneous
database system.

2.2 Global database schema

An important goal of a distributed system is to hide
In this paper, we present an autonomous database
the fact that its processes and resources are physically
middleware approach for data integration and distribution
distributed across multiple computers. A distributed
in a distributed heterogeneous environment. In section 2,
system that is able to present itself to users and
we start with investigating the requirements for such
applications as if it were only a single computer system is
database middleware. The proposed architecture and
said to be transparent. For distributed database system,
implementation is presented in section 3. Section 4 gives
the access transparency hides differences in data
a brief overview of the related work. We finally conclude
representation and how a data source is accessed, where
the paper and give an outlook of the future work in
the location transparency hides where data source is
section 5.
located. To achieve data integration in a heterogeneous
distributed database with access and location
2 Background and Middleware transparencies, the middleware layer imposes a global
Requirements data schema on top of the individual schema used by
each data source. The global schema stores not only the
2.1 Types of distributed database systems global database structures but also the information of data
distribution such as address and specifications of vertical
Distributed database systems (DDBS) can be and horizontal fragmentations for each site. The global
characterized with respect to the degree of local schema also needs to include the access interface
autonomy, the degree of heterogeneity, and the degree of
information for each data source. This information 2.7 Scalability
enables the applications to transparently access the data
sources, no matter if they are stored locally or remotely, The scalability is evaluated as the ability to add
or can be accessed by means of SQL or APIs. With this heterogeneous data sources. The middleware should be
mechanism, client applications serviced by the able to handle the addition of data source and data servers
middleware system are provided with a uniform view and to the system and update the global catalog without
access interface to the data sets stored by each data having to modify the client applications.
source.
3 Overall Architecture
2.3 Client Interfaces To solve the problems described in section 1 and
To access data in a distributed database system, the fulfill the requirement listed in section 2, we propose the
middleware has to provide interfaces for user client or middleware architecture Autonomous Data Integration
applications. The possible interfaces mostly include GUI Middleware (ADIM) as depicted in Figure 2. There are
for users to pose queries against the database schema and four main components in ADIM: Global database
visualize their results, and APIs for programmer to Schema, Client, Server, and Web Service.
develop complicated applications. The query is
represented in a generic format and evaluated in the
WSDL
middleware. The middleware is responsible to connect to
the data source, get the result, and convert the data into
the format that can be displayed to the client. SERVERS
CLIENT OODBMS, RDBMS, FILES, XMLDB

2.4 Data Server Interfaces Schema API GUI QEP API GUI Schema
Repository Repository
QMP
Data Distribution &
The middleware has to implement interfaces for the Security Manager
Access Controller
Access Rights
Manager
data sources to be integrated. These interfaces must Query Processing Engine
Security Manager
overcome the heterogeneous communication protocols as XML Processor
XML Processor
well as heterogeneous platforms including operating
systems and programming languages. Since the results
SOAP
are typically returned in different formats, the interfaces CORBA, RMI, PRC

should translate them into the reference data model which

is used inside the middleware. In summary, the ADIM
middleware has to provide a homogeneous view across
Figure 2 Autonomous Data Integration Middleware
heterogeneous and distributed application interfaces.

2.5 Middleware Query Processing 3.1 Global Database Schema

As mentioned in previous sections, each distributed
For data integration, client applications issue queries
data source may implement different data models. To deal
against the global database schema. Providing with
with this heterogeneity, ADIM uses XML document to
location and access transparencies, client applications
represent the global database schema. Because of its
have no idea where the data located, how to connect to
extensibility, the XML document can also include the
the data server, or what are the access interfaces. All
distribution information such as rules for horizontal and
these tasks are handled by the middleware. The database
vertical fragmentations, addressees, and access interfaces
middleware decomposes the query and find
and protocols of distributed data servers. Figure 3 shows
corresponding sites for each data item which can handle
part of the global schema represented by XML document.
the request. The middleware makes query execution plans
In Figure 3, the XML document describes the table
and performs join operations if necessary.
definition, distributed sites and the local server and access
services.
2.6 Security
There are four levels of security can be defined [10] 3.2 Client Components
in the middleware architecture: the login right of the
There are four logical layers in the client
application to the middleware, the access rights of the
component: API/GUI, Security Manager, Query
middleware to the global schema, the login right of the
Processing Engine (QPE), and XML processor. For user
application to the foreign data server, and the privileges
clients, after logging into the middleware, the GUI
of the application to the foreign database.
displays the global database schema which is tailored
from the access rights of the local middleware to the
distributed database systems. The user can use the GUI to
issue queries against the global schema and visualize the
results. The queries will be parsed in QPE. Provided with
the distribution information from the global schema,
requests will be sent to data sources at which the
requested data located. As we will discuss in the
following section, all these requests are sent by using
SOAP [13] messages which are prepared in XML
processor. While receiving the results, the XML
processor parses the SOAP message for QPE to process
and the results either displayed on the GUI or returned to
the application program.

3.3 Web Services

As mentioned earlier, in a distributed heterogeneous
database system, different data sites may provide
different access interface for same request and
communicated with different protocols. For the database
gateway approach, it provides a point-to-point connection
to one type of database. In this case, each data source Figure 3 Database Global Schema
needs to implement one gateway for each of the other
data sources and communicate with their proprietary
protocols. As we can see from Figure 4(a), while this
approach solves the heterogeneity problem, it introduces
exponential complexity and limits the system scalability.
In ADIM, all the server access interfaces are wrapped
with Web Services and accessed with SOAP, as shown in
4(b), which makes the system scalable in linear
complexity. Web services [15] describes a standardized
way of integrating internet based applications using the
XML, SOAP [13], WSDL [16] and UDDI [14] open
standards over the Internet. Web services allow different Figure 4 (a) Database Gateway Approach, Exponential API
applications from different sources to communicate with
each other without time-consuming custom coding, and
because all communication is in XML, Web services are
not tied to any one operating system, programming
language, or communication protocols. In this way, it
does not matter whether the data is accessed through a
database server, Java server, CORBA server, RPC server,
or even a file server, ADIM can communicate uniformly
with SOAP. Figure 5 gives an example of the SOAP
messages for query and response. Currently there are
seven web services defined for the data server: Figure 4 (b) Web Service Wrapper, Linear Growth
UpdateSchema, Execute, ExecuteQuery, ExecuteUpdate,
PrepCommit, Commit, and Abort. Because of the space The server can develop more local services and
limit, we will not describe the detail implementation here. update the WSDL and also the global schema to be
The service names are pretty much self-explained. available for the middleware. We implement a SOAP
Wrapper Generator to automatically generate Java
programs as a bridge to the backend services. As depicted
in Figure 6(a), having CORBA as the backend service,
the generator reads input from the IDL file and a
configure file containing the host address and port
information, it will then generate Java programs acting as
SOAP service call to the CORBA and the also the WSDL
for middleware to access the service. The calling process
is shown in Figure 6(b). Figure 7 depicts the Java code
generated from the CORBA IDL and also the WSDL addresses, horizontal and/or vertical fragmentation rules,
definition. and site access authorization. These specifications will
then be added to the global schema, sent to each site, and
stored in the middleware schema repository.

Figure 5 Sample SOAP message

Figure 7 Web Service Wrapper

Figure 6 (a) SOAP Wrapper Generator 4 Related Work

There are related proposed architectures for
heterogeneous data integration [1,5,6,7,8,9]. Oracle
distributed database system uses the database gateway
approach (transparent gateway) to access non-Oracle
database systems. MOCHA [10] and [8] are middleware
based integration architectures that implement distributed
query processing for data and function integration in
Figure 6 (b) Call to the CORBA Wrapper federated database systems. CoDIMS-G [5] offers a
standard based communication paradigm based on Web
3.4 Server Component Service and distributed query processing for distributed
multi-database system in the grid environment. SkyQuery
There are also four logical layers in the server [9] is a Web Service based integration architecture for the
component: Query Execution Processor (QEP), Query integration of astronomy data archives available on the
Mapping Processor (QMP), Security Manager, and XML internet. It adopts Web Services as the main
Processor. The XML processor parses the SOAP communication and encapsulation interface both for the
message received from the client. The security manager query processor and data sources wrappers.
send login information to the server on behalf of the
client for authentication and authorization check. The
Query Mapping Processor (QMP) maps the query against 5 Conclusion and Future Work
the local definition for the Query Execution Processor In modern enterprise systems, databases and
(QEP) to extract data from the data source. The results applications are distributed over the network for the
will be translated into the global schema in QMP and benefits of better performance, economy, reliability, and
prepared in XML processor as SOAP response message availability. Nowadays, distributed databases in
and send back to the client. The GUI module provide an organizations commonly are heterogeneous, i.e., data are
interfaces for administrator at central control site to organized and managed by a mix of different data model
define data distribution specifications, such as sites and data management systems from different vendors and
different operating systems that use different network the Edge of the Internet, in Proceedings of the 13th ACM
protocols. With the emergence of component-based international World Wide Web conference, May 2004.
engineering [6], enterprise systems may consist of
application systems built with different distributed [4] Fernando de Ferreira Rezende and Klaudia Hergula,
paradigms, such as EJB, .NET, or CORBA. Databases “The Heterogeneity Problem and Middleware
encapsulated in such application systems can only be Technology: Experiments with and Performance of
accessed via provided APIs and connected using Database Gateways,” in Proc. VLDB Conference, pp.146-
proprietary protocols. In such highly heterogeneous 157, 1998.
environment, data integration by using traditional [5] V. Fontes et al. “CoDIMS-G: a Data and Program
federated database systems or database gateway are no Integration Service for the Grid,” in Proceedings of the
longer feasible or sufficient. 2nd Workshop on Middleware for Grid Computing, ACM
2004.
In this paper, we propose a database middleware
architecture ADIM (Autonomous Data Integration [6] Ondrej Galik and Tomas Bures, “Connecting
Middleware) with WebService for data distribution and middlewares: Generating Connectors for Heterogeneous
integration in distributed heterogeneous database Deployment”, in Proceedings of the 5th International
systems. The data schema integration is achieved by Workshop on Software Engineering and Middleware,
XML based global schema which is used by QPE and ACM, 2005.
QMP in the client and server components to translate and
exchange metadata between data sources. The [7] D. Heimbigner and D. McLeod, “A Federated
heterogeneity of access interfaces and protocols provided Architecture for Information Management.” ACM Trans.
by each distributed paradigm is resolved by adopting Office Information System, Vol. 3 No. 3, pp. 253-278,
WebService as the communication model for uniformity July, 1985.
of communication protocol in HTTP, message exchange [8] Klaudia Hergula, DaimlerChrysler AG, and Theo
in SOAP for service request and response, and service Harder, “A Middleware Approach for Combining
definition and description in WSDL. We also develop the Heterogeneous Data Sources - Integration of Generic
SOAP wrapper generator which can generate Java code Query and Predefined Function Access,” in Proceedings
to wrap the server API into Web Services. We have of the 1st International Conference on Web Information
implemented a prototype of ADIM in the distributed Systems Engineering, 2000.
environment described in section 2. We are now
continuing working on the improvement of query [9] Tanu Malik, Alex S. Szalay, Tamas Budavari, and
optimization and XML document join. We are also Ani R. Thakar. “SkyQuery: A Web Service Approach to
working on the two-phase-commit service and distributed Federate Databases,” Proceedings of CIDR 2003.
scheduler to extend ADIM to allow database update
operations. [10] M.R. Martinez, N. Roussopoulos. “MOCHA: A
Self-Extensible Database Middleware System for
Distributed Data Sources,” ACM SIGMOD Record ,
Acknowlegements
Proceedings of the 2000 ACM SIGMOD international
conference on Management of data SIGMOD 2000,
This research was partially supported by the University of
Volume 29 Issue 2.
Central Arkansas URC grant.
[11] T. Milo, S. Abiteboul, B. Amann, O. Benjelloun,
6 References F.D. Ngoc, Exchanging Intensional XML Data, in
Proceedings of the 2003 ACM SIGMOD international
[1] Rakesh Agrawal, Dmitri Asonov, and R. Srikant., conference on Management of Data, June 2003.
Enabling Sovereign Information Sharing Using Web
Services, in Proceedings of the 2004 ACM SIGMOD [12] M. T. Özsu, P. Valduriez. Principles of Distributed
international conference on Management of Data, June Database Systems. Prentice Hall, 1999.
2004.
[13] SOAP, XML Protocol. http://www.w3.org/2000/
[2] Francisco Álvarez Cavazos and Juan Carlos xp/Group/.
Lavariega Jarquín , A 3-Tiered Client-Server Distributed
Database System Component-Based, in Proceedings of [14] UDDI Technical Committee Specification.
the winter international synposium on Information and http://uddi.org/pubs/uddi_v3.htm, 2003.
communication technologies, 2004 [15] Web Services. http://www.w3.org/2002/ws.
[3] A. Davis, J. Parikh, and W.E. Weihl, [16] Web Services Description Languages, WSDL.
EdgeComputing: Extending Enterprise Applications to http://www.w3.org/TR/wsdl.