Setting Up Highly
Available Server
On
Cent OS 5
CONTENT
1. Server Setup
2. Configuring Static and Virtual IP
3. Configure Network File System(NFS)
4. Synchronize System Time
5. Install DRBD
6. Configure DRBD
7. Configure MYSQL on DRBD
8. Configure httpd for Virtual IP
9. Testing DRBD Setup without Heartbeat
10. Installing Heartbeat
11. Configure Heartbeat
12. Testing DRBD with Heartbeat
13. Load Balancer using mod_proxy and mod_proxy_balancer
14. Appendix
Server Setup
At the time of installation of both the server, setup partition as specified.
/dev/sda1 150MB /boot (ext3)
/dev/sda2 300MB unmounted (LVM, set primary) metadata info.
/dev/sda3 10GB / (ext3/LVM)
/dev/sda5 2GB /swap (swap)
/dev/sda6 25GB unmounted (LVM) data
/dev/sda7 15GB unmounted (LVM) database
/dev/sda8 15GB /var (ext3/LVM)
Configure above partition manually. In above case we had an Hard disk of size 76
GB as per the disk size and number of drbd partition, we can vary above setup. We have
configure 300MB for metadata since we had 2 Drbd partitions (data,database). Metadata
size will be decided depend on the number of drbd partition. To calculate the size of
Metadata we use this formula: (no. of Drbd partition) * 150MB.
Configuring Static and Virtual IP
We have to configure local IP for each server and a common virtual IP for these
servers. For an example.
To configure static IP
To the server run:
# setup
command select Network configuration. Select Ethernet module and set a static IP
to the server. Make sure that those IP which will configure are not in use. After exiting
setup module run this command.
# service network restart
Now if you run
# ifconfig
Output:
eth0 Link encap:Ethernet HWaddr 00:14:85:98:BF:07
inet addr:172.16.13.251 Bcast:172.16.13.255 Mask:255.255.255.0
inet6 addr: fe80::214:85ff:fe98:bf07/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:391619 errors:0 dropped:0 overruns:0 frame:0
TX packets:152008 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:400311050 (381.7 MiB) TX bytes:13941568 (13.2 MiB)
Interrupt:201 Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1375 errors:0 dropped:0 overruns:0 frame:0
TX packets:1375 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2252232 (2.1 MiB) TX bytes:2252232 (2.1 MiB)
To configure virtual IP
Create a file ifcfg-eth0:1 in “/etc/sysconfig/network-scripts/” folder
# Please read /usr/share/doc/initscripts-*/sysconfig.txt
# for the documentation of these parameters.
GATEWAY=172.16.13.201
TYPE=Ethernet
DEVICE=eth0:1
BOOTPROTO=none
NETMASK=255.255.255.0
IPADDR=172.16.13.250
USERCTL=no
IPV6INIT=no
PEERDNS=yes
Note: Configure Netmask, IP and Gateway respective to the network.
Configure hosts file:
Add these lines to “/etc/hosts” file
#Server 1 IP as node1
172.16.13.253 node1 node1
#Server 2 IP as node2
172.16.13.251 node2 node2
Synchronize System Time
It's important that both node1 and node2 have the same system time. Therefore
we install an NTP client on both:
node1/node2:
# yum install ntp ntpdate
Afterwards you can check that both have the same time by running
node1/node2:
# date
Configure Network File System(NFS)
A Network File System (NFS) allows remote hosts to mount file systems over a
network and interact with those file systems as though they are mounted locally.
This enables system administrators to share the resources on the network.
Then we remove the system bootup links for NFS because NFS will be started and
controlled by heartbeat in our setup:
node1/node2:
# chkconfig nfs off
# chkconfig nfs-server off
How to check weather NFS is running or not.
# service nfs status
To configure resources to share on the network, edit /etc/exports file and add this
line on the both the server.
On node1
/mnt/data 172.16.13.251(rw,no_root_squash,no_all_squash,sync)
On node2
/mnt/data 172.16.13.253(rw,no_root_squash,no_all_squash,sync)
NFS stores some important information (e.g. information about file locks, etc.)
in /var/lib/nfs. Now what happens if node1 goes down? node2 takes over, but its
information in /var/lib/nfs will be different from the information in node1's
/var/lib/nfs directory. Therefore we do some tweaking so that these details will be
stored on our /mnt/data partition which is mirrored by DRBD between node1 and
node2. So if node1 goes down node2 can use the NFS details of node1.
node1/node2:
# mkdir /mnt/data
node1:
# mount -t ext3 /dev/drbd0 /mnt/data
# mv /var/lib/nfs/ /mnt/data/
# ln -s /mnt/data/nfs/ /var/lib/nfs
# mkdir /mnt/data/export
# umount /data
node2:
# rm -fr /var/lib/nfs/
# ln -s /mnt/data/nfs/ /var/lib/nfs
Edit /etc/fstab file and add these line
node1
172.16.13.251:/mnt/data /mnt/data nfs defaults 00
node2
172.16.13.253:/mnt/data /mnt/data nfs defaults 00
After editing and saving the files restart nfs.
# service nfs restart
Install DRBD
Install DRBD software and DRBD kernel module on both the servers.
Use following command to install.
# yum install kmod-drbd -y
Load DRBD kernel module with insmod
# insmod /lib/modules/<kernel-ver>/extra/drbd/drbd.ko
Verify DRBD weather its properly installed.
# lsmod |grep drbd
Output:
drbd 188284 4
# REBOOT SERVER
# reboot
Configure DRBD
The configuration file of DRBD is placed in “/etc/drbd.conf”. Edit/replace this file
with following configuration data.
resource drbd0 {
protocol C;
handlers {
pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
}
startup {
degr-wfc-timeout 120;# 2 minutes.
}
disk {
on-io-error detach;
}
net {
timeout 120;
connect-int 20;
ping-int 20;
max-buffers 2048;
max-epoch-size2048;
ko-count 30;
cram-hmac-alg "sha1";
shared-secret "FooFunFactory";
}
syncer {
rate 10M;
al-extents 257;
}
on node1 {
device /dev/drbd0;
disk /dev/sda6;
address172.16.13.251:7788;
meta-disk/dev/sda2[0];
}
on node2 {
device /dev/drbd0;
disk /dev/sda6;
address172.16.13.252:7788;
meta-disk/dev/sda2[0];
}
}
resource drbd1 {
protocol C;
handlers {
pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
}
startup {
degr-wfc-timeout 120;# 2 minutes.
}
disk {
on-io-error detach;
}
net {
timeout 120;
connect-int 20;
ping-int 20;
max-buffers 2048;
max-epoch-size2048;
ko-count 30;
cram-hmac-alg "sha1";
shared-secret "FooFunFactory";
}
syncer {
rate 10M;
al-extents 257;
}
on node1 {
device /dev/drbd1;
disk /dev/sda7;
address172.16.13.251:7789;
meta-disk /dev/sda2[1];
}
on node2 {
device /dev/drbd1;
disk /dev/sda7;
address172.16.13.252:7789;
meta-disk/dev/sda2[1];
}
}
Save the file and follow the following steps on the both the servers
# groupadd haclient
# chgrp haclient /sbin/drbdsetup
# chmod o-x /sbin/drbdsetup
# chmod u+s /sbin/drbdsetup
# chgrp haclient /sbin/drbdmeta
# chmod o-x /sbin/drbdmeta
# chmod u+s /sbin/drbdmeta
Before executing below command verify what is the name of your metadisk
This command may take some time and it will ask some user inputs
# dd if=/dev/zero bs=1M count=1 of=/dev/sda2
This command will create meta disks for drbd drives
# drbdadm create-md drbd0 drbd1
Restart drbd service
# service drbd restart OR # /etc/init.d/drbd restart
Check the output of following command
# cat /proc/drbd
Output:
0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/7 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
1: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/7 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
Note:To know the meaning of status mention above refer this link
http://www.drbd.org/users-guide. To solve the split-brain issue either refer
Appendix or refer above user guide.
Run this command on the MASTER server. This will synchronized both the server
drbd partitions. It will make the server as a Primary server and second server will
become Secondary.
# drbdadm -- --overwrite-data-of-peer primary all; cat /proc/drbd
Output:
0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent r---
ns:65216 nr:0 dw:0 dr:65408 al:0 bm:3 lo:0 pe:7 ua:6 ap:0
[>...................] sync'ed: 2.3% (3083548/3148572)K
finish: 0:04:43 speed: 10,836 (10,836) K/sec
resync: used:1/7 hits:4072 misses:4 starving:0 dirty:0 changed:4
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
1: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent r---
ns:65216 nr:0 dw:0 dr:65408 al:0 bm:3 lo:0 pe:7 ua:6 ap:0
[>...................] sync'ed: 2.3% (3083548/3148572)K
finish: 0:04:43 speed: 10,836 (10,836) K/sec
resync: used:1/7 hits:4072 misses:4 starving:0 dirty:0 changed:4
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
Create the filesystem by typing the following on the MASTER system:
# mkfs.ext3 /dev/drbd0
# mkfs.ext3 /dev/drbd1
Configure MYSQL on DRBD
We will create our database on the drbd1 partition. To get this go through the
following steps.
Stop mysql process on both the nodes
# service mysql stop OR # service mysqld stop
# /etc/init.d/mysql stop OR # /etc/init.d/mysqld stop
Mount drbd disk
# mount -t ext3 /dev/drbd1 /var/lib/mysql
Change owner of the disk to mysql user
# chown -R mysql:mysql /var/lib/mysql
Start mysql server
# service mysql start OR # service mysqld start
# /etc/init.d/mysql start OR # /etc/init.d/mysqld start
Check weather mysql is working properly
# mysql -uroot -p
Configure httpd for Virtual IP
httpd is mainly used for redirecting request coming on 80 port to respective
tomcat ajp ports.
To do this we have to configure vhost file.
Edit httpd.conf file which is present in /etc/httpd/conf/httpd.conf and add below
line at the end of the file.
Include /etc/httpd/conf/vhost-*.conf
Create a vhost file in /etc/httpd/conf/vhost-localhost.conf OR
/etc/httpd/conf/vhost-<domain-name>.conf if domain exist.
<VirtualHost *:80>
ServerName <domain-name|localhost>
<Directory /var/www/html/localhost >
Order Deny,Allow
Allow from all
Deny from all
</Directory>
ProxyPass / ajp://localhost:8109/
ServerAlias localhost
DocumentRoot "/home/onmobile/"
CustomLog "|/usr/sbin/rotatelogs /var/log/httpd/access-%Y%m%d.log 86400
+330 %I %O" combined
ErrorLog "|/usr/sbin/rotatelogs /var/log/httpd/error.log.%Y-%m-%d 86400
+330"
<Directory /home/onmobile/>
AllowOverride All
</Directory>
</VirtualHost>
Testing DRBD Setup without Heartbeat
Stop mysql and unmount both the drbd disks on node1 and node2
Make the node1 as Primary
# drbdadm primary all
Make the node2 as Secondary
# drbdadm secondary all
* Mysql
Stop mysql process and mount mysql disk on node1
# service mysql stop
# mount -t ext3 /dev/drbd1 /var/lib/mysql
Create a test database with sample tables and add data to it on node1.
Stop mysql and unmount drbd1 disk on node1
# service mysqld stop
# umount /var/lib/mysql
After this make node1 as Secondary
# drbdadm secondary all
Make node2 as Primary
# drbdadm primary all
mount drbd1 disk on node2
# mount -t ext3 /dev/drbd1 /var/lib/mysql
Start mysql on node2
# service mysql start
Check weather your database is present. If it exist then your setup is perfect. You
can proceed to Heart beat installation and configuration.
* DATA
Mount data drbd0 disk on node2
# mount -t ext3 /dev/drbd0 /mnt/data
Copy some files test.txt to this /mnt/data directory on node2
Unmount drbd0 disk on node2
# umount /mnt/drbd0
After this make node2 as Secondary
# drbdadm secondary all
Make node1 as Primary
# drbdadm primary all
Mount drbd0 disk on node1
# mount -t ext3 /dev/drbd0 /mnt/data
Check weather your test.txt file exist. If it exist then your setup is perfect. You can
proceed to Heart beat installation and configuration.
Installing Heartbeat
Install heartbeat on both the servers
# yum install heartbeat -y
Check weather you find the heartbeat file in /var/lib location. If it exist then setup
is perfect, else you can run above command again to reinstall remaining
components.
Configure Heartbeat
To Configure heartbeat you need to edit three files on both the servers.
Configuring ha.cf file which is present in the /etc/heartbeat/ folder.
Check the node names before configuring it in this file using
# uname -n OR # hostname
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 10
udpport 694
bcast eth0
auto_failback on
udp eth0
node node1
node node2
Configure haresources file which is present in the /etc/ha.d/ folder. Configure
this on the both the servers
The first word is the output of uname -n on node1, no matter if you create the file
on node1 or node2! After IPaddr we put our virtual IP address 172.16.13.250, and
after drbddisk we use the resource name of our DRBD resource which is drbd0 and
drbd1 here (remember, that is the resource names we use in /etc/drbd.conf - if
you use another one, you must use it here, too).
node1 IPaddr::172.16.13.250 drbddisk::drbd0 drbddisk::drbd1
Filesystem::/dev/drbd0::/mnt/data::ext3
Filesystem::/dev/drbd1::/var/lib/mysql::ext3 nfs mysqld httpd
Note: Write above lines in one line while creating file. Also check weather it is
mysql or mysqld. You can check this in /etc/init.d/ folder.
Configure authkeys file which is present in the /eth/ha.d folder. Configure this on
the both the servers.
auth 2
#1 crc
2 sha1 HI!
#3 md5 Hello!
This file is readable to root only so we change the permission of this file.
# chmod 600 /etc/ha.d/authkeys
Now we are set to start DRBD and heartbeat on both the servers.
# service drbd start
# service heartbeat start
When you will start drbd and heartbeat you can check weather it is properly
working or not. You will find all the below results on one of the server on which
you have set it primary.
Verification of Setup
Check drbd disk are mount or not
# df -h
Output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 19G 2.6G 16G 15% /
/dev/sda8 23G 266M 22G 2% /var
/dev/sda1 145M 16M 122M 12% /boot
tmpfs 501M 0 501M 0% /dev/shm
/dev/drbd0 9.7G 151M 9.0G 2% /mnt/data
/dev/drbd1 20G 193M 19G 2% /var/lib/mysql
Check network virtual IP
# ifconfig
Output:
eth0 Link encap:Ethernet HWaddr 00:0C:29:A1:C5:9B
inet addr:172.16.13.133 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fea1:c59b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:18992 errors:0 dropped:0 overruns:0 frame:0
TX packets:24816 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2735887 (2.6 MiB) TX bytes:28119087 (26.8 MiB)
Interrupt:177 Base address:0x1400
eth0:0 Link encap:Ethernet HWaddr 00:0C:29:A1:C5:9B
inet addr:172.16.13.250 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:177 Base address:0x1400
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:71 errors:0 dropped:0 overruns:0 frame:0
TX packets:71 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5178 (5.0 KiB) TX bytes:5178 (5.0 KiB)
Check mysql and httpd services are working properly
# service httpd status
# service mysqld status
Testing DRBD with Heartbeat
Checking different scenarios which may occur real time on servers.
Considering both the servers are properly configured and Drbd and Heartbeat are
running on it.
Server1 (Master node1) is set as Primary having all the resources.
Server2 (node2) is set as Secondary.
Server1 goes down (Shutdown)
Server2 will switch from Secondary to Primary and take all the resources (IP, Drbd
disk will get mounted, Mysql and Httpd service will come up)
Perform above Verification step on the Sever2.
Server 2 is up and Server1 is down, Now Server1 comes up.
Drbd will first synchronized both the drbd disk and then Heartbeat will switch the
resource from server2 to server1. Server 1 will now become Primary and Server2
will become secondary.
Same above scenarios you will be experience when network goes down of one
server.
Load Balancer using mod_proxy and mod_proxy_balancer
Apache (httpd) can also be used to perform load balancing of the web application
deployed on the servers. It can be done either by using mod_jk or by using
mod_proxy_balancer. This document will explain how we can configure load
balancer using mod_proxy_balancer.
Perform following operation on both the servers.
Install Apache tomcat 5 or above version on both the servers.
Then check weather Apache contain mod_proxy_balancer module.
Use this command to verify:
# httpd -M
This will display all Shared as well as Static modules. See if you can find
“proxy_balancer_module”. If you find then you can continue else download it form
apache site, for this you need to recompile Apache httpd module if exist.
Note: Centos 5 has this module pre-installed at the time of installation.
Once you are done with the module installation, edit vhost-<domain-name>.conf
file which we have create earlier. This file is present in /etc/httpd/conf folder.
Replace “ProxyPass / ajp://localhost:8109/” statement with the below lines.
ProxyPass / balancer://mycluster/
<Proxy balancer://mycluster>
BalancerMember http://172.16.13.251:8080
BalancerMember http://172.16.13.253:8080
ProxySet lbmethod=bytraffic
</Proxy>
You can add both the server web application references in the BalanceMember.
ProxySet can have values “bytraffic” or “byrequests” by default if you don't specify
its “byrequets”.
Save this settings verify and Reload the httpd module.
Verify:
# httpd -S
Output:
*:80 drbd.onmobile.com (/etc/httpd/conf/vhost-drbd.onmobile.com .conf:1)
Syntax OK
Reload:
# service httpd restart
Testing:
Either Deploy a web application on both the server tomcat OR use jsp-examples
for testing. Edit index.html file which is present in the <tomcat
home>/webapps/jsp-examples/ folder on one server. [Change the text of
background color just for our identification purpose]
Make a request from the browser on the domain name which we are using:
http://drbd.onmobile.com/jsp-examples/index.html
If ProxySet is bytraffic you will find on every request, page will change.
If ProxySet is byrequests page wont change frequently. It will only change when
the load on the first server is more than expected.
Appendix
Common issues you will face at the time of installation.
Q. Installation of kernel module using yum gives and error. It stops with an error
of checksum.
A. Edit /etc/yum.conf, /etc/yum.repos.d/*.repo files and change gpgcheck=0.
Q. Getting out from Split Brain scenario.
A. When you find drbd status like this:
0: cs:StandAlone st:Primary/Secondary ds:Inconsistent/Inconsistent r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/7 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
1: cs:StandAlone st:Primary/Secondary ds:Inconsistent/Inconsistent r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/7 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
To come out of this use below steps:
If the server is Primary you have to first make it Secondary.
# drbdadm disconnect all OR # drbdadm disconnect drbd0
# drbdadm secondary all OR # drbdadm secondary drbd0
Now to get it out from StandAlone state to connecting state run this
# drbdadm -- --discard-my-data connect drbd0
To connect drbd to the server
# drbdadm connect all
Q. Heartbeat log file give following error.
heartbeat: [2407]: ERROR: should_drop_message: attempted replay attack
[node1]? [gen = 16, curgen = 62]
A. To ride off this error stop heartbeat.
# service heartbeat stop
Then edit /var/lib/heartbeat/hb_generation file of (Server1)node1 and set the
value to one lesser then the gen number (16-1=15)
Restart heartbeat.
# service heartbeat start
References
DRBD
http://www.drbd.org
Heartbeat
http://www.novell.com/documentation/sles10/heartbeat/index.html?
page=/documentation/sles10/heartbeat/data/heartbeat.html
Load Balancer using mod_proxy
http://httpd.apache.org/docs/2.2/mod/mod_proxy.html
http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html
NFS
http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat
http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/ref-guide/ch-
nfs.html
http://linux-ha.org/DRBD/NFS
