Version 1.0
Author: Falko Timme
Last edited 04/26/2006
This tutorial shows how to set up a two-node Apache web server cluster that provides high-availability. In front of the Apache cluster we create a load balancer that splits up incoming requests between the two Apache nodes. Because we do not want the load balancer to become another "Single Point Of Failure", we must provide high-availability for the load balancer, too. Therefore our load balancer will in fact consist out of two load balancer nodes that monitor each other using heartbeat, and if one load balancer fails, the other takes over silently.
The advantage of using a load balancer compared to using round robin DNS is that it takes care of the load on the web server nodes and tries to direct requests to the node with less load, and it also takes care of connections/sessions. Many web applications (e.g. forum software, shopping carts, etc.) make use of sessions, and if you are in a session on Apache node 1, you would lose that session if suddenly node 2 served your requests. In addition to that, if one of the Apache nodes goes down, the load balancer realizes that and directs all incoming requests to the remaining node which would not be possible with round robin DNS.
For this setup, we need four nodes (two Apache nodes and two load balancer nodes) and five IP addresses: one for each node and one virtual IP address that will be shared by the load balancer nodes and used for incoming HTTP requests.
I will use the following setup here:
Apache node 1: webserver1.example.com (webserver1) - IP address: 192.168.0.101; Apache document root: /var/www
Apache node 2: webserver2.example.com (webserver2) - IP address: 192.168.0.102; Apache document root: /var/www
Load Balancer node 1: loadb1.example.com (loadb1) - IP address: 192.168.0.103
Load Balancer node 2: loadb2.example.com (loadb2) - IP address: 192.168.0.104
Virtual IP Address: 192.168.0.105 (used for incoming requests)
Have a look at the drawing on http://www.linuxvirtualserver.org/docs/ha/ultramonkey.html to understand how this setup looks like.
In this tutorial I will use Debian Sarge for all four nodes. I assume that you have installed a basic Debian installation on all four nodes, and that you have installed Apache on webserver1 and webserver2, with /var/www being the document root of the main web site.
I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!
1 Enable IPVS On The Load Balancers
First we must enable IPVS on our load balancers. IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4 switching.
loadb1/loadb2:
echo ip_vs_dh >> /etc/modules
echo ip_vs_ftp >> /etc/modules
echo ip_vs >> /etc/modules
echo ip_vs_lblc >> /etc/modules
echo ip_vs_lblcr >> /etc/modules
echo ip_vs_lc >> /etc/modules
echo ip_vs_nq >> /etc/modules
echo ip_vs_rr >> /etc/modules
echo ip_vs_sed >> /etc/modules
echo ip_vs_sh >> /etc/modules
echo ip_vs_wlc >> /etc/modules
echo ip_vs_wrr >> /etc/modules
Then we do this:
loadb1/loadb2:
modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_lc
modprobe ip_vs_nq
modprobe ip_vs_rr
modprobe ip_vs_sed
modprobe ip_vs_sh
modprobe ip_vs_wlc
modprobe ip_vs_wrr
If you get errors, then most probably your kernel wasn't compiled with IPVS support, and you need to compile a new kernel with IPVS support (or install a kernel image with IPVS support) now.
2 Install Ultra Monkey On The Load Balancers
Ultra Monkey is a project to create load balanced and highly available services on a local area network using Open Source components on the Linux operating system; the Ultra Monkey package provides heartbeat (used by the two load balancers to monitor each other and check if the other node is still alive) and ldirectord, the actual load balancer.
To install Ultra Monkey, we must edit /etc/apt/sources.list now and add these two lines (don't remove the other repositories):
loadb1/loadb2:
vi /etc/apt/sources.list
deb http://www.ultramonkey.org/download/3/ sarge maindeb-src http://www.ultramonkey.org/download/3 sarge main
Afterwards we do this:
loadb1/loadb2:
apt-get update
and install Ultra Monkey:
loadb1/loadb2:
apt-get install ultramonkey
If you see this warning:
¦ libsensors3 not functional ¦ ¦ ¦ ¦ It appears that your kernel is not compiled with sensors support. As a ¦ ¦ result, libsensors3 will not be functional on your system. ¦ ¦ ¦ ¦ If you want to enable it, have a look at "I2C Hardware Sensors Chip ¦ ¦ support" in your kernel configuration. ¦
you can ignore it.
During the Ultra Monkey installation you will be asked a few question. Answer as follows:
Do you want to automatically load IPVS rules on boot?
<-- No
Select a daemon method.
<-- none
3 Enable Packet Forwarding On The Load Balancers
The load balancers must be able to route traffic to the Apache nodes. Therefore we must enable packet forwarding on the load balancers. Add the following lines to /etc/sysctl.conf:
loadb1/loadb2:
vi /etc/sysctl.conf
# Enables packet forwardingnet.ipv4.ip_forward = 1
Then do this:
loadb1/loadb2:
sysctl -p
4 Configure heartbeat And ldirectord
Now we have to create three configuration files for heartbeat. They must be identical on loadb1 and loadb2!
loadb1/loadb2:
vi /etc/ha.d/ha.cf
logfacility local0bcast eth0 # Linuxmcast eth0 225.0.0.1 694 1 0auto_failback offnode loadb1node loadb2respawn hacluster /usr/lib/heartbeat/ipfailapiauth ipfail gid=haclient uid=hacluster
Important: As nodenames we must use the output of
uname -n
on loadb1 and loadb2.
loadb1/loadb2:
vi /etc/ha.d/haresources
loadb1 \ ldirectord::ldirectord.cf \ LVSSyncDaemonSwap::master \ IPaddr2::192.168.0.105/24/eth0/192.168.0.255
The first word is the output of
uname -n
on loadb1, no matter if you create the file on loadb1 or loadb2! After IPaddr2 we put our virtual IP address 192.168.0.105.
loadb1/loadb2:
vi /etc/ha.d/authkeys
auth 33 md5 somerandomstring
somerandomstring is a password which the two heartbeat daemons on loadb1 and loadb2 use to authenticate against each other. Use your own string here. You have the choice between three authentication mechanisms. I use md5 as it is the most secure one.
/etc/ha.d/authkeys should be readable by root only, therefore we do this:
loadb1/loadb2:
chmod 600 /etc/ha.d/authkeys
ldirectord is the actual load balancer. We are going to configure our two load balancers (loadb1.example.com and loadb2.example.com) in an active/passive setup, which means we have one active load balancer, and the other one is a hot-standby and becomes active if the active one fails. To make it work, we must create the ldirectord configuration file /etc/ha.d/ldirectord.cf which again must be identical on loadb1 and loadb2.
loadb1/loadb2:
vi /etc/ha.d/ldirectord.cf
checktimeout=10checkinterval=2autoreload=nologfile="local0"quiescent=yesvirtual=192.168.0.105:80 real=192.168.0.101:80 gate real=192.168.0.102:80 gate fallback=127.0.0.1:80 gate service=http request="ldirector.html" receive="Test Page" scheduler=rr protocol=tcp checktype=negotiate
In the virtual= line we put our virtual IP address (192.168.0.105 in this example), and in the real= lines we list the IP addresses of our Apache nodes (192.168.0.101 and 192.168.0.102 in this example). In the request= line we list the name of a file on webserver1 and webserver2 that ldirectord will request repeatedly to see if webserver1 and webserver2 are still alive. That file (that we are going to create later on) must contain the string listed in the receive= line.
Afterwards we create the system startup links for heartbeat and remove those of ldirectord because ldirectord will be started by the heartbeat daemon:
loadb1/loadb2:
update-rc.d heartbeat start 75 2 3 4 5 . stop 05 0 1 6 .
update-rc.d -f ldirectord remove
Finally we start heartbeat (and with it ldirectord):
loadb1/loadb2:
/etc/init.d/ldirectord stop
/etc/init.d/heartbeat start
5 Test The Load Balancers
Let's check if both load balancers work as expected:
loadb1/loadb2:
ip addr sh eth0
The active load balancer should list the virtual IP address (192.168.0.105):
2: eth0:
The hot-standby should show this:
2: eth0:
loadb1/loadb2:
ldirectord ldirectord.cf status
Output on the active load balancer:
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 1455
Output on the hot-standby:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
loadb1/loadb2:
ipvsadm -L -n
Output on the active load balancer:
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.0.105:80 rr -> 192.168.0.101:80 Route 0 0 0 -> 192.168.0.102:80 Route 0 0 0 -> 127.0.0.1:80 Local 1 0 0
Output on the hot-standby:
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn
loadb1/loadb2:
/etc/ha.d/resource.d/LVSSyncDaemonSwap master status
Output on the active load balancer:
master running(ipvs_syncmaster pid: 1591)
Output on the hot-standby:
master stopped
If your tests went fine, you can now go on and configure the two Apache nodes.
6 Configure The Two Apache Nodes
Finally we must configure our Apache cluster nodes webserver1.example.com and webserver2.example.com to accept requests on the virtual IP address 192.168.0.105.
webserver1/webserver2:
apt-get install iproute
Add the following to /etc/sysctl.conf:
webserver1/webserver2:
vi /etc/sysctl.conf
# Enable configuration of arp_ignore optionnet.ipv4.conf.all.arp_ignore = 1# When an arp request is received on eth0, only respond if that address is# configured on eth0. In particular, do not respond if the address is# configured on lonet.ipv4.conf.eth0.arp_ignore = 1# Ditto for eth1, add for all ARPing interfaces#net.ipv4.conf.eth1.arp_ignore = 1# Enable configuration of arp_announce optionnet.ipv4.conf.all.arp_announce = 2# When making an ARP request sent through eth0 Always use an address that# is configured on eth0 as the source address of the ARP request. If this# is not set, and packets are being sent out eth0 for an address that is on# lo, and an arp request is required, then the address on lo will be used.# As the source IP address of arp requests is entered into the ARP cache on# the destination, it has the effect of announcing this address. This is# not desirable in this case as adresses on lo on the real-servers should# be announced only by the linux-director.net.ipv4.conf.eth0.arp_announce = 2# Ditto for eth1, add for all ARPing interfaces#net.ipv4.conf.eth1.arp_announce = 2
Then run this:
webserver1/webserver2:
sysctl -p
Add this section for the virtual IP address to /etc/network/interfaces:
webserver1/webserver2:
vi /etc/network/interfaces
auto lo:0iface lo:0 inet static address 192.168.0.105 netmask 255.255.255.255 pre-up sysctl -p > /dev/null
Then run this:
webserver1/webserver2:
ifup lo:0
Finally we must create the file ldirector.html. This file is requested by the two load balancer nodes repeatedly so that they can see if the two Apache nodes are still running. I assume that the document root of the main apache web site on webserver1 and webserver2 is /var/www, therefore we create the file /var/www/ldirector.html:
webserver1/webserver2:
vi /var/www/ldirector.html
Test Page
7 Further Testing
You can now access the web site that is hosted by the two Apache nodes by typing http://192.168.0.105 in your browser.
Now stop the Apache on either webserver1 or webserver2. You should then still see the web site on http://192.168.0.105 because the load balancer directs requests to the working Apache node. Of course, if you stop both Apaches, then your request will fail.
Now let's assume that loadb1 is our active load balancer, and loadb2 is the hot-standby. Now stop heartbeat on loadb1:
loadb1:
/etc/init.d/heartbeat stop
Wait a few seconds, and then try http://192.168.0.105 again in your browser. You should still see your web site because loadb2 has taken the active role now.
Now start heartbeat again on loadb1:
loadb1:
/etc/init.d/heartbeat start
loadb2 should still have the active role. Do the tests from chapter 5 again on loadb1 and loadb2, and you should see the inverse results as before.
If you have also passed these tests, then your loadbalanced Apache cluster is working as expected. Have fun!
8 Further Reading
This tutorial shows how to loadbalance two Apache nodes. It does not show how to keep the files in the Apache document root in sync or how to create a storage solution like an NFS server that both Apache nodes can use, nor does it provide a solution how to manage your MySQL database(s). You can find solutions for these issues here:
Mirror Your Web Site With rsync
Setting Up A Highly Available NFS Server
How To Set Up A Load-Balanced MySQL Cluster
How To Set Up Database Replication In MySQL
9 Links
heartbeat / The High-Availability Linux Project: http://linux-ha.org
The Linux Virtual Server Project: http://www.linuxvirtualserver.org
Ultra Monkey: http://www.ultramonkey.org
previous up
How To Set Up A Loadbalanced High-Availability Apache Cluster - Page 3
Copyright © 2006 Falko Timme
All Rights Reserved.
add comment | view as pdf | print: this | all page(s) |
Related Tutorials
Setting Up A Highly Available NFS Server
How To Set Up A Load-Balanced MySQL Cluster
How To Set Up Database Replication In MySQL
Mirror Your Web Site With rsync
Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Nice article, although I
Submitted by Anonymous Coward (not registered) on Tue, 2008-11-25 12:33.
Nice article, although I think Load Balancing is a little misleading.
This is High Availability - an active / passive server relationship, there is no load sharing
reply | view as pdf
Fantastic Tutorial!
Submitted by namwodahs (registered user) on Thu, 2006-10-19 22:06.
This is a great tutorial! Worked first time through. I've been very impressed with your selection of tutorials on this site. Thank You!
reply | view as pdf
How To
Submitted by Anonymous (not registered) on Mon, 2006-05-15 12:06.
How To very mutch!!!!
By Marcos Abadi
Brazilian
reply | view as pdf
Re: How To
Submitted by Anonymous (not registered) on Sun, 2008-09-28 20:43.
Hi,
seems to be a great tutorial !
I'd like to know if this kind of load balancing setup can be SSL aware, means, an SSL session is able load balanced between nodes ?
Thanks a lot
reply | view as pdf
Re: Re: How To
Submitted by Anonymous (not registered) on Tue, 2008-09-30 23:03.
can you tell me if this tutorial implements session-aware load-balancing as we need to use this for big site which is session driven.
Hello all,
in my opinion the persistant flag is missing. As described in the introduction of this tutorial the load balancers also take care of connections/sessions.
This does NOT happen.
After I added the option "persistent" to ldirectord.cf it worked:
virtual=192.168.0.105:80 real=192.168.0.101:80 gate real=192.168.0.102:80 gate fallback=127.0.0.1:80 gate service=http request="ldirector.html" receive="Test Page" scheduler=rr persistent=600 protocol=tcp checktype=negotiate600 means 600 seconds which also could be the default setting. In my opinion to short for stuff like forum software.
Maybe there is a easier way to handle this. Our setup is depending mostly on PHP and PHP session based stuff, so it did not work correctly before.
Kindest regards,
Martin
No comments:
Post a Comment