High Availability HA cluster solution for FreeBSD
HAST + CARP + ZFS
HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.
CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.
http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
Showing posts with label Load Balancing. Show all posts
Showing posts with label Load Balancing. Show all posts
Tuesday, October 2, 2012
負載平衡器優缺點比較(Nginx vs HAProxy vs LVS)
負載平衡器優缺點比較(Nginx vs HAProxy vs LVS)
在這3款負載平衡伺服器中，丫忠比較有接觸的是Nginx，直到最近才對HAProxy比較有接觸，LVS則是完全沒印象，不過既然找到了負載平衡伺服器的比較資料，丫忠就順便做個筆本，再找個時候來研究看看，甚至寫個安裝心得筆記囉!
2.功能較齊全，除了當負載平衡伺服器外，還可以像apache一樣當Web伺服器，且可以透過Geo模式(註1)來達到流量分配功能。
3.支援的模組比較多
4.支援Gzip proxy.
2.對於檢查後端伺服器狀態的支援度不夠好。只支援透過埠號(port)來檢查，無法透過url檢查
3.對big request header的支援不是很好，如果 client_header_buffer_size 的設定值比較小，就會返回400 bad request的頁面
也許您對 Nginx V1.5 中文技術手冊 有興趣。
2.透過指定的url檢查後端伺服器的狀態
3.支援tcp協定的負載平衡，譬如：可以給mysql伺服器、mail server郵件伺服器做負載平衡
4.支援虛擬主機
2.LVS的DR模式支援透過廣域網路進行負載平衡，這是個相當大的功能特性，因為其他2款負載平衡器不具備此功能
註1：所謂Geo模式是指全域的負載模式，根據不同客戶端(Client)的ip分配到不同的伺服器(Server)。譬如：將特定客戶端的IP分配到特定的伺服器，一般網路使用者則分配到一般Web伺服器。
註2：Nagios 為提高效能和準確性的網路監控軟體(詳細)
註3：Cacti 是一套類似 MRTG 的snmp流量監控跟系統資訊監控軟體，除此 Cacti 還可以外掛 Script 及 Templates作出各種的監控圖 ，有興趣的網友可以參考 CaCti官網 的說明
幸運的是有一套軟體 cacti 的發展就是基於讓 RRDTool 使用者更方便使用該軟體，
除了基本的 Snmp 流量跟系統資訊監控外，Cacti 也可外掛 Scripts 及加上 Templates 來作出各式各樣的監控圖
Reference: http://homeserver.com.tw/proxy-server/%E8%B2%A0%E8%BC%89%E5%B9%B3%E8%A1%A1%E5%99%A8%E5%84%AA%E7%BC%BA%E9%BB%9E%E6%AF%94%E8%BC%83nginx-vs-haproxy-vs-lvs/
在這3款負載平衡伺服器中，丫忠比較有接觸的是Nginx，直到最近才對HAProxy比較有接觸，LVS則是完全沒印象，不過既然找到了負載平衡伺服器的比較資料，丫忠就順便做個筆本，再找個時候來研究看看，甚至寫個安裝心得筆記囉!
Nginx的優點
1.效能不錯，同時負載效能可以達到1萬2.功能較齊全，除了當負載平衡伺服器外，還可以像apache一樣當Web伺服器，且可以透過Geo模式(註1)來達到流量分配功能。
3.支援的模組比較多
4.支援Gzip proxy.
Nginx的缺點
1.不支援session keep alive2.對於檢查後端伺服器狀態的支援度不夠好。只支援透過埠號(port)來檢查，無法透過url檢查
3.對big request header的支援不是很好，如果 client_header_buffer_size 的設定值比較小，就會返回400 bad request的頁面
也許您對 Nginx V1.5 中文技術手冊 有興趣。
HAProxy的優點
1.支援session keep alive2.透過指定的url檢查後端伺服器的狀態
3.支援tcp協定的負載平衡，譬如：可以給mysql伺服器、mail server郵件伺服器做負載平衡
4.支援虛擬主機
HAProxy的缺點
1.目前沒有支援 nagios(註2) 和 cacti (註3)的網路監控功能LVS的優點
1.效能好，接近硬體式負載平衡設備的效能和連接負載效能2.LVS的DR模式支援透過廣域網路進行負載平衡，這是個相當大的功能特性，因為其他2款負載平衡器不具備此功能
LVS的缺點
1.比較複雜，模組支援度不如 Nginx註1：所謂Geo模式是指全域的負載模式，根據不同客戶端(Client)的ip分配到不同的伺服器(Server)。譬如：將特定客戶端的IP分配到特定的伺服器，一般網路使用者則分配到一般Web伺服器。
註2：Nagios 為提高效能和準確性的網路監控軟體(詳細)
註3：Cacti 是一套類似 MRTG 的snmp流量監控跟系統資訊監控軟體，除此 Cacti 還可以外掛 Script 及 Templates作出各種的監控圖 ，有興趣的網友可以參考 CaCti官網 的說明
幸運的是有一套軟體 cacti 的發展就是基於讓 RRDTool 使用者更方便使用該軟體，
除了基本的 Snmp 流量跟系統資訊監控外，Cacti 也可外掛 Scripts 及加上 Templates 來作出各式各樣的監控圖
Reference: http://homeserver.com.tw/proxy-server/%E8%B2%A0%E8%BC%89%E5%B9%B3%E8%A1%A1%E5%99%A8%E5%84%AA%E7%BC%BA%E9%BB%9E%E6%AF%94%E8%BC%83nginx-vs-haproxy-vs-lvs/
Tuesday, January 24, 2012
Multiple default routes gateways with two different ISP ipfw PF setfib load balancing
Multiple default routes gateways with two different ISP ipfw PF setfib load balancing
Thanks to phoenix help I was able to setup multiple default routes, or a default route per network/interface to be precise, in Debian/Linux it is as simple as that:
# cat /etc/network/interfaces
That would be example topology (but more than 2 interfaces is also possible).
Now, You can not use the 'casual' defaultrouter="X" cause it will be only for one network.
We will have to use setfib(1) to create two (or more) separete routing tables per network/interface.
Note: FIB (Forward Information Base, synonym for a routing table here)
Add these lines to /boot/loader.conf file:
# vim /boot/loader.conf
### select one of following firewalls to enable.
#ipfw_load="YES" # Firewall
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log
### set number of routing tables to support.
net.fibs=2
Note: to view a list of options: cat /boot/defaults/loader.conf.
It will unfortunately require kernel recompile, but its not as that hard:
# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2
# vim MYKERNEL8.2
options ROUTETABLES=2 # max 16. 1 is back compatible.
Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
# cat /usr/src/sys/`uname -m`/conf/DEFAULTS
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2
# sync ; reboot
Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html
Remove the files generated during recompiling the kernel.
# rm -rf /usr/obj/usr/src/sys/MYKERNEL8.2
After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.
After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:
# setfib 0 /sbin/route add -net default 10.0.0.1
# setfib 1 /sbin/route add -net default 20.0.0.1
Note: with packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output.
# vi /etc/rc.conf
### WAN interfaces. You do not need to set defaultrouter="XXX" here.
ifconfig_wan0="inet 10.0.0.2/32"
ifconfig_wan1="inet 20.0.0.2/32"
### Note: commented out for multiple WAN routes setup. Routes will be added in /etc/rc.local file.
#defaultrouter="10.0.0.1"
### LAN static route
#static_routes="lan"
#route_lan="-net 192.168.50.0/24 192.168.0.11"
### PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""
### Enable as LAN gateway
gateway_enable="YES"
All the rest configuration resides in /etc/rc.local file:
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 10.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 20.0.0.1
### assing route tables to interfaces
### Note: uncomment following lines if you are using ipfw. Leave it if you are using PF.
###
#ipfw -f flush
#ipfw add allow ip from any to any via lo0
#ipfw add setfib 0 ip from any to any via wan0
#ipfw add setfib 1 ip from any to any via wan1
#ipfw add allow ip from any to any
These would be handy for restarting:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# /etc/rc.d/local restart
# pfctl -f /etc/pf.conf
View the fib:
# sysctl -a | grep fib
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 2
View the default route of each routing table:
# setfib 0 netstat -rn
# setfib 1 netstat -rn
Try to ping:
# setfib 0 ping google.com
# setfib 1 ping google.com
===============================================================
pf.conf
Looking for something like this?
# man 1 setfib
# man 2 setfib
And pf.conf(5) has the route-to and reply-to directives, of course.
# man 5 pf.conf | less +/rtable
# man 5 pf.conf | less +/route-to
# man 5 pf.conf | less +/reply-to
# vim /etc/pf.conf
### [VARIABLES]
wan_if0 = "wan0"
wan_if1 = "wan1"
dmz_if0 = "dmz0"
www0_ip0 = "192.168.0.3"
www0_ip1 = "192.168.0.4"
##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"
### [NAT]
nat on $wan_if0 from $dmz_if0/24 to any -> $wan_if0
nat on $wan_if1 from $dmz_if0/24 to any -> $wan_if1
### [HTTP] RDR Outside to DMZ
rdr on $wan_if0 proto tcp from any to $wan_if0 port 80 -> $www0_ip0
rdr on $wan_if1 proto tcp from any to $wan_if1 port 80 -> $www0_ip1
### [HTTP] outside to router
pass in quick on $wan_if0 proto {tcp} from any to $wan_if0 port 80 rtable 0
pass in quick on $wan_if1 proto {tcp} from any to $wan_if1 port 80 rtable 1
### [HTTP] router to dmz
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip0 port 80 rtable 0
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip1 port 80 rtable 1
Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53
You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp
Or just to/from a certain host:
# tcpdump -ni re0 icmp host 172.16.70.12
Reading a PF (packet filter) log file
The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.
To view the log file:
# tcpdump -n -e -ttt -r /var/log/pf.log
Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:
# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0
Filtering Log Output
Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:
# tcpdump -n -e -ttt -r /var/log/pf.log port 80
set up PF (packet filter) firewall on FreeBSD 8.1
Other Solution:
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html
http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
===============================================================
Just a quick note, if doing this on a router with a single internal interface.
Traffic originating on the local network will go out the default route of FIB 0.
Only traffic coming in on the second public interface will go out the same interface.
IOW, the setup above is really only useful for incoming traffic, to make sure that it goes back out the correct interface.
However, a few more IPFW rules can be added to classify traffic on the internal NIC.
It all depends on what you want to accomplish.
Yes, its mainly for that purpose, to assign proper gateways to networks/interfaces (which is a lot more easier @ Debian in /etc/network/interfaces)
We may even use FIB# 1 and 2 for these networks, and use FIB# 0 with other NIC as a default interface, posibilities are of course big, but I wanted to point posibility of having a gateway per network.
http://www.daemonforums.org/showthread.php?t=4610
===============================================================
setfib selects the routing table for locally originated
outgoing packets. Besides locally originated packets, there
are packets arriving from the network and need to be forwarded.
These packets can be classified in a specific routing table
with the aid of ipfw. That's all there is. I can't think
of something else that needs to be thought with regard to
multiple routing tables.
HTH, Nikos
http://lists.freebsd.org/pipermail/freebsd-questions/2009-July/202462.html
===============================================================
Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I've never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.
I have finally, at long last, found the solution.
The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn't route it's responses to the interface that they were received on, it uses the routing table to determine where to send it's responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It's really that simple. There is no logic for having multiple paths to non-local networks.
Frequently Offered Solution: Since I use pf for my firewall, I'm frequently told to use pf's route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.
The Real Solution: FreeBSD has support for multiple routing tables. It's little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support ("options ROUTETABLES=2"), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.
So here are the steps to solving this problem:
Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.
# cat /usr/src/sys/`uname -m`/conf/MYKERNEL8.2 | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
Step 2: Disable the SSH daemon in your rc.conf:
# cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.local
Step 3: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1
# vim /etc/ssh/sshd_config.rtable0
ListenAddress 20.0.0.2
ListenAddress 192.168.0.1
# vim /etc/ssh/sshd_config.rtable1
ListenAddress 30.0.0.2
ListenAddress 192.168.0.2
Note: Multiple ListenAddress options are permitted.
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1
### Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1
Step 4: Add rtable awareness to your pf.conf file:
[root@router]~ # cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on wan0 inet proto tcp from wan0:network to (wan0) port 22 rtable 0
Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.
In my above config it's worth pointing out that it doesn't actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.
High Availability HA cluster solution for FreeBSD
HAST + CARP + ZFS
HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.
CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.
# man carp
# man ng_one2many
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
===============================================================
Reference:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html
High Availability HA cluster solution for freebsd
http://www.daemonforums.org/showthread.php?t=4610
http://freebsd-forums.liquidneon.com/showthread.php?t=20166
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
http://www.melen.org/u/jan/wp/?p=102
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
Running PF & IPFW Together
Prevent default gateway route and resolv.conf being overwritten by DHCP
Thanks to phoenix help I was able to setup multiple default routes, or a default route per network/interface to be precise, in Debian/Linux it is as simple as that:
# cat /etc/network/interfaces
iface eth0 inet static
address 10.0.0.2
netmask 255.255.255.252
gateway 10.0.0.1
iface eth1 inet static
address 20.0.0.2
netmask 255.255.255.252
gateway 20.0.0.1
That would be example topology (but more than 2 interfaces is also possible).
ISP NETWORK 0 ISP NETWORK 1
\ /
\ /
\ /
ROUTER 0 ROUTER 1
10.0.0.1/30 20.0.0.1/30
\ /
+------\----------------/------+
| \ / |
| wan0 wan1 |
| 10.0.0.2 20.0.0.2 |
| |
| FREEBSD (Firewall Router) |
| |
| dmz0 |
| 192.168.0.1/24 |
| 192.168.0.2/24 |
+------------------------------+
|
+--------+
| switch +
+--------+
|
/
______/
/
+--------/-----+
| / |
| lan0 |
| 192.168.0.3 |
| 192.168.0.4 |
| |
| Web Server 0 |
+--------------+
Now, You can not use the 'casual' defaultrouter="X" cause it will be only for one network.
We will have to use setfib(1) to create two (or more) separete routing tables per network/interface.
Note: FIB (Forward Information Base, synonym for a routing table here)
Add these lines to /boot/loader.conf file:
# vim /boot/loader.conf
### select one of following firewalls to enable.
#ipfw_load="YES" # Firewall
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log
### set number of routing tables to support.
net.fibs=2
Note: to view a list of options: cat /boot/defaults/loader.conf.
It will unfortunately require kernel recompile, but its not as that hard:
# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2
# vim MYKERNEL8.2
options ROUTETABLES=2 # max 16. 1 is back compatible.
Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
# cat /usr/src/sys/`uname -m`/conf/DEFAULTS
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2
# sync ; reboot
Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html
Remove the files generated during recompiling the kernel.
# rm -rf /usr/obj/usr/src/sys/MYKERNEL8.2
After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.
After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:
# setfib 0 /sbin/route add -net default 10.0.0.1
# setfib 1 /sbin/route add -net default 20.0.0.1
Note: with packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output.
# vi /etc/rc.conf
### WAN interfaces. You do not need to set defaultrouter="XXX" here.
ifconfig_wan0="inet 10.0.0.2/32"
ifconfig_wan1="inet 20.0.0.2/32"
### Note: commented out for multiple WAN routes setup. Routes will be added in /etc/rc.local file.
#defaultrouter="10.0.0.1"
### LAN static route
#static_routes="lan"
#route_lan="-net 192.168.50.0/24 192.168.0.11"
### PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""
### Enable as LAN gateway
gateway_enable="YES"
All the rest configuration resides in /etc/rc.local file:
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 10.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 20.0.0.1
### assing route tables to interfaces
### Note: uncomment following lines if you are using ipfw. Leave it if you are using PF.
###
#ipfw -f flush
#ipfw add allow ip from any to any via lo0
#ipfw add setfib 0 ip from any to any via wan0
#ipfw add setfib 1 ip from any to any via wan1
#ipfw add allow ip from any to any
These would be handy for restarting:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# /etc/rc.d/local restart
# pfctl -f /etc/pf.conf
View the fib:
# sysctl -a | grep fib
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 2
View the default route of each routing table:
# setfib 0 netstat -rn
# setfib 1 netstat -rn
Try to ping:
# setfib 0 ping google.com
# setfib 1 ping google.com
===============================================================
pf.conf
Looking for something like this?
# man 1 setfib
# man 2 setfib
And pf.conf(5) has the route-to and reply-to directives, of course.
# man 5 pf.conf | less +/rtable
# man 5 pf.conf | less +/route-to
# man 5 pf.conf | less +/reply-to
# vim /etc/pf.conf
### [VARIABLES]
wan_if0 = "wan0"
wan_if1 = "wan1"
dmz_if0 = "dmz0"
www0_ip0 = "192.168.0.3"
www0_ip1 = "192.168.0.4"
##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"
### [NAT]
nat on $wan_if0 from $dmz_if0/24 to any -> $wan_if0
nat on $wan_if1 from $dmz_if0/24 to any -> $wan_if1
### [HTTP] RDR Outside to DMZ
rdr on $wan_if0 proto tcp from any to $wan_if0 port 80 -> $www0_ip0
rdr on $wan_if1 proto tcp from any to $wan_if1 port 80 -> $www0_ip1
### [HTTP] outside to router
pass in quick on $wan_if0 proto {tcp} from any to $wan_if0 port 80 rtable 0
pass in quick on $wan_if1 proto {tcp} from any to $wan_if1 port 80 rtable 1
### [HTTP] router to dmz
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip0 port 80 rtable 0
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip1 port 80 rtable 1
Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53
You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp
Or just to/from a certain host:
# tcpdump -ni re0 icmp host 172.16.70.12
Reading a PF (packet filter) log file
The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.
To view the log file:
# tcpdump -n -e -ttt -r /var/log/pf.log
Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:
# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0
Filtering Log Output
Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:
# tcpdump -n -e -ttt -r /var/log/pf.log port 80
set up PF (packet filter) firewall on FreeBSD 8.1
Other Solution:
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html
http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
===============================================================
Just a quick note, if doing this on a router with a single internal interface.
Traffic originating on the local network will go out the default route of FIB 0.
Only traffic coming in on the second public interface will go out the same interface.
IOW, the setup above is really only useful for incoming traffic, to make sure that it goes back out the correct interface.
However, a few more IPFW rules can be added to classify traffic on the internal NIC.
It all depends on what you want to accomplish.
Yes, its mainly for that purpose, to assign proper gateways to networks/interfaces (which is a lot more easier @ Debian in /etc/network/interfaces)
We may even use FIB# 1 and 2 for these networks, and use FIB# 0 with other NIC as a default interface, posibilities are of course big, but I wanted to point posibility of having a gateway per network.
http://www.daemonforums.org/showthread.php?t=4610
===============================================================
setfib selects the routing table for locally originated
outgoing packets. Besides locally originated packets, there
are packets arriving from the network and need to be forwarded.
These packets can be classified in a specific routing table
with the aid of ipfw. That's all there is. I can't think
of something else that needs to be thought with regard to
multiple routing tables.
HTH, Nikos
http://lists.freebsd.org/pipermail/freebsd-questions/2009-July/202462.html
===============================================================
Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I've never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.
I have finally, at long last, found the solution.
The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn't route it's responses to the interface that they were received on, it uses the routing table to determine where to send it's responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It's really that simple. There is no logic for having multiple paths to non-local networks.
Frequently Offered Solution: Since I use pf for my firewall, I'm frequently told to use pf's route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.
The Real Solution: FreeBSD has support for multiple routing tables. It's little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support ("options ROUTETABLES=2"), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.
So here are the steps to solving this problem:
Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.
# cat /usr/src/sys/`uname -m`/conf/MYKERNEL8.2 | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
Step 2: Disable the SSH daemon in your rc.conf:
# cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.local
Step 3: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1
# vim /etc/ssh/sshd_config.rtable0
ListenAddress 20.0.0.2
ListenAddress 192.168.0.1
# vim /etc/ssh/sshd_config.rtable1
ListenAddress 30.0.0.2
ListenAddress 192.168.0.2
Note: Multiple ListenAddress options are permitted.
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1
### Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1
Step 4: Add rtable awareness to your pf.conf file:
[root@router]~ # cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on wan0 inet proto tcp from wan0:network to (wan0) port 22 rtable 0
Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.
In my above config it's worth pointing out that it doesn't actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.
High Availability HA cluster solution for FreeBSD
HAST + CARP + ZFS
HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.
CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.
# man carp
# man ng_one2many
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
===============================================================
Reference:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html
High Availability HA cluster solution for freebsd
http://www.daemonforums.org/showthread.php?t=4610
http://freebsd-forums.liquidneon.com/showthread.php?t=20166
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
http://www.melen.org/u/jan/wp/?p=102
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
Running PF & IPFW Together
Prevent default gateway route and resolv.conf being overwritten by DHCP
Friday, January 6, 2012
How to install and configure HAProxy as an HTTP load balancer
# cd /usr/ports/net/haproxy ; make install
Check the following directories for further info:
documentation: '/usr/local/share/doc/haproxy'
examples: '/usr/local/share/examples/haproxy'
# ls /usr/local/share/examples/haproxy
# vim /usr/local/etc/haproxy.conf
global
maxconn 4096
pidfile /var/run/haproxy.pid
daemon
defaults
mode http
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen YOUR_GROUP_NAME aaa.bbb.ccc.ddd:80
mode http
cookie GALAXY insert
balance roundrobin
option httpclose
option forwardfor
stats enable
stats auth myuser:mypass
server EARTH 192.168.100.120:80 cookie MYGROUP_SERVER_01 check
server MOON 192.168.100.156:80 cookie MYGROUP_SERVER_02 check
Note: aaa.bbb.ccc.ddd is your public IP address. or set it to 0.0.0.0:80.
# echo 'haproxy_enable="YES"' >> /etc/rc.conf
# /usr/local/etc/rc.d/haproxy start
Go to See HAProxy Stats
http://test.local/haproxy?stats
Nginx
In the configuration above, we used the port 8080 for the local/private Web servers. If these Web servers are running Nginx, you will need to include the following configuration in your http, server or location block (nginx.conf):
port_in_redirect off;
This is needed because if you try to reach this:
http://aaa.bbb.ccc.ddd/test (NO ending slash)
Then Nginx will try to redirect you there:
http://aaa.bbb.ccc.ddd:8080/test/
Which will obviously not work. So if you're running Nginx and that you decided to use a port different of 80, you will need this setting.
===
haproxy with multiple IP in one server
there are two things you can do. First, you can have as many "bind" lines as you want in a "listen" or "frontend" section. So if the only thing that changes is the IP, then it's the best thing to do. Second, if you have any reason to apply a different processing to each IP (eg: different ACLs, or rewrite rules), then you should use "frontend"+"backend" instead of "listen". A "listen" section is exactly a frontend plus a backend, both in the same section. By having multiple frontends, you can define what type of processing you want on each IP address, and make them all use the same backend (using the "default_backend" rule). And all your servers will only be placed in this single backend, with the same cookie name, same LB algorithm, same health checks, etc...
Hoping this helps, Willy
Reference:
http://www.techbabu.com/2010/03/installing-haproxy-on-freebsd/
http://www.softwareprojects.com/resources/programming/t-how-to-install-and-configure-haproxy-as-an-http-loa-1752.html
http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/
http://serverfault.com/questions/99497/how-many-reverse-proxies-nginx-haproxy-is-too-many
http://serverfault.com/questions/51691/need-haproxy-varnish-nginx-setup-suggestions
Check the following directories for further info:
documentation: '/usr/local/share/doc/haproxy'
examples: '/usr/local/share/examples/haproxy'
# ls /usr/local/share/examples/haproxy
# vim /usr/local/etc/haproxy.conf
global
maxconn 4096
pidfile /var/run/haproxy.pid
daemon
defaults
mode http
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen YOUR_GROUP_NAME aaa.bbb.ccc.ddd:80
mode http
cookie GALAXY insert
balance roundrobin
option httpclose
option forwardfor
stats enable
stats auth myuser:mypass
server EARTH 192.168.100.120:80 cookie MYGROUP_SERVER_01 check
server MOON 192.168.100.156:80 cookie MYGROUP_SERVER_02 check
Note: aaa.bbb.ccc.ddd is your public IP address. or set it to 0.0.0.0:80.
# echo 'haproxy_enable="YES"' >> /etc/rc.conf
# /usr/local/etc/rc.d/haproxy start
Go to See HAProxy Stats
http://test.local/haproxy?stats
Nginx
In the configuration above, we used the port 8080 for the local/private Web servers. If these Web servers are running Nginx, you will need to include the following configuration in your http, server or location block (nginx.conf):
port_in_redirect off;
This is needed because if you try to reach this:
http://aaa.bbb.ccc.ddd/test (NO ending slash)
Then Nginx will try to redirect you there:
http://aaa.bbb.ccc.ddd:8080/test/
Which will obviously not work. So if you're running Nginx and that you decided to use a port different of 80, you will need this setting.
===
haproxy with multiple IP in one server
there are two things you can do. First, you can have as many "bind" lines as you want in a "listen" or "frontend" section. So if the only thing that changes is the IP, then it's the best thing to do. Second, if you have any reason to apply a different processing to each IP (eg: different ACLs, or rewrite rules), then you should use "frontend"+"backend" instead of "listen". A "listen" section is exactly a frontend plus a backend, both in the same section. By having multiple frontends, you can define what type of processing you want on each IP address, and make them all use the same backend (using the "default_backend" rule). And all your servers will only be placed in this single backend, with the same cookie name, same LB algorithm, same health checks, etc...
Hoping this helps, Willy
Reference:
http://www.techbabu.com/2010/03/installing-haproxy-on-freebsd/
http://www.softwareprojects.com/resources/programming/t-how-to-install-and-configure-haproxy-as-an-http-loa-1752.html
http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/
http://serverfault.com/questions/99497/how-many-reverse-proxies-nginx-haproxy-is-too-many
http://serverfault.com/questions/51691/need-haproxy-varnish-nginx-setup-suggestions
Friday, December 30, 2011
combine two or multiple NIC network cards ISP using network link aggregation and load balance round robin
Hard disk are getting bigger and bigger. Network interface card (NIC) and switch are getting cheaper and cheaper. It usually means more services can squeeze into a server. Combining multiple network interface into 1 big pipe would be a big plus for services that is bandwidth hungry. e.g. file server, email server & etc
Combining 2 or more network interface have a few advantages. One of them is load balancing. Load balancing enables the network interface card to share the load among the 2 or more network interface card defined. It serves as redundancy as well. But load balancing does not makes full use of the 2 or more network interface, it is meant to share network traffic load.
To fully utilize 2 or more network interface links, link aggregation should be used. Link aggregation is a method to combine 2 or more network interface to become a bigger pipe. It is also a way to combine 2 or more interface to use the same IP address. Using link aggregation can also create redundancy, if one of the link fails, the other links will take over.
This post is using FreeBSD as a server because FreeBSD have all the ports (software packages) you will need and server administration on it is made to ease systems administration daily task.Without further ado, below are the steps to configure 2 (or more) network interfaces to work as 1 big trunk or round robin :
Scenario
rl0 = 1st interface card device name
rl1 = 2nd interface card device name
192.168.0.78 = IP address used on both 1st & 2nd network interface card
Setup Link Aggregation on 2 (or more) network interface card
kldload /boot/kernel/ng_ether.ko
kldload /boot/kernel/ng_one2many.ko
ifconfig rl0 up
ifconfig rl1 up
ngctl mkpeer rl0: one2many upper one
ngctl connect rl0: rl0:upper lower many0
ngctl connect rl1: rl0:upper lower many1
ngctl msg rl1: setpromisc 1
ngctl msg rl1: setautosrc 0
ngctl msg rl0:upper \
setconfig "{ xmitAlg=1 failAlg=2 enabledLinks=[ 1 1 ] }"
ifconfig rl0 inet 192.168.0.78
Restore NIC to original status from the previous setup
ngctl rmhook em0: lower
(remove notes / services from inteface rl0)
Setup Round Robin on 2 (or more) network interface card
kldload /boot/kernel/ng_ether.ko
kldload /boot/kernel/ng_one2many.ko
ifconfig rl0 up
ifconfig rl1 up
ngctl mkpeer rl0: one2many upper one
ngctl connect rl0: rl0:upper lower many0
ngctl connect rl1: rl0:upper lower many1
ngctl msg rl1: setpromisc 1
ngctl msg rl1: setautosrc 0
ngctl msg rl0:upper \
setconfig "{ xmitAlg=2 failAlg=2 enabledLinks=[ 1 1 ] }"
ifconfig rl0 inet 192.168.0.78
Permanently make the changes take effect, put it in rc.local
e.g.
kldload /boot/kernel/ng_ether.ko
kldload /boot/kernel/ng_one2many.ko
ifconfig rl0 up
ifconfig rl1 up
ngctl mkpeer rl0: one2many upper one
ngctl connect rl0: rl0:upper lower many0
ngctl connect rl1: rl0:upper lower many1
ngctl msg rl1: setpromisc 1
ngctl msg rl1: setautosrc 0
ngctl msg rl0:upper \
setconfig "{ xmitAlg=1 failAlg=2 enabledLinks=[ 1 1 ] }"
ifconfig rl0 inet 192.168.0.78
ngctl can be run interactively for diagnostic purpose. Below list a few example :
list ### show list of nodes ###
show em0: ### show information of a node tied interface em0 ###
Au revoir !!!
Reference:
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
Combining 2 or more network interface have a few advantages. One of them is load balancing. Load balancing enables the network interface card to share the load among the 2 or more network interface card defined. It serves as redundancy as well. But load balancing does not makes full use of the 2 or more network interface, it is meant to share network traffic load.
To fully utilize 2 or more network interface links, link aggregation should be used. Link aggregation is a method to combine 2 or more network interface to become a bigger pipe. It is also a way to combine 2 or more interface to use the same IP address. Using link aggregation can also create redundancy, if one of the link fails, the other links will take over.
This post is using FreeBSD as a server because FreeBSD have all the ports (software packages) you will need and server administration on it is made to ease systems administration daily task.Without further ado, below are the steps to configure 2 (or more) network interfaces to work as 1 big trunk or round robin :
Scenario
rl0 = 1st interface card device name
rl1 = 2nd interface card device name
192.168.0.78 = IP address used on both 1st & 2nd network interface card
Setup Link Aggregation on 2 (or more) network interface card
kldload /boot/kernel/ng_ether.ko
kldload /boot/kernel/ng_one2many.ko
ifconfig rl0 up
ifconfig rl1 up
ngctl mkpeer rl0: one2many upper one
ngctl connect rl0: rl0:upper lower many0
ngctl connect rl1: rl0:upper lower many1
ngctl msg rl1: setpromisc 1
ngctl msg rl1: setautosrc 0
ngctl msg rl0:upper \
setconfig "{ xmitAlg=1 failAlg=2 enabledLinks=[ 1 1 ] }"
ifconfig rl0 inet 192.168.0.78
Restore NIC to original status from the previous setup
ngctl rmhook em0: lower
(remove notes / services from inteface rl0)
Setup Round Robin on 2 (or more) network interface card
kldload /boot/kernel/ng_ether.ko
kldload /boot/kernel/ng_one2many.ko
ifconfig rl0 up
ifconfig rl1 up
ngctl mkpeer rl0: one2many upper one
ngctl connect rl0: rl0:upper lower many0
ngctl connect rl1: rl0:upper lower many1
ngctl msg rl1: setpromisc 1
ngctl msg rl1: setautosrc 0
ngctl msg rl0:upper \
setconfig "{ xmitAlg=2 failAlg=2 enabledLinks=[ 1 1 ] }"
ifconfig rl0 inet 192.168.0.78
Permanently make the changes take effect, put it in rc.local
e.g.
kldload /boot/kernel/ng_ether.ko
kldload /boot/kernel/ng_one2many.ko
ifconfig rl0 up
ifconfig rl1 up
ngctl mkpeer rl0: one2many upper one
ngctl connect rl0: rl0:upper lower many0
ngctl connect rl1: rl0:upper lower many1
ngctl msg rl1: setpromisc 1
ngctl msg rl1: setautosrc 0
ngctl msg rl0:upper \
setconfig "{ xmitAlg=1 failAlg=2 enabledLinks=[ 1 1 ] }"
ifconfig rl0 inet 192.168.0.78
ngctl can be run interactively for diagnostic purpose. Below list a few example :
list ### show list of nodes ###
show em0: ### show information of a node tied interface em0 ###
Au revoir !!!
Reference:
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
Monday, November 14, 2011
用 pf + relayd 架設 Layer 4 Switch (Server Load Balancer)
源於 OpenBSD，port 到 FreeBSD 的 pf 本身就可以作 load balancer，但是不能作 healthy check。要作 healthy check 的話，必須加上relayd。另外用 CARP(4) 與 PFSYNC(4) 可以做到 Active-Standby 的 Redundancy HA，也就是當一台 SLB (Server Load Balancer) 死掉的時候，另一台會自動起來提供服務。詳細的作法在參考資料有，因此這裡不贅述。基本的想法為：
- 用 pf 設定防火牆與 NAT，把 incoming port 80 的 destination 換成 server pool 內的任一台。
- 用 carp 來保證 HA。
- 用 pfsync 來 sync 兩台 SLB 之間的 pf firewall state。
- 用 relayd 來檢查 server 的狀態，如果不能連線的時候，把 server 從 pool 內拿掉。
這樣就可以達到 Layer 4 Switch 的功能，在 relayd 裡面稱為 REDIRECT 模式，缺點是 pool 內所有的 server 都必須把 default gateway 設為 SLB。
另外 relayd 也可以採用類似 haproxy 的方式，由 relayd 跟 client 與 server 各建立一個連線，作 Layer 7 的 forwarding。這在 relayd 裡面稱為 RELAY 模式。
設定 relay 模式的時候，可以對 HTTP Header 作修改，例如加上 X-Forwarded-For Header，或是根據 Host Header 或者 URI 來決定要分配到的 Pool。另外 healthy check 也可以由管理者自行撰寫 script，因此可以作比較複雜的檢查。設定方式可以參考 RELAYD.CONF(5)。
另外 relayd 也可以採用類似 haproxy 的方式，由 relayd 跟 client 與 server 各建立一個連線，作 Layer 7 的 forwarding。這在 relayd 裡面稱為 RELAY 模式。
設定 relay 模式的時候，可以對 HTTP Header 作修改，例如加上 X-Forwarded-For Header，或是根據 Host Header 或者 URI 來決定要分配到的 Pool。另外 healthy check 也可以由管理者自行撰寫 script，因此可以作比較複雜的檢查。設定方式可以參考 RELAYD.CONF(5)。
比起 haproxy 來說，relayd 可以輕易的達到 Full Transparent Proxy （後端的 Server 看到的 source ip 是真正的 ip）的功能而不需要 patch kernel，而且也可以作 graceful restart（ haproxy 在 restart 的時候服務會中斷）；不過 haproxy 有許多的實例已經驗證過它的效能，可以吃滿 10Gbps 網路，但 relayd 在 RELAY 模式的 效能還是未知數。另外，由於 relayd 與 pf 綁的很緊，因此目前只有 OpenBSD 與 FreeBSD 版，不能在 Linux 上執行。
參考資料：
Reference:
Tuesday, December 14, 2010
scaling drupal - an open-source infrastructure for high-traffic drupal sites
scaling drupal - an open-source infrastructure for high-traffic drupal sites
the authors of drupal have paid considerable attention to performance and scalability. consequently even a default install running on modest hardware can easily handle the demands of a small website. my four year old pc in my garage running a full lamp install, will happily serve up 50,000 page views in a day, providing solid end-user performance without breaking a sweat.
in this article i discuss scaling the database tier up and out. i compare database optimization and different database clustering techniques. i go on to explore the idea of database segmentation as a possibility for moderate drupal scaling.
the holy grail of drupal database scaling might very well be a drupal deployment on mysql cluster. if you've tried this, plan to try this or have opinions on the feasibility of an ndb "port" of drupal, i'd love to hear it.
Reference: http://www.johnandcailin.com/blog/john/scaling-drupal-open-source-infrastructure-high-traffic-drupal-sites
the authors of drupal have paid considerable attention to performance and scalability. consequently even a default install running on modest hardware can easily handle the demands of a small website. my four year old pc in my garage running a full lamp install, will happily serve up 50,000 page views in a day, providing solid end-user performance without breaking a sweat.when the times comes for scalability. moving of of the garage
if you are lucky, eventually the time comes when you need to service more users than your system can handle. your initial steps should clearly focus on getting the most out of the built-in drupal optimization functionality, considering drupal performance modules, optimizing your php (including considering op-code caching) and working on database performance. John VanDyk and Matt Westgate have an excellent chapter on this subject in their new book, "pro drupal development"once these steps are exhausted, inevitability you'll start looking at your hardware and network deployment.
a well designed deployment will not only increase your scalability, but will also enhance your redundancy by removing single points of failure. implemented properly, an unmodified drupal install can run on this new deployment, blissfully unaware of the clustering, routing and caching going on behind the scenes.
incremental steps towards scalability
in this article, i outline a step-by-step process for incrementally scaling your deployment, from a simple single-node drupal install running all components of the system, all the way to a load balanced, multi node system with database level optimization and clustering.
since you almost certainly don't want to jump straight from your single node system to the mother of all redundant clustered systems in one step, i've broken this down into 5 incremental steps, each one building on the last. each step along the way is a perfectly viable deployment.
tasty recipes
i give full step-by-step recipes for each deployment, that with a decent working knowledge of linux, should allow you to get a working system up and running. my examples are for apache2, mysql5 and drupal5 on debian etch, but may still be useful for other versions / flavors.note that these aren't battle-hardened production configurations, but rather illustrative minimal configurations that you can take and iterate to serve your specific needs.
the 5 deployment configurations
the table below outlines the properties of each of the suggested configurations:| step 0 | step 1 | step 2 | step 3 | step 4 | step 5 | |
|---|---|---|---|---|---|---|
| separate web and db | no | yes | yes | yes | yes | yes |
| clustered web tier | no | no | yes | yes | yes | yes |
| redundant load balancer | no | no | no | yes | yes | yes |
| db optimization and segmentation | no | no | no | no | yes | yes |
| clustered db | no | no | no | no | no | yes |
| scalabilty | poor- | poor | fair | fair | good | great |
| redundancy | poor- | poor- | fair | good | fair | great |
| setup ease | great | good | good | fair | poor | poor- |
step 0 - a basic drupal install
in step 0, i outline how to install drupal, mysql and apache to get a get a basic drupal install up-and-running on a single node. i also go over some of the basic configuration steps that you''ll probably want to follow, including cron scheduling, enabling clean urls, setting up a virtual host etc.
step 1 - a dedicated data server
in step 1, i go over a good first step to scaling drupal; creating a dedicated data server. by "dedicated data server" i mean a server that hosts both the database and a fileshare for node attachments etc. this splits the database server load from the web server, and lays the groundwork for a clustered web server deployment.
step 2 - sticky load balancing with apache mod_proxy
in step 2, i go over how to cluster your web servers. drupal generates a considerable load on the web server and can quickly become resource constrained there. having multiple web servers also increases the the redundancy of your deployment.
step 3 - using heartbeat to implement a redundant load balancer
in step 3, i discuss clustering your load balancer. one way to do this is to use heartbeat to provide instant failover to a redundant load balancer should your primary fail. while the method suggested below doesn't increase the loadbalancer scalability, which shouldn't be an issue for a reasonably sized deployment, it does increase your the redundancy.
step 4 - database segmentation using mysql proxy
in this article i discuss scaling the database tier up and out. i compare database optimization and different database clustering techniques. i go on to explore the idea of database segmentation as a possibility for moderate drupal scaling.
step 5 - the holy grail?
the holy grail of drupal database scaling might very well be a drupal deployment on mysql cluster. if you've tried this, plan to try this or have opinions on the feasibility of an ndb "port" of drupal, i'd love to hear it.
Reference: http://www.johnandcailin.com/blog/john/scaling-drupal-open-source-infrastructure-high-traffic-drupal-sites
Tuesday, December 7, 2010
high performance caching reverse proxy: Varnish (安裝架設篇)
在上禮拜跟 DarkHero 兄聊到 How To Build a Scalable Web Site (3/6) 的上課講義，互相討論了 MySQL Load balance 以及 http reverse proxy 的方式，以前自己有用 HAProxy 當作 Web 平衡負載，順便紀錄了 HAProxy FreeBSD 安裝方式，這次要來介紹今天重點：Varnish Cache Server，在近幾年流行的 Caching 機制，大家會想到 Squid，只要您設定良好的 Squid 參數，它一定運作的非常穩定，然而它的核心依然是 forward proxy，要架設成 Reverse Proxy 還必需要設定一些參數才可以達到，是有一定的困難性，然而 Varnish Cache Server 底層就是高效能 caching reverse proxy，也因為 Squid 是 1980 年發展出來的，程式架構過於老舊，可以參考 ArchitectNotes 瞭解這部份詳情。也許您會問到 Varnish 可以架設成 forward proxy 嗎？答案是可以的，但是您也許不會這麼做，因為它需要 DNS 技術，以及需要一個非常大且複雜的 Varnish VCL(Varnish Configuration Language) file。
Reference: http://blog.wu-boy.com/2010/03/21/2054/
1. 今天要介紹如何在 FreeBSD 系統安裝，在介紹之前，系統必須先安裝好 apache，這樣才可以正確啟動，利用 ports 安裝：
cd /usr/ports/www/varnish/
make install clean
make install clean
2. 修改 /etc/rc.conf
# varnishd
varnishd_enable="YES"
varnishd_listen=":80"
varnishd_admin=":81"
varnishd_backend=":5566"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_enable="YES"
varnishd_listen=":80"
varnishd_admin=":81"
varnishd_backend=":5566"
varnishd_config="/usr/local/etc/varnish/default.vcl"
上面設定意思是說 Varnish listen on port 80，傳送 traffic 到後端 5566 port，管理連接埠是 81，也可以使用指令方式：
varnishd -a :80 -b localhost:8080 -T localhost:6082
Varnishd listen on port 80，and forwarding traffic to a web server listen on localhost port 8080. It also turns on the management interface on port 6082.
3. 修改 default.vcl (Varnish Configuration Language)
VCL 檔案告訴 Varnishd 正確的處理每個 request processing，包含在接受到 request 之前所處理的行為 vcl_recv()，另外還有 vcl_hit()、vcl_miss() 等…，都是用來處理 cache 如果存在或者是不存在時的情境 request。FreeBSD 預設放在 /usr/local/etc/varnish/default.vcl。打開此檔案，您會看到：
backend default {
.host = "127.0.0.1";
.port = "80";
}
.host = "127.0.0.1";
.port = "80";
}
您只要把 host = “127.0.0.1″ 改成你後端要連接的 ip 或者是 host，這樣 Varnish 會 forward traffic 到您的 web server。接下來只要啟動 apache 跟 Varnish 就算是初步架設完成。
/usr/local/etc/rc.d/apache22 restart
/usr/local/etc/rc.d/varnishd restart
/usr/local/etc/rc.d/varnishd restart
大家可以看到 61.*.*.* 連到本機 80 port，接下來 Varnish 在開啟隨機 57475 port 連接到 Web Server 5566 port。
Q:如何讓 apache 紀錄正確的 Client IP 到 log 檔案呢？
1. 打開 Vcl config 檔案，寫入
Varnish configuration:
sub vcl_recv {
# Add a unique header containing the client address
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
# [...]
}
# Add a unique header containing the client address
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
# [...]
}
2. 開啟 apache httpd.conf 加入此行：
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined
Q:如何讓 PHP 程式紀錄正確的 Client IP？
可以參考底下程式碼就可以完全抓到 Proxy 後面真正使用者IP，否則您的 Web 只會抓到 Reverse Proxy 的 IP Address。
function getIP() {
if (validip($_SERVER["HTTP_CLIENT_IP"])) {
return $_SERVER["HTTP_CLIENT_IP"];
}
foreach(explode(",",$_SERVER["HTTP_X_FORWARDED_FOR"]) as $ip) {
if (validip(trim($ip))) {return $ip;}
}
if (validip($_SERVER["HTTP_X_FORWARDED"]))
{
return $_SERVER["HTTP_X_FORWARDED"];
}
else if (validip($_SERVER["HTTP_FORWARDED_FOR"]))
{
return $_SERVER["HTTP_FORWARDED_FOR"];
}
else if (validip($_SERVER["HTTP_FORWARDED"]))
{
return $_SERVER["HTTP_FORWARDED"];
}
else if (validip($_SERVER["HTTP_X_FORWARDED"]))
{
return $_SERVER["HTTP_X_FORWARDED"];
}
else
{
return $_SERVER["REMOTE_ADDR"];
}
}
function validip($ip) {
if (!empty($ip) && ip2long($ip)!=-1) {
$reserved_ips = array (
array('10.0.0.0','10.255.255.255'),
array('127.0.0.0','127.255.255.255'),
array('169.254.0.0','169.254.255.255'),
array('172.16.0.0','172.31.255.255'),
array('192.168.0.0','192.168.255.255'),
);
foreach ($reserved_ips as $r) {
$min = ip2long($r[0]);
$max = ip2long($r[1]);
if ((ip2long($ip) >= $min) && (ip2long($ip) <= $max)) return false;
}
return true;
}
else
{
return false;
}
}
if (validip($_SERVER["HTTP_CLIENT_IP"])) {
return $_SERVER["HTTP_CLIENT_IP"];
}
foreach(explode(",",$_SERVER["HTTP_X_FORWARDED_FOR"]) as $ip) {
if (validip(trim($ip))) {return $ip;}
}
if (validip($_SERVER["HTTP_X_FORWARDED"]))
{
return $_SERVER["HTTP_X_FORWARDED"];
}
else if (validip($_SERVER["HTTP_FORWARDED_FOR"]))
{
return $_SERVER["HTTP_FORWARDED_FOR"];
}
else if (validip($_SERVER["HTTP_FORWARDED"]))
{
return $_SERVER["HTTP_FORWARDED"];
}
else if (validip($_SERVER["HTTP_X_FORWARDED"]))
{
return $_SERVER["HTTP_X_FORWARDED"];
}
else
{
return $_SERVER["REMOTE_ADDR"];
}
}
function validip($ip) {
if (!empty($ip) && ip2long($ip)!=-1) {
$reserved_ips = array (
array('10.0.0.0','10.255.255.255'),
array('127.0.0.0','127.255.255.255'),
array('169.254.0.0','169.254.255.255'),
array('172.16.0.0','172.31.255.255'),
array('192.168.0.0','192.168.255.255'),
);
foreach ($reserved_ips as $r) {
$min = ip2long($r[0]);
$max = ip2long($r[1]);
if ((ip2long($ip) >= $min) && (ip2long($ip) <= $max)) return false;
}
return true;
}
else
{
return false;
}
}
Q:rotate Varnish log file every day？
打開 /etc/newsyslog.conf，加入底下兩行
/var/log/varnish.log 640 7 * @T00 JB /var/run/varnishlog.pid
/var/log/varnishncsa.log 640 7 * @T00 JB /var/run/varnishncsa.pid
/var/log/varnishncsa.log 640 7 * @T00 JB /var/run/varnishncsa.pid
每天12點進行 log 備份，使用 gzip 壓縮 log 檔案。
Reference: http://blog.wu-boy.com/2010/03/21/2054/
網站分流：簡易架設 HAProxy 伺服器
最近在玩這套 Web 的 Load Balance 軟體，其實這是之前我寫的一篇：[筆記] FreeBSD 一張網卡多重 ip 實現 Round Robin DNS Load Balancing，有網友留言給我說可以玩看看 HAProxy，這樣的確改善了很多效能，而且也是正確達到 load balance 的效果，不然用 DNS Robin DNS Load Balancing 的方式的卻沒辦法做的很好，關於 HAProxy 在 google 了一下，好像國內很少人在寫這方面的教學，我自己來寫一下筆記好了，其實我還不是對設定很熟悉，只是大概知道他的原理罷了，底下我們來看看官網的一張圖
[url=http://www.flickr.com/photos/appleboy/2603471691/][img]http://farm4.static.flickr.com/3211/2603471691_c77e333374_o.png[/img][/url]
其實原理很簡單，就是在最前面擺一台簡單的電腦架設 HAProxy 來達成分流功能，後端擺幾台 WEB 伺服器，然後最後面在擺一台資料庫MySQL或者是其他的資料庫，然後去設定 HAProxy 看看哪個ip進來就把它導向到後端的部份，然後後端機器就是設定一下 apache 的 virtualhost，其實這樣就差不多了
第一步：架設 FreeBSD 的 HAProxy
Port: haproxy-devel-1.3.12.2
Path: /usr/ports/net/haproxy-devel
Info: The Reliable, High Performance TCP/HTTP Load Balancer
Maint: hugo@barafranca.com
B-deps:
R-deps:
WWW: http://haproxy.1wt.eu/
Path: /usr/ports/net/haproxy-devel
Info: The Reliable, High Performance TCP/HTTP Load Balancer
Maint: hugo@barafranca.com
B-deps:
R-deps:
WWW: http://haproxy.1wt.eu/
進行安裝：
#
# cd /usr/ports/net/haproxy-devel
#
make install clean
#
# cd /usr/ports/net/haproxy-devel
#
make install clean
第二步：設定 HAProxy 的 conf 檔
#
# vi /usr/local/etc/haproxy.conf
#
global
maxconn 32768
user nobody
group nobody
daemon
nbproc 8
listen blog-balancer
bind 140.123.107.54:80
mode http
balance roundrobin
maxconn 32768
clitimeout 60000
srvtimeout 60000
contimeout 5000
retries 3
server blog1 192.168.1.1:80 weight 3 check
server blog2 192.168.1.2:80 weight 3 check
server blog3 192.168.1.3:80 weight 3 check
option forwardfor
option httpclose
option httplog
option redispatch
option dontlognull
# vi /usr/local/etc/haproxy.conf
#
global
maxconn 32768
user nobody
group nobody
daemon
nbproc 8
listen blog-balancer
bind 140.123.107.54:80
mode http
balance roundrobin
maxconn 32768
clitimeout 60000
srvtimeout 60000
contimeout 5000
retries 3
server blog1 192.168.1.1:80 weight 3 check
server blog2 192.168.1.2:80 weight 3 check
server blog3 192.168.1.3:80 weight 3 check
option forwardfor
option httpclose
option httplog
option redispatch
option dontlognull
上面資料，我是參考 網路大神 Gslin 大大的 wiki，其實我自己還不太瞭解怎麼設定，所以要去參考官網寫的設定教學
第三步：要在後端主機設定 apache virtualhost
<VirtualHost 10.1.2.3>
ServerAdmin webmaster@host.foo.com
DocumentRoot /www/docs/host.foo.com
ServerName host.foo.com
ErrorLog logs/host.foo.com-error_log
TransferLog logs/host.foo.com-access_log
</VirtualHost>
ServerAdmin webmaster@host.foo.com
DocumentRoot /www/docs/host.foo.com
ServerName host.foo.com
ErrorLog logs/host.foo.com-error_log
TransferLog logs/host.foo.com-access_log
</VirtualHost>
後端三台都是要加上這些設定，這樣在轉過去的時候，才會出現正確網站，大致設定這樣就可以 work
參考網站：
http://haproxy.1wt.eu/
商業服務的Rails HTTP Cluster觀念及測試
Load Balancing & QoS with HAProxy
http://www.ecase.com.cn/blog/?p=14
http://wiki.gslin.org/haproxy
期刊/商業服務的Rails伺服器叢集觀念與實做(下)
Reference: http://blog.wu-boy.com/2008/06/23/283/
FreeBSD 一張網卡多重 ip 實現 Round Robin DNS Load Balancing
來紀錄一下好了，其實這兩年前，我自己就試過了，當時在弄 web 系統 Load Balancing，因為找不到更好的方法，所以使用 Round Robin DNS Load Balancing 技術來達到這功能，可是當然這不是很準確的做到 load balance，只是能解決暫時性的問題，真正要做到 Load Balancing 可能要靠硬體的技術了。
我自己是在 FreeBSD 系統上面實做的，當然弄 Load Balancing 一定要在網卡上面綁定多重ip，Linux 作法跟 FreeBSD 不大相同
#
# Linux 作法
#
ifconfig eth0:0 inet xxx.xxx.xxx.xx(1~9) netmask 255.255.255.0 broadcast xxx.xxx.xxx.255
# Linux 作法
#
ifconfig eth0:0 inet xxx.xxx.xxx.xx(1~9) netmask 255.255.255.0 broadcast xxx.xxx.xxx.255
#
# FreeBSD 作法
#
# /etc/rc.conf - add a new IP address to the NIC
# 在 rc.conf 加入底下 entry
ifconfig_rl0_alias0="192.168.0.57 netmask 0xffffffff"
#
# 指令
#
ifconfig rl0 alias 192.168.0.57 netmask 0xffffffff
# FreeBSD 作法
#
# /etc/rc.conf - add a new IP address to the NIC
# 在 rc.conf 加入底下 entry
ifconfig_rl0_alias0="192.168.0.57 netmask 0xffffffff"
#
# 指令
#
ifconfig rl0 alias 192.168.0.57 netmask 0xffffffff
利用 Round Robin DNS Load Balancing 技術達到分流，有兩種作法，一種使用 CNAME 另一種就是 A record
1.DNS load balancing implementation (Multiple CNAMES)
先在正解設定檔裡面加入：
srv1 IN A 123.45.67.1
srv2 IN A 123.45.67.2
srv3 IN A 123.45.67.3
srv4 IN A 123.45.67.4
srv2 IN A 123.45.67.2
srv3 IN A 123.45.67.3
srv4 IN A 123.45.67.4
然後在使用 CNAME
www IN CNAME srv1.domain.tld.
IN CNAME srv2.domain.tld.
IN CNAME srv3.domain.tld.
IN CNAME srv4.domain.tld.
IN CNAME srv2.domain.tld.
IN CNAME srv3.domain.tld.
IN CNAME srv4.domain.tld.
然後在 named.conf 加入底下設定 [ For BIND 8 name servers ]
options {
multiple-cnames yes;
};
multiple-cnames yes;
};
2.DNS load balancing implementation (Multiple A Records) 我是用此方法：
www.domain.tld. 60 IN A 123.45.67.1
www.domain.tld. 60 IN A 123.45.67.2
www.domain.tld. 60 IN A 123.45.67.3
www.domain.tld. 60 IN A 123.45.67.4
www.domain.tld. 60 IN A 123.45.67.2
www.domain.tld. 60 IN A 123.45.67.3
www.domain.tld. 60 IN A 123.45.67.4
這樣上面的 server 就會互相轉換，非常方便，我是用此方法，TTL 60 這個可以在設定短一點，這樣他在切換的速度會更快喔
相關網站：
[url=http://blog.wu-boy.com/2007/03/11/74/][www] 網站分流問題[/url]
參考網站：
http://www.freebsddiary.org/ip-address-change.php
http://bbs.linuxsky.org/thread-638-1-5.html
http://content.websitegear.com/article/load_balance_dns.htm
Reference:
http://blog.wu-boy.com/2008/06/01/274/
Wednesday, November 25, 2009
High Availability with FreeBSD and CARP
High Availability with FreeBSD and CARP
Introduction
Recently i was contacted to implement a content filter proxy using OpenSource tools. It is a very simple task to do, but my client asked to make the solution with HA (high availability) and Linux.
Well, Linux is a great operating system, but I have never built a Linux HA solution before, so I started to look for some information about the options I had.
My first search was for Linux clusters and I found them to be very complicate to build and manage. I needed my solution to be simple, fast, secure and easy to manage and implement.
During my research, I found some references to the CARP (Common Address Redundancy Protocol) protocol, a very smart and simple solution from the great folks at the OpenBSD project.
In a very simple way, the CARP protocol can make 2 or more network interfaces share the same IP address number. When the MASTER interface goes down, the BACKUP interface automagicaly takes its place.
It was not very difficult to find that CARP was ported to FreeBSD, NetBSD and with UCARP (Userland CARP) it is possible to use it under Linux too.
I like OpenBSD, but I am more comfortable working with FreeBSD, so it is my first option.
Now you may be asking, what about Linux? Why not use it?
In my opinion FreeBSD (and others BSDs as well) are better documented and, particularly, the FreeBSD handbook has almost everything you need to implement a good server. In Linux you have to look at various sites and search a lot to find some help. Maybe I am wrong, but it is my experience until now.
Implementation
I will not cover FreeBSD installation here. Look at FreeBSD handbook for more information. It is one of the best software documentations out there for an OpenSource OS.
On FreeBSD, CARP must be compiled in the kernel before using it. Build a new kernel is simple and take only a few minutes in current machines.
Follow these steps on all machines that will be using CARP:
- cd /usr/src/sys/i386/conf
- cp GENERIC GENERIC-CARP
- echo "device carp" >> GENERIC-CARP
- config GENERIC-CARP
- cd ../compile/GENERIC-CARP
- make depend
- make
- make install
- make clean
- shutdown -r now
Those steps will compile the carp into FreeBSD kernel and reboot the system so the changes will take effect.
To understand carp, lets see the options first.
- Preemption: When you have your hosts configured with carp they can use preemption. It will make possible for one host, the one with lower advskew, to always be authoritative for the address in the carp interface. If you don’t use preemption one of the backup’s machines may become a MASTER (when the original master fails), but when the original master is online again, it will be a BACKUP. Using preemption the original MASTER will recover it’s status as soon as it goes up again.
- vhid: This is a number indicating the group of the carp interface. You may have various carp groups to build very elaborated and complex HA solutions. Here we will be using only one group. More than one group may be used to build an arp balance solution (not covered here).
- pass: This is a passphrase used to authenticate the hosts on your carp group. You will be using the same password on all hosts for the same carp group.
- advskew: This number controls the frequency on witch the master send advertisements to the other hosts. The host with the lowest number will be MASTER. You can build an hierarchy of hosts using this, determining the order that must be used in case of failures. The higher this number less frequently the advertisements will be sent, so this host may be a backup if there are others with lower advsew.
This are the options used here. There are other options that can be reviewed by reading the manual pages for carp.
Our configuration is very simple. 2 hosts sharing 1 IP address. When the master goes down, the backup takes over the IP, when master comes back, it must be master again. I don’t want load balance here and just want to keep my services in case of failures.
The MASTER /etc/rc.conf must use a network configuration like this:
# Master net conf
defaultrouter="10.1.1.1"
hostname="master.localdomain"
cloned_interfaces="carp0"
ifconfig_em0="inet 192.168.200.1 netmask 255.255.255.0"
ifconfig_em1="inet 10.1.1.2 netmask 255.255.255.0"
ifconfig_carp0="inet 10.1.1.10 netmask 255.255.255.0 vhid 1 pass mypassword advskew 0"
# End master net conf
The BACKUP /etc/rc.conf must be like this:
# BACKUP net conf
defaultrouter="10.1.1.1"
hostname="backup.localdomain"
cloned_interfaces="carp0"
ifconfig_em0="inet 192.168.200.2 netmask 255.255.255.0"
ifconfig_em1="inet 10.1.1.3 netmask 255.255.255.0"
ifconfig_carp0="inet 10.1.1.10 netmask 255.255.255.0 vhid 1 pass mypassword advskew 100"
# End BACKUP net conf
On both hosts you must add the following line to /etc/sysctl.conf:
net.inet.carp.preempt=1
Notice: The 192.168.200 interfaces are connected directly to provide a faster route between the hosts so I can use it to sync files, share resources without compromising the real network. This interface is not used in the carp configuration that will work only using the other interfaces.
That’s it. Just reboot and all configuration will be up. To test it, ping the 10.1.1.10 address from a remote machine and shutdown the MASTER. You will see that the ping will not stop.
If you run an ifconfig, it will be possible to see the interfaces running:
MASTER ifconfig:
carp0: flags=41 mtu 1500
inet 10.1.1.10 netmask 0xffff0000
carp: MASTER vhid 1 advbase 1 advskew 0
BACKUP ifconfig:
carp0: flags=41 mtu 1500
inet 10.1.1.10 netmask 0xffff0000
carp: BACKUP vhid 1 advbase 1 advskew 100
Conclusion
Now you can configure your services to listen on carp0 interface. An HTTP, FTP or proxy server can be listening on this interface and when master goes down the backup will be up and running. You will only need to set up some synchronization for your files, to do this you can use rsync over ssh in a cron job or even share a remote storage to not waste space. The best solution is the one that fits your needs, so be creative and use this to increase your availability.
CARP is a very simple solution to a very common problem. The configuration is very easy to build and understand and it can be used to build different kinds of HA services.
I hope this article helps you. Please leave your comments.
4 Comments »
The URI to TrackBack this entry is: http://tmartins.blogsome.com/2006/07/28/high-availability-with-freebsd-and-carp/trackback/
1.
Hi! Have you try use carp with multicast (for ex. mrouted)?
Comment by Andrey — June 7, 2008 @ 10:59 am
2.
I have never used it with mrouted. Sice CARP is a multicast protocol that may be an easy task.
Try the OpenBSD documentation. There lots of info in there (http://www.openbsd.org/faq/faq6.html#CARP).
On FreeBSD I remember to see some posts in a mail list talking about problems with CARP after mrouted being configured. Is that your problem?
Comment by tmartins — June 7, 2008 @ 9:49 pm
3.
I think this line:
echo “options carp” >> GENERIC-CARP
Needs to be:
echo “device carp” >> GENERIC-CARP
Good article!
Comment by Justin — July 9, 2009 @ 3:28 pm
4.
Thanks Justin. You are right for newer FreeBSD versions. This article is old and I wrote it for FreeBSD 5. It was “options carp” back on 2006. :)
Now is “device carp”. The article was changed to reflect this.
Comment by Thiago — July 9, 2009 @ 3:50 pm
Introduction
Recently i was contacted to implement a content filter proxy using OpenSource tools. It is a very simple task to do, but my client asked to make the solution with HA (high availability) and Linux.
Well, Linux is a great operating system, but I have never built a Linux HA solution before, so I started to look for some information about the options I had.
My first search was for Linux clusters and I found them to be very complicate to build and manage. I needed my solution to be simple, fast, secure and easy to manage and implement.
During my research, I found some references to the CARP (Common Address Redundancy Protocol) protocol, a very smart and simple solution from the great folks at the OpenBSD project.
In a very simple way, the CARP protocol can make 2 or more network interfaces share the same IP address number. When the MASTER interface goes down, the BACKUP interface automagicaly takes its place.
It was not very difficult to find that CARP was ported to FreeBSD, NetBSD and with UCARP (Userland CARP) it is possible to use it under Linux too.
I like OpenBSD, but I am more comfortable working with FreeBSD, so it is my first option.
Now you may be asking, what about Linux? Why not use it?
In my opinion FreeBSD (and others BSDs as well) are better documented and, particularly, the FreeBSD handbook has almost everything you need to implement a good server. In Linux you have to look at various sites and search a lot to find some help. Maybe I am wrong, but it is my experience until now.
Implementation
I will not cover FreeBSD installation here. Look at FreeBSD handbook for more information. It is one of the best software documentations out there for an OpenSource OS.
On FreeBSD, CARP must be compiled in the kernel before using it. Build a new kernel is simple and take only a few minutes in current machines.
Follow these steps on all machines that will be using CARP:
- cd /usr/src/sys/i386/conf
- cp GENERIC GENERIC-CARP
- echo "device carp" >> GENERIC-CARP
- config GENERIC-CARP
- cd ../compile/GENERIC-CARP
- make depend
- make
- make install
- make clean
- shutdown -r now
Those steps will compile the carp into FreeBSD kernel and reboot the system so the changes will take effect.
To understand carp, lets see the options first.
- Preemption: When you have your hosts configured with carp they can use preemption. It will make possible for one host, the one with lower advskew, to always be authoritative for the address in the carp interface. If you don’t use preemption one of the backup’s machines may become a MASTER (when the original master fails), but when the original master is online again, it will be a BACKUP. Using preemption the original MASTER will recover it’s status as soon as it goes up again.
- vhid: This is a number indicating the group of the carp interface. You may have various carp groups to build very elaborated and complex HA solutions. Here we will be using only one group. More than one group may be used to build an arp balance solution (not covered here).
- pass: This is a passphrase used to authenticate the hosts on your carp group. You will be using the same password on all hosts for the same carp group.
- advskew: This number controls the frequency on witch the master send advertisements to the other hosts. The host with the lowest number will be MASTER. You can build an hierarchy of hosts using this, determining the order that must be used in case of failures. The higher this number less frequently the advertisements will be sent, so this host may be a backup if there are others with lower advsew.
This are the options used here. There are other options that can be reviewed by reading the manual pages for carp.
Our configuration is very simple. 2 hosts sharing 1 IP address. When the master goes down, the backup takes over the IP, when master comes back, it must be master again. I don’t want load balance here and just want to keep my services in case of failures.
The MASTER /etc/rc.conf must use a network configuration like this:
# Master net conf
defaultrouter="10.1.1.1"
hostname="master.localdomain"
cloned_interfaces="carp0"
ifconfig_em0="inet 192.168.200.1 netmask 255.255.255.0"
ifconfig_em1="inet 10.1.1.2 netmask 255.255.255.0"
ifconfig_carp0="inet 10.1.1.10 netmask 255.255.255.0 vhid 1 pass mypassword advskew 0"
# End master net conf
The BACKUP /etc/rc.conf must be like this:
# BACKUP net conf
defaultrouter="10.1.1.1"
hostname="backup.localdomain"
cloned_interfaces="carp0"
ifconfig_em0="inet 192.168.200.2 netmask 255.255.255.0"
ifconfig_em1="inet 10.1.1.3 netmask 255.255.255.0"
ifconfig_carp0="inet 10.1.1.10 netmask 255.255.255.0 vhid 1 pass mypassword advskew 100"
# End BACKUP net conf
On both hosts you must add the following line to /etc/sysctl.conf:
net.inet.carp.preempt=1
Notice: The 192.168.200 interfaces are connected directly to provide a faster route between the hosts so I can use it to sync files, share resources without compromising the real network. This interface is not used in the carp configuration that will work only using the other interfaces.
That’s it. Just reboot and all configuration will be up. To test it, ping the 10.1.1.10 address from a remote machine and shutdown the MASTER. You will see that the ping will not stop.
If you run an ifconfig, it will be possible to see the interfaces running:
MASTER ifconfig:
carp0: flags=41
inet 10.1.1.10 netmask 0xffff0000
carp: MASTER vhid 1 advbase 1 advskew 0
BACKUP ifconfig:
carp0: flags=41
inet 10.1.1.10 netmask 0xffff0000
carp: BACKUP vhid 1 advbase 1 advskew 100
Conclusion
Now you can configure your services to listen on carp0 interface. An HTTP, FTP or proxy server can be listening on this interface and when master goes down the backup will be up and running. You will only need to set up some synchronization for your files, to do this you can use rsync over ssh in a cron job or even share a remote storage to not waste space. The best solution is the one that fits your needs, so be creative and use this to increase your availability.
CARP is a very simple solution to a very common problem. The configuration is very easy to build and understand and it can be used to build different kinds of HA services.
I hope this article helps you. Please leave your comments.
4 Comments »
The URI to TrackBack this entry is: http://tmartins.blogsome.com/2006/07/28/high-availability-with-freebsd-and-carp/trackback/
1.
Hi! Have you try use carp with multicast (for ex. mrouted)?
Comment by Andrey — June 7, 2008 @ 10:59 am
2.
I have never used it with mrouted. Sice CARP is a multicast protocol that may be an easy task.
Try the OpenBSD documentation. There lots of info in there (http://www.openbsd.org/faq/faq6.html#CARP).
On FreeBSD I remember to see some posts in a mail list talking about problems with CARP after mrouted being configured. Is that your problem?
Comment by tmartins — June 7, 2008 @ 9:49 pm
3.
I think this line:
echo “options carp” >> GENERIC-CARP
Needs to be:
echo “device carp” >> GENERIC-CARP
Good article!
Comment by Justin — July 9, 2009 @ 3:28 pm
4.
Thanks Justin. You are right for newer FreeBSD versions. This article is old and I wrote it for FreeBSD 5. It was “options carp” back on 2006. :)
Now is “device carp”. The article was changed to reflect this.
Comment by Thiago — July 9, 2009 @ 3:50 pm
Realtime File System Replication On FreeBSD
Realtime File System Replication On FreeBSD
This article describes a concept on how to implement realtime file system replication on a dual-node FreeBSD cluster to provide real HA services.
Maybe you are familiar with DRBD (distributed replicated block device) from the Linux world already, which basically does something we could call network-RAID1.
Since DRBD does not run on FreeBSD one might be tempted to believe that realtime file system replication would not be possible at all. This is not true however. FreeBSD provides you with two valuable geom classes which will allow you to implement a very similar setup: ggate and gmirror.
Requirements
The absolute minimum requirements for this setup are as follows:
- two hardware nodes running FreeBSD
- ethernet connection between both nodes
- a free (as in “unused”) disk slice on each node
All right, this is just good enough to get it going.
If you are serious in useing it you may want to stick to something better than that:
- use FreeBSD 6.x whenever possible, 5.x has some serious locking issues
- Don’t use the same ethernet connection for public access AND replication, use a dedicated interface instead, preferrably over Gigabit ethernet. We’re talking about data replication over a LAN here, so latency and network load is a concern after all.
- Fore the same reasons as above you should not do any geographic separation, especially not over slow links or VPN. Stay within the same network segment.
- Use identical hardware for both nodes.
- Use identical disk partition and slice setup on both nodes.
- Use fast disks and fast disk controllers with good IO performance.
- Refrein from useing geom/ataraid or other software RAID on partitions/slices mirrored to the second node. Use a real hardware RAID controller instead. If you don’t, deadlocks may occur.
- Keep the partitions to be mirrored as small as possible. The reason for this is the fact that a complete resync is required if the mirror brakes. While a 20 GB partition might synchronize within ~30 minutes across a 100 Mbit network, a 500 GB partition will take over 11 hours.
- You should propably not export more than one disk slice to a remote node. Every request (especially with lots and lots of write transactions) will be sent over your network. This causes load and latency on both nodes.
Pros
- Build a two-node HA cluster useing FreeBSD
- Implement realtime file system replication for mission critical failover scenarios
- Use commodity hardware, no need for special shared storage like SAN or iSCSI
- Do not rely on snapshot-based synchronisation (like rsync for example)
- Do not rely on NFS or other file servers which could impose a single point of failure on their own
Cons
- Yet experimental, not tested under heavy-load, possibly unstable
- No support, if it brakes you’re on your own
- Implementation not as mature as DRBD
- Yet, a lot of hand work involved
#1 General System Setup
I have already pointed out some recommendations about the system setup previously. So if you stick with these you may save yourself from trouble.
When you install FreeBSD make sure you take a current 6.x series release. The 5.x series might work too though happened to be a bit flacky at my site due to locking issues. YMMV.
There are no special considerations except for the partition layout: reserve a partition which shall contain the data to be replicated to the remote-host. Don’t make it to big as the whole thing has to be synchronized over the network.
Choose the size according to your actual disk space requirement, the network speed and latency and also the IO performance of your system. A 500 GB partition may be too big, even when running over Gigabit ethernet. A size anywhere from 100 megs to 20 gigs may be ok though.
Since you would hopefully have two identical nodes, make the partition tables/disk slices match each other. This will help greatly to reduce any issues because of different device names.
You should also refrain from useing any geom/ataraid software RAID on the disks/slices to be exported. Remember that you will do a software RAID1 over the network already. Placeing another software RAID onto the underlying device will lead to deadlocks in most cases. Also your system will have twice the load as the data has to be written out four times actually.
If you really want the additional safety of local disk RAID do yourself a favor and use a real hardware RAID controller instead. This will even help you in getting good IO performance. Of course fask disks are a must then.
My setup consisted of two machines with Intel P-III 800 MHz CPU, 1 GB RAM, two 100 mbit network interfaces (one public, one private) and a RAID1 array with 20 GB SCSI disks (I used an ICP Vortex controllers).
This is what my disk slices look like:
/dev/da0s1a / 8 GB
/dev/da0s1b swap 2 GB
/dev/da0s1d [unused] 10 GB
#2 Enable Kernel Modules
Now make sure both nodes support the GEOM mirroring module. Enable it by adding the following line to your /boot/loader.conf:
geom_mirror_load=”YES”
Do the same for the GEOM gate module:
geom_gate_load=”YES”
If your secure level allows to load kernel modules at runtime you may omit these steps.
Check it like this:
#sysctl kern.securelevel
Any return value other than 0 or -1 denote that kernel modules may not be loaded at runtime. In this case a reboot is required to load the modules. But check out step #3 first.
#3 Configure Network Interfaces
Make sure your network interfaces are configured properly.
Since I have two of them I would use one as public interface and the other as private.
The latter one will be useing private IP addresses according to RFC1918 and is connected to the remote host useing a crossover cable.
On both hosts fxp0 is the public interface (which later on use the address 172.16.100.1 for the master node and 172.16.100.2 for the failover node).
On the master node the additional public IP address 172.16.100.12 is bound as an alias and used to provide public services. It will be monitored by freevrrpd and conditionally move over to the failover node.
fxp1 is the private interface used for data replication (192.168.100.1 for the master node and 192.168.100.2 for the failover node).
Restart networking or reboot the machine (if required by step #2), whatever applies to you.
#4 Install Failover Software
On FreeBSD freevrrpd may be used for IP takeovers and optional script execution. Install it from the ports (/usr/ports/net/freevrrpd/) or as a binary package (pkg_add -r freevrrpd).
The configuration of the failover setup is fairly easy and well documented in the freevrrpd man page. An example might look like this:
#
# config for usual master server
#
[VRID]
serverid = 1
interface = fxp0
priority = 255 # denotes priority = master
addr = 172.16.100.12/32 # denotes failover IP
password = anyoneulike
masterscript = /usr/local/bin/become_master
backupscript = /usr/local/bin/become_standby
And this would be an example for a standby node:
#
# config for usual standby server
#
[VRID]
serverid = 1
interface = fxp0
priority = 240 # denotes priority = failover
addr = 172.16.100.12/32 # denotes failover IP
password = anyoneulike
masterscript = /usr/local/bin/become_master
backupscript = /usr/local/bin/become_standby
Now I’d stronly recommend to read the man page and change the config file according to your needs. You will also need to write the master and backup scripts which do the actions required for the failover to work properly.
I leave this up to you as this is beyond the scope of this howto.
#5 Export Disk Slices
Now export the slices which shall be used for replication (/dev/da0s1d in my case). You do this by creating a file called /etc/gg.exports on the master server:
192.168.0.2 RW /dev/da0s1d
And the same on the standby server:
192.168.0.1 RW /dev/da0s1d
You’ll find more on this in the ggated man page. Basically you’re just exporting the underlying device to the given IP address in read/write mode.
Now since ggated does not support any password protection or encryption at all it is best to use a dedicated network for this anyway. This will also lower the load you place on the public network segment.
For optimum performance Gigabit ethernet is recommended.
When you’re set with the config files, ggated must be started on the failover node (yes: the failover node, not on the master!). You do this by running:
#ggated -v
This will place ggated in verbose mode and run in foreground, which is useful for debugging purposes. Later on, when everything works fine, this can be omitted.
Please note that you should not export the partion on both nodes at the same time. Run ggated only on the host which is the current failover node. Use the freevrrpd master/backup scripts to start/stop the service as required.
#6 Import Disk Slices
Looking at the primary node, the remote disk slices must no be imported.
This is done through ggatec, the client component of ggated. Run it as follows:
#ggatec create 192.168.100.2 /dev/da0s1d
This command will return the device node name. If it is the first one created usally ‘ggate0′.
Consider that you should run ggatec only on the designated primary node. Use the freevrrpd master/backup script facilities to create/delete the ggate device node according to it’s state.
Do not create the device node on the failover node as long as it is not in primary state. Do not delete the device node as long as the host is in master state (except for recovery purpose, but this will be covered later).
#7 Setup Replication
Now it’s actually time to bring up replication. This is where gmirror kernel module enabled previously comes in handy.
Make sure you’re on the primary node, then initialize a new GEOM mirror:
#gmirror label -v -n -b prefer gm0 /dev/ggate0
Then insert the local disk slice:
#gmirror insert -p 100 gm0 /dev/ad0s1e
Rebuild the mirror:
#gmirror rebuild rm0 ggate0
If you want to use the geom mirror auto synchronisation features, you can enable these as follows:
#gmirror configure -a gm0
This will cause the disk slices to be synchronized, actually the data from the local ad0s1e will be copied over to the ggate0 remote device.
This will surely take some time, depending on the size of your partition and the speed of your network. When finished, a message like this will appear in the dmesg log of your primary node:
GEOM_MIRROR: Device gm0: rebuilding provider ggate0 finished.
GEOM_MIRROR: Device gm0: provider ggate0 activated.
You may have noticed the “prefer” balance algorithm. This setting actually means that read requests shall only be directed to the geom provider with the highest priority.
By adding the /dev/ad0s1e (which is always the local disk) with a priority of 100 (actually any priority highter then the one of ggate0 according to “gmirror list gm0″ output is fine) you force all read requests to be directed to this device only.
You could actually use the “round-robin” balance algorithm as well, however this requires fast network connection with low latency, otherwise your read performance will drop significantly.
You may now “newfs” your gm0 device, mount and use it as you would with any other data partition.
In the first place you should now test the setup. Monitor the system performance on both hosts by using “vmstat” or a similar tool. Keep an eye on network interface and IO statistics.
If you experience lags, timeouts or slowisch behaviour during usual actions like copying files and directories then the above will certainly help you. In most cases it’s related to network bandwidth or limits in disk IO.
#8 Failing-Over To The Standby Node
Now that your replication is up and running it’s time to test a failover scenario. We do it by hand so you can see what you actually need to put in freevrrpd master/backup scripts for this purpose.
So go and unplug your current master node (yes, really do it. If you don’t do it now you’ll never do it and it is likely to never work properly).
So you unplugged it? Fine, that’s what we want.
Now connect to your failover node and stop the ggated service.
This should cause geom mirror to pick up the gm0 device with provider /dev/da0s1e automatically.
GEOM_MIRROR: Device gm0 created (id=2381431211).
GEOM_MIRROR: Device gm0: provider ad0s1e detected.
GEOM_MIRROR: Force device gm0 start due to timeout.
GEOM_MIRROR: Device gm0: provider ad0s1e activated.
GEOM_MIRROR: Device gm0: provider mirror/gm0 launched.
It may take a few moments for the device to become ready.
Now you must run fsck to ensure filesystem integrity (you really must do this as the filesystem will always be dirty):
#fsck -t ufs /dev/mirror/gm0
Then you can mount the device:
#mount /dev/mirror/gm0 /mnt
Step #9 will explain how the mirror may be rebuilt if the previous master node becomes available again.
#9 Recovering
To bring back the master host into the active combound you will need to make sure that the gm0 device is actually shut down on the failed host.
You remember that we enabled permanent loading of the geom mirror module previously?
This is required to circumvent some problematic situations when kernel secure level is in effect. But it also means that geom mirror will automatically pick up the gm0 device. This will however prevent you from exporting the underlying device through geom gate so the gm0 must disabled first. You can do it like this:
#gmirror stop gm0
As soon as it is stopped you may then run ggated to export the partiton (we’re doing it in debug mode):
#ggated -v
If you get an error stating failure to open the /dev/da0s1e device it may still be locked by the geom mirror class. Just look at “gmirror list” output and stop the device as required.
If ggated is running after all, turn to your failover host and turn off auto configuration on the geom mirror:
#gmirror configure -n gm0
Then make the ggate device available to your node:
#ggatec create 192.168.100.2 /dev/da0s1e
Reinsert the ggate device to the geom mirror using a low priority of ‘0′
#gmirror insert -p 0 gm0 /dev/ggate
and re-enable auto-configuration on the mirror
#gmirror cinfigure -a gm0
I’d recommend to always rebuild the mirror unless your absolutely sure that no new data has been added to the gm0 device in the meantime.
#gmirror rebuild gm0 ggate0
Make sure you give the ggate0 device as last argument which makes it the “sync target”. If you happen to do “gmirror rebuild gm0 da0s1e” accidentally this will sync the other way round leaving you most likely with corrupt or lost data.
The rebuild will take some time depending on the partitition size and network speed. After finishing you will see a message like this in your kernel log:
GEOM_MIRROR: Device gm0: rebuilding provider ggate0 finished.
GEOM_MIRROR: Device gm0: provider ggate0 activated.
Now you will have to remove the local /dev/ad0s1d device from the mirror and reinsert it using a high priority:
#gmirror remove gm0 /dev/ad0s1d
#gmirror insert -p 100 gm0 /dev/ad0s1d
The geom mirror will automatically rebuild the provider if required.
This is actually required to fix the read priority I previously talked about, although only required if you want the previous failover node to become your new master node.
If you do not intend switching designated roles and make your failed primary the active node again, have a look at the next sections.
#10 What If The Failover Node Fails?
Imagine you need to reboot your Failover Node, let’s say to install some updates. Or even more worse: It has rebooted due to some kernel panic, power loss or other real-life situations.
In any case you should put the geom mirror on the master host into degraded mode by forcibly removing the ggate0 device:
When you’re on the master, just make sure the ggate0 is disconnected from the mirror:
#ggatec destroy -f -u 0
This will result in this kernel message:
GEOM_GATE: Device ggate0 destroyed.
GEOM_MIRROR: Device gm0: provider ggate0 disconnected.
The gm0 is now running in degraded state until you re-insert your fail-over node to the configuration.
There is no problem in doing it this way anyway as you have to do a full resync in either case afer the failover node is up again.
The reason to remove the ggate0 device is to prevent IO locking on the geom mirror device.
#11 How To Recover Replication
To bring back the fail-over host into the active combound you will need to make sure that the gm0 device is actually shut down on the failed host.
#gmirror stop gm0
As soon as it is stopped you may then run ggated to export the partiton (we’re doing it in debug mode):
#ggated -v
If you get an error stating failure to open the /dev/da0s1e device it may still be locked by the geom mirror class. Just look at “gmirror list” output and stop the device as required.
If ggated is running after all, make the remote disk slice available on the other host:
#ggatec create 192.168.100.2 /dev/da0s1e
This will have the ggate0 device created and added automatically to your gm0 device.
GEOM_MIRROR: Device gm0: provider ggate0 detected.
GEOM_MIRROR: Device gm0: provider ggate0 activated.
I’d recommend to always rebuild the mirror unless your absolutely sure that no new data has been added to the gm0 device in the meantime.
#gmirror rebuild gm0 ggate0
Make sure you give the ggate0 device as last argument which makes it the “sync target”. If you happen to do “gmirror rebuild gm0 da0s1e” accidentally this will sync the other way round leaving you most likely with corrupt or lost data.
The rebuild will take some time depending on the partitition size and network speed. After finishing you will see a message like this in your kernel log:
GEOM_MIRROR: Device gm0: rebuilding provider ggate0 finished.
GEOM_MIRROR: Device gm0: provider ggate0 activated.
#12 Data Integrity Considerations
Some special considerations must be taken to ensure data integrity:
- ggated cannot export a slice if it is in use by geom mirror
- don’t try any fancy primary-primary replication stuff, it is not possible
- never (as in never) mount the filesystem (the underlying partition to be exact), on the failover node
- to access the data mount the geom mirror device, hence it’s only possible on the master node. Don’t ever do it on the failover node unless you have taken proper recovery action as described above
- always run fsck on the geom mirror after failover
- it’s better not to mount the geom mirror through fstab automatically. Use some freevrrpd recovery magic instead
- Always take backups. This solution is to allow realtime replication for HA services. It is no substitute for proper backups at any time.
#13 Security Considerations
As you may have noticed ggated doesn’t support any security or encryption mechanism by default. “Security” is only implemented upon IP based access restrictions combined with read/write or read/only flags.
To enhance security a bit you should always use a dedicated network interface for data replication, preferrably a private one which is not connected to the internet. Crossover host-to-host cabling is fine.
If you need to go over the (insecure) public network please use additonal firewall rules to block port access to authorized hosts.
Both ggated and ggatec also allow useing a port different from their default so it would be possible to setup a redirect through stunnel. This may however pose another performance impact onto your hosts, especially if your network connection is laggy or slow.
#14 Observations
It may look a bit complicated at a first glance, but it is basically nothing else than spanning a software RAID1 accross networked hosts.
In theory its possible to apply any RAID configuration supported by geom accross networked hosts, but there is no practical reason in doing so.
The possibilites offered by this setup are huge if implemented properly. You can easily apply HA conditions to services which do not support such on their own.
If you happen to implement a live environment upon this technology some time, just let me know how it worked out.
7 Responses to “Realtime File System Replication On FreeBSD”
Subscribe to:
Posts (Atom)
January 30th, 2007 at 5:01 pm
I have this in production for our CVS server which is also serving Mantics and MediaWiki. Everything works great until someone checks in a 40MB file. That seems to clog the dedicated gigabit ethernet connection and cause some sort of deadlock. At that point I must break the mirror (gmirror deactivate data ggate0) and all works just fine. Then I have to rebuild, which takes about 3-4 hours for the 300G partition that is being replicated.
I have increased the ggate buffers to 8192 (ggatec create -q 8192 10.10.10.x /dev/ar1), but that doesn’t fix the problem.
Maybe I should turn off soft updates ?
June 26th, 2007 at 2:39 pm
scsi cable…
I found it very useful. Thanks for the knowledge. I am personally trying to follow the advice & try to be independent….
August 9th, 2007 at 2:17 am
I have this in production, too.
Everything works fine, but sometimes geom detects a timeout and it deactivates ggate provider. Then, i have to rebuild (which takes aboute 3 hours for 500G partition).
September 20th, 2007 at 8:54 am
Good Job
January 8th, 2008 at 12:10 am
very interesting.
i’m adding in RSS Reader
December 16th, 2008 at 11:12 am
Good stuff
But instead of using freevrrpd, you may use CARP since it is integrated by FreeBSD 6+ !
Anyway I like this
February 3rd, 2009 at 1:39 pm
I have tried this on my servers, however the replication performance is very poor. It takes more than 4 hours to rebuild 160GB of partition over gigabit connection and sometimes I had timeout if someone is trying to write big files.
After I tweak my sysctl into
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=262144
kern.ipc.maxsockbuf=1048576
now it takes only half an hour to rebuild 160GB partition and no more timeout. Further information can be found on http://www.geekfarm.org/wu/muse/GeomGate.html