To search all files under /dir for "blah", run the following:
# grep -r blah /dir
You can not use wildards directly in this manner, for example, the following will not search all text files:
# grep -r blah *.txt
This doesn't work because the wildcard is expanded by the shell before grep is called. Instead, search the current directory (or whichever one you want) and pass the --include option:
# grep -r blah . --include "*.txt"
Reference:
http://mindspill.net/computing/linux-notes/recursive-grep-and-filename-wildcards/
Tuesday, January 31, 2012
Debugging load library path issues
Debugging load library path issues
If you get a problem with a library that can't find another, you should check the library's dependencies and make sure that the directory in which the missing library resides is part of the load library path.
One example of this problem, that I experienced recently, occurred when the Magick.so library was unable to open the libMagick.so.10 library. The error message was as follows:
Can't load '/usr/lib/perl5/site_perl/5.8.7/i686-linux/auto/Image/Magick/Magick.so' for module Image::Magick: libMagick.so.10: cannot open shared object file: No such file or directory at /usr/lib/perl5/5.8.7/i686-linux/DynaLoader.pm line 230. at (eval 113) line 1 Compilation failed in require at (eval 113) line 1. BEGIN failed--compilation aborted at (eval 113) line 1.
Find out library dependencies with ldd
The ldd command will check the dependencies of a library.
ldd /the/library/path/libWhatever.so
Using the above example problem:
username@localhost [~]# ldd /usr/lib/perl5/site_perl/5.8.7/i686-linux/auto/Image/Magick/Magick.so
libMagick.so.10 => not found
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x0099a000)
libz.so.1 => /usr/lib/libz.so.1 (0x00111000)
libtiff.so.3 => /usr/lib/libtiff.so.3 (0x00121000)
libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x00d54000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x001f4000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x0016e000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x00910000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x0039d000)
libXt.so.6 => /usr/X11R6/lib/libXt.so.6 (0x005bc000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x004cc000)
libm.so.6 => /lib/tls/libm.so.6 (0x00216000)
libc.so.6 => /lib/tls/libc.so.6 (0x00bda000)
libdl.so.2 => /lib/libdl.so.2 (0x00177000)
/lib/ld-linux.so.2 (0x002c2000)
You can see in that the libMagick.so.10 library was not found.
Locate the library with find or locate or slocate or by any other means
Find the location of the library that couldn't be found by ldd. The chances are that the library will be in a subdirectory of /usr. Continuing with the above example...
username@localhost [~]# find /usr -name "libMagick.so.10"
/usr/local/lib/libMagick.so.10
Add the library to the load library path with ldconfig
In the words of the man page, ldconfig will configure dynamic linker run time bindings. Add your missing library to the bindings with the following command:
ldconfig /path/to/add
For our libMagick example, this would be:
username@localhost [~]# ldconfig /usr/local/lib
You can see a list of librarys that are being used with the -p flag.
username@localhost [~]# ldconfig -p
907 libs found in cache `/etc/ld.so.cache'
libzvt.so.2 (libc6) => /usr/lib/libzvt.so.2
libzvt.so (libc6) => /usr/lib/libzvt.so
libzip.so (libc6, hwcap: 0x0001000000000000) => /opt/blackdown-jdk-1.4.2.03/jre/lib/i386/libzip.so
libz.so.1 (libc6) => /lib/libz.so.1
libz.so (libc6) => /lib/libz.so
libxslt.so.1 (libc6) => /usr/lib/libxslt.so.1
libxslt.so (libc6) => /usr/lib/libxslt.so
[...cut...]
If you want to make make the changes more permanent, so that you don't have to remember to specify the directory next time you run the command, you can add it to the /etc/ld.so.conf file, though different linux distributions manage this file in different ways.
Reference:
http://mindspill.net/computing/linux-notes/debugging-load-library-path-issues/
If you get a problem with a library that can't find another, you should check the library's dependencies and make sure that the directory in which the missing library resides is part of the load library path.
One example of this problem, that I experienced recently, occurred when the Magick.so library was unable to open the libMagick.so.10 library. The error message was as follows:
Can't load '/usr/lib/perl5/site_perl/5.8.7/i686-linux/auto/Image/Magick/Magick.so' for module Image::Magick: libMagick.so.10: cannot open shared object file: No such file or directory at /usr/lib/perl5/5.8.7/i686-linux/DynaLoader.pm line 230. at (eval 113) line 1 Compilation failed in require at (eval 113) line 1. BEGIN failed--compilation aborted at (eval 113) line 1.
Find out library dependencies with ldd
The ldd command will check the dependencies of a library.
ldd /the/library/path/libWhatever.so
Using the above example problem:
username@localhost [~]# ldd /usr/lib/perl5/site_perl/5.8.7/i686-linux/auto/Image/Magick/Magick.so
libMagick.so.10 => not found
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x0099a000)
libz.so.1 => /usr/lib/libz.so.1 (0x00111000)
libtiff.so.3 => /usr/lib/libtiff.so.3 (0x00121000)
libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x00d54000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x001f4000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x0016e000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x00910000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x0039d000)
libXt.so.6 => /usr/X11R6/lib/libXt.so.6 (0x005bc000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x004cc000)
libm.so.6 => /lib/tls/libm.so.6 (0x00216000)
libc.so.6 => /lib/tls/libc.so.6 (0x00bda000)
libdl.so.2 => /lib/libdl.so.2 (0x00177000)
/lib/ld-linux.so.2 (0x002c2000)
You can see in that the libMagick.so.10 library was not found.
Locate the library with find or locate or slocate or by any other means
Find the location of the library that couldn't be found by ldd. The chances are that the library will be in a subdirectory of /usr. Continuing with the above example...
username@localhost [~]# find /usr -name "libMagick.so.10"
/usr/local/lib/libMagick.so.10
Add the library to the load library path with ldconfig
In the words of the man page, ldconfig will configure dynamic linker run time bindings. Add your missing library to the bindings with the following command:
ldconfig /path/to/add
For our libMagick example, this would be:
username@localhost [~]# ldconfig /usr/local/lib
You can see a list of librarys that are being used with the -p flag.
username@localhost [~]# ldconfig -p
907 libs found in cache `/etc/ld.so.cache'
libzvt.so.2 (libc6) => /usr/lib/libzvt.so.2
libzvt.so (libc6) => /usr/lib/libzvt.so
libzip.so (libc6, hwcap: 0x0001000000000000) => /opt/blackdown-jdk-1.4.2.03/jre/lib/i386/libzip.so
libz.so.1 (libc6) => /lib/libz.so.1
libz.so (libc6) => /lib/libz.so
libxslt.so.1 (libc6) => /usr/lib/libxslt.so.1
libxslt.so (libc6) => /usr/lib/libxslt.so
[...cut...]
If you want to make make the changes more permanent, so that you don't have to remember to specify the directory next time you run the command, you can add it to the /etc/ld.so.conf file, though different linux distributions manage this file in different ways.
Reference:
http://mindspill.net/computing/linux-notes/debugging-load-library-path-issues/
Monday, January 30, 2012
FreeBSD - Backup and restore FreeBSD using Fixit CD
Since FreeSBIE have not been updated since year 2007, its kernel recognizing new hardware starts to worry me as it is based on FreeBSD 6.2 and it is going to reach EoL (end of life) by end of this year, 2010. Is time to experiment on new way restoring backup.
The FreeBSD installation process does mentioned about "Fixit" CD booting. After meddling around with it, it's actually referring to another bootable CD which its label name consits of "livefs" (Live File System). It's also known as "disk 2" prior to FreeBSD 7.
e.g.
Installation disc -- > 8.0-RELEASE-i386-disc1.iso
Live File systems disc --> 8.0-RELEASE-i386-livefs.iso
Do take note that the dump/restore instruction from the previous post "Freebsd – Backup & restore for disaster recovery" is still valid. This post is served as a "update" as it won't be using FreeSBIE, rather, it will using the livefs to start the restoration of the partition(s).
Without further ado, here's the instruction :
A quick note, after experimenting with a few backup compress method, below are the summary of my findings :
dump and compress using bzip2 (not so good as bzip is the slowest)
dump -0auLf - /dev/ad0s1a | bzip2 > root_partition.bzip2
dump and compress using gzip (good choice as gzip perform faster than bzip2 and the compression is fairly good)
dump -0auLf - /dev/ad0s1a | gzip > root_partition.gzip
plain dump without compression (fast if target disk can write fast enough)
dump -0auLf root_partition.dump /dev/ad0s1a
My preferred compress method will be using gzip as it's compression ratio is good with fair compressing time taken. In this case, the backup of the partition(s) is done with the command :
dump -0auLf - /dev/ad0s1a | gzip > root_partition.gzip
This is the updated version of how to use Live File System CD to start the restoration process :
Partition & Label the disks
Boot the Installation disc (disc 1)
At the "sysinstall Main Menu", go to "Configure --> Fdisk" and perform the following :
Partition the disk as desire.
Next, type "w" to write changes to disk.
BEWARE:this step will wipe all the contents of the hard disk.
Choose "Label" to label the partitions created in the previous step. Perform the following (similar to previous steps) :
Label the partition as desire.
After labeling the partition, note down the partition label. e.g.
ad0s1a --> "/" or root partition
ad0s2b --> swap partition
Move the up/down keys to highlight the "Disk:" and NOT the partition ("ad0s1a").
Next, type "w" to write changes to disk.
Start the Fixit CD
At the main menu, go to "Fixit --> CDROM/DVD"
At the message "Please insert a FreeBSD live filesystem CD/DVD and press return" message, change the installation disc in the drive to Livefs disc (aka disc2).
Then press enter to continue.
Restore the backup
Use the command "mount" to confirm the partitions are mounted accordingly. "/" or root partition should be mounted as /mnt
Create the external mount point & temporary directory. In this scenario, the backup file is stored in a FAT32 formatted USB external hard disk. The temporary directory is for later use with gzip command.
mkdir /tmp/usb /mnt/writable_tmp
Mount the external USB hard disk :
mount_msdosfs /dev/da0s1 /tmp/usb
*** using the usual command to mount USB hard disk "mount -t msdosfs /dev/da0s1 /tmp/usb" will get an error. See below caveats.
The temporary directory environment variable originally points to a read only directory. This result in "restore" command complaining about having not enough disk space to restore.
Re-point it to a writable disk, use the below command :
export TMPDIR=/tmp/writable_tmp/
Start the restoration process :
cd /mnt
gzcat /tmp/usb/root_partition.gzip | restore -rvf -
Repeat the previous step for the rest of the partitions until all partitions are restored.
CAVEATS :
When executing the command :
mount -t msdosfs /dev/da0s1 /tmp/usb
An error return :
mount: exec mount_msdosfs not found in /sbin:/usr/sbin: No such file or directory
For some reason, mount_msdosfs is in /mnt2/sbin/ but "mount" cannot find it.
Solution 1 :
Replace the "mount" command with equivalent "mount_msdosfs". Execute it without the need to specify parameter "-t msdosfs"
e.g.
mount_msdosfs /dev/da0s1 /tmp/usb
Solution 2
Soft link the command "mount_msdosfs" to the same directory where mount resides :
cd /sbin;ln -s /mnt2/sbin/mount_msdosfs mount_msdosfs
then mount it again with the usual command.
Nemaste !!!
Reference:
http://scratching.psybermonkey.net/2010/01/freebsd-backup-and-restore-freebsd.html
The FreeBSD installation process does mentioned about "Fixit" CD booting. After meddling around with it, it's actually referring to another bootable CD which its label name consits of "livefs" (Live File System). It's also known as "disk 2" prior to FreeBSD 7.
e.g.
Installation disc -- > 8.0-RELEASE-i386-disc1.iso
Live File systems disc --> 8.0-RELEASE-i386-livefs.iso
Do take note that the dump/restore instruction from the previous post "Freebsd – Backup & restore for disaster recovery" is still valid. This post is served as a "update" as it won't be using FreeSBIE, rather, it will using the livefs to start the restoration of the partition(s).
Without further ado, here's the instruction :
A quick note, after experimenting with a few backup compress method, below are the summary of my findings :
dump and compress using bzip2 (not so good as bzip is the slowest)
dump -0auLf - /dev/ad0s1a | bzip2 > root_partition.bzip2
dump and compress using gzip (good choice as gzip perform faster than bzip2 and the compression is fairly good)
dump -0auLf - /dev/ad0s1a | gzip > root_partition.gzip
plain dump without compression (fast if target disk can write fast enough)
dump -0auLf root_partition.dump /dev/ad0s1a
My preferred compress method will be using gzip as it's compression ratio is good with fair compressing time taken. In this case, the backup of the partition(s) is done with the command :
dump -0auLf - /dev/ad0s1a | gzip > root_partition.gzip
This is the updated version of how to use Live File System CD to start the restoration process :
Partition & Label the disks
Boot the Installation disc (disc 1)
At the "sysinstall Main Menu", go to "Configure --> Fdisk" and perform the following :
Partition the disk as desire.
Next, type "w" to write changes to disk.
BEWARE:this step will wipe all the contents of the hard disk.
Choose "Label" to label the partitions created in the previous step. Perform the following (similar to previous steps) :
Label the partition as desire.
After labeling the partition, note down the partition label. e.g.
ad0s1a --> "/" or root partition
ad0s2b --> swap partition
Move the up/down keys to highlight the "Disk:" and NOT the partition ("ad0s1a").
Next, type "w" to write changes to disk.
Start the Fixit CD
At the main menu, go to "Fixit --> CDROM/DVD"
At the message "Please insert a FreeBSD live filesystem CD/DVD and press return" message, change the installation disc in the drive to Livefs disc (aka disc2).
Then press enter to continue.
Restore the backup
Use the command "mount" to confirm the partitions are mounted accordingly. "/" or root partition should be mounted as /mnt
Create the external mount point & temporary directory. In this scenario, the backup file is stored in a FAT32 formatted USB external hard disk. The temporary directory is for later use with gzip command.
mkdir /tmp/usb /mnt/writable_tmp
Mount the external USB hard disk :
mount_msdosfs /dev/da0s1 /tmp/usb
*** using the usual command to mount USB hard disk "mount -t msdosfs /dev/da0s1 /tmp/usb" will get an error. See below caveats.
The temporary directory environment variable originally points to a read only directory. This result in "restore" command complaining about having not enough disk space to restore.
Re-point it to a writable disk, use the below command :
export TMPDIR=/tmp/writable_tmp/
Start the restoration process :
cd /mnt
gzcat /tmp/usb/root_partition.gzip | restore -rvf -
Repeat the previous step for the rest of the partitions until all partitions are restored.
CAVEATS :
When executing the command :
mount -t msdosfs /dev/da0s1 /tmp/usb
An error return :
mount: exec mount_msdosfs not found in /sbin:/usr/sbin: No such file or directory
For some reason, mount_msdosfs is in /mnt2/sbin/ but "mount" cannot find it.
Solution 1 :
Replace the "mount" command with equivalent "mount_msdosfs". Execute it without the need to specify parameter "-t msdosfs"
e.g.
mount_msdosfs /dev/da0s1 /tmp/usb
Solution 2
Soft link the command "mount_msdosfs" to the same directory where mount resides :
cd /sbin;ln -s /mnt2/sbin/mount_msdosfs mount_msdosfs
then mount it again with the usual command.
Nemaste !!!
Reference:
http://scratching.psybermonkey.net/2010/01/freebsd-backup-and-restore-freebsd.html
Setup a Pure-FTPd FTP server with virtual users
Setup a Pure-FTPd FTP server with virtual users
Pure-FTPd is a free (BSD), secure, production-quality and standard-conformant FTP server.
This guide provides instructions for using the virtual user system to manage and control users. By using virtual users, FTP accounts can be administrated without affecting system accounts.
Let's initiate Pure-FTPd's installation by entering the following commands:
% su
# portsnap update
# cd /usr/ports/ftp/pure-ftpd
# make config-recursive
A menu containing Pure-FTPd options will pop-up. In my case, I've opted to leave these options at their defaults.
# cat /var/db/ports/pure-ftpd/options
WITH_UTF8=true
WITH_LARGEFILE=true
# make install clean distclean
# rehash
Having finished the installation process we now move into the configuration stage. We'll start by copying the sample configuration file and set the configuration options:
# cd /usr/local/etc
# cp pure-ftpd.conf.sample pure-ftpd.conf
# chmod 444 pure-ftpd.conf
The chmod command was run to be able to edit the file (default permissions are set to -r--r--r--).
# vim pure-ftpd.conf
# Port range for passive connections replies. - for firewalling.
PassivePortRange 30000 30900
# IP address/port to listen to (default=all IP and port 21).
Bind 192.168.100.1,21
# If you want to log all client commands, set this to "yes".
# This directive can be duplicated to also log server responses.
VerboseLog yes
# PureDB user database (see README.Virtual-Users)
PureDB /usr/local/etc/pureftpd.pdb
# Create an additional log file with transfers logged in the standard W3C
# format (compatible with most commercial log analyzers)
AltLog w3c:/var/log/pureftpd.log
# Automatically create home directories if they are missing
CreateHomeDir yes
# Disallow anonymous connections. Only allow authenticated users.
NoAnonymous yes
# Disallow anonymous users to upload new files (no = upload is allowed)
AnonymousCantUpload yes
The CreateHomeDir option makes adding virtual users more easy by creating a user's home directory upon login (if it doesn't already exist).
We can either import users with system-level accounts (defined in /etc/master.passwd) at once or create new users manually. To import users that already exist on your system into the virtual user database, enter these commands:
# pure-pwconvert >> /usr/local/etc/pureftpd.passwd
# chmod 600 /usr/local/etc/pureftpd.passwd
# pure-pw mkdb
It should be noted that pure-pwconvert only imports accounts that have shell access. Accounts with the shell set to nologin have to be added manually.
To add users to the Pure-FTPd virtual user database manually, we need to create a system-level account that will be associated with virtual users. Create a new user named ftp like this:
# pw useradd -n ftp -s /sbin/nologin -w no -d /home/ftp -c "FTP user" -m
Note: When you install pureftp, an ftp group is created, but no ftp user; this results in the error "mail pure-ftpd:(?:?) [ERROR] Unable to find the 'ftp' account". So we need to manually create the ftp user.
Having done this we can now add users to the virtual users database using the commands below:
# pure-pw useradd ftp_test -u ftp -g ftp -d /home/ftp/ftp_test
# pure-pw mkdb
Replace user with the desired username. With -d flag, the user will be chrooted. If you want to give user access to the whole filesystem, use -D instead of -d.
If you want to add additional users, just repeat the commands above with a different user.
Allow clients' IP addresse:
# pure-pw usermod user_name -r 209.17.11.12
# pure-pw mkdb
To reset a user's password:
# pure-pw passwd user_name
# pure-pw mkdb
To view info about one user:
# pure-pw show user_name
To view a list of users:
# pure-pw list
To remove a user:
# pure-pw userdel user_name
# pure-pw mkdb
Now to start Pure-FTPd:
# /usr/local/etc/rc.d/pure-ftpd onestart
Initiate a FTP connection to test the server:
% ftp localhost
To configure Pure-FTPd to start at boot time:
# echo 'pureftpd_enable="YES"' >> /etc/rc.conf
To restart Pure-FTPd and determine if it is running:
# /usr/local/etc/rc.d/pure-ftpd restart
# /usr/local/etc/rc.d/pure-ftpd status
# sockstat | grep :21
Rotate pure-ftpd log file:
# vim /etc/newsyslog.conf
/var/log/pureftpd.log 600 7 100000 * JC /var/run/pure-ftpd.pid
# /etc/rc.d/newsyslog restart
Pure-FTPd provides useful features for personal users as well as hosting providers. I've only touched the tip of the iceberg so do take a look at the project's website for the excellent documentation that is available.
PF supports FTP servers and FTP clients behind NAT
WAN0 IP: 11.11.11.11
WAN1 IP: 22.22.22.22
FTP0 server (behind NAT) IP: 192.168.100.1
FTP1 server (behind NAT) IP: 192.168.100.2
# vim /etc/rc.local
### ftp-proxy for outside world FTP clients accessing internal FTP server behand NAT.
/usr/sbin/setfib 0 /usr/sbin/ftp-proxy -R 192.168.100.1 -p 21 -b 11.11.11.11
/usr/sbin/setfib 1 /usr/sbin/ftp-proxy -R 192.168.100.2 -p 21 -b 22.22.22.22
### Note:
### -b // bind to the external IP.
### -R // redirect to internal FTP server IP.
### ftp-proxy for internal FTP clients behand NAT accessing outside world FTP servers.
/usr/sbin/ftp-proxy -p 8021 -b 127.0.0.1
# vim /etc/pf.conf
### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from $dmz_if/24 to any port 21 -> 127.0.0.1 port 8021
### enabloe log for debugging purpose.
block log all
### [FTP] from ROUTER to ANY
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port 21 rtable 0
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port > 1023 rtable 0
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port 21 rtable 1
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port > 1023 rtable 1
### [FTP] from ANY to $dmz_ftp0_ip0 $dmz_ftp1_ip0
pass in quick log on $wan_if0 proto tcp from any to $wan_if0 port 21 rtable 0
pass in quick log on $wan_if0 proto tcp from any to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass in quick log on $wan_if1 proto tcp from any to $wan_if1 port 21 rtable 1
pass in quick log on $wan_if1 proto tcp from any to $dmz_ftp1_ip0 port 30000:30900 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 21 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 21 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 30000:30900 rtable 1
### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"
### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto tcp from $dmz_if/24 to any port 21 rtable 0
Flush the pf.conf setting:
# pfctl -f /etc/pf.conf
Try to connect to ftp:
# ftp ftp.freebsd.org
Name: anonymous
Password: anonymous
Reference:
Setup a Pure-FTPd FTP server with virtual users
http://gala4th.blogspot.com/2012/01/setup-pure-ftpd-ftp-server-with-virtual.html
http://gala4th.blogspot.com/2012/01/pf-supports-ftp-servers-and-ftp-clients.html
http://forums.freebsd.org/showthread.php?t=591
Pure-FTPd is a free (BSD), secure, production-quality and standard-conformant FTP server.
This guide provides instructions for using the virtual user system to manage and control users. By using virtual users, FTP accounts can be administrated without affecting system accounts.
Let's initiate Pure-FTPd's installation by entering the following commands:
% su
# portsnap update
# cd /usr/ports/ftp/pure-ftpd
# make config-recursive
A menu containing Pure-FTPd options will pop-up. In my case, I've opted to leave these options at their defaults.
# cat /var/db/ports/pure-ftpd/options
WITH_UTF8=true
WITH_LARGEFILE=true
# make install clean distclean
# rehash
Having finished the installation process we now move into the configuration stage. We'll start by copying the sample configuration file and set the configuration options:
# cd /usr/local/etc
# cp pure-ftpd.conf.sample pure-ftpd.conf
# chmod 444 pure-ftpd.conf
The chmod command was run to be able to edit the file (default permissions are set to -r--r--r--).
# vim pure-ftpd.conf
# Port range for passive connections replies. - for firewalling.
PassivePortRange 30000 30900
# IP address/port to listen to (default=all IP and port 21).
Bind 192.168.100.1,21
# If you want to log all client commands, set this to "yes".
# This directive can be duplicated to also log server responses.
VerboseLog yes
# PureDB user database (see README.Virtual-Users)
PureDB /usr/local/etc/pureftpd.pdb
# Create an additional log file with transfers logged in the standard W3C
# format (compatible with most commercial log analyzers)
AltLog w3c:/var/log/pureftpd.log
# Automatically create home directories if they are missing
CreateHomeDir yes
# Disallow anonymous connections. Only allow authenticated users.
NoAnonymous yes
# Disallow anonymous users to upload new files (no = upload is allowed)
AnonymousCantUpload yes
The CreateHomeDir option makes adding virtual users more easy by creating a user's home directory upon login (if it doesn't already exist).
We can either import users with system-level accounts (defined in /etc/master.passwd) at once or create new users manually. To import users that already exist on your system into the virtual user database, enter these commands:
# pure-pwconvert >> /usr/local/etc/pureftpd.passwd
# chmod 600 /usr/local/etc/pureftpd.passwd
# pure-pw mkdb
It should be noted that pure-pwconvert only imports accounts that have shell access. Accounts with the shell set to nologin have to be added manually.
To add users to the Pure-FTPd virtual user database manually, we need to create a system-level account that will be associated with virtual users. Create a new user named ftp like this:
# pw useradd -n ftp -s /sbin/nologin -w no -d /home/ftp -c "FTP user" -m
Note: When you install pureftp, an ftp group is created, but no ftp user; this results in the error "mail pure-ftpd:(?:?) [ERROR] Unable to find the 'ftp' account". So we need to manually create the ftp user.
Having done this we can now add users to the virtual users database using the commands below:
# pure-pw useradd ftp_test -u ftp -g ftp -d /home/ftp/ftp_test
# pure-pw mkdb
Replace user with the desired username. With -d flag, the user will be chrooted. If you want to give user access to the whole filesystem, use -D instead of -d.
If you want to add additional users, just repeat the commands above with a different user.
Allow clients' IP addresse:
# pure-pw usermod user_name -r 209.17.11.12
# pure-pw mkdb
To reset a user's password:
# pure-pw passwd user_name
# pure-pw mkdb
To view info about one user:
# pure-pw show user_name
To view a list of users:
# pure-pw list
To remove a user:
# pure-pw userdel user_name
# pure-pw mkdb
Now to start Pure-FTPd:
# /usr/local/etc/rc.d/pure-ftpd onestart
Initiate a FTP connection to test the server:
% ftp localhost
Trying 127.0.0.1... Connected to localhost. 220---------- Welcome to Pure-FTPd [TLS] ---------- 220-You are user number 2 of 50 allowed. 220-Local time is now 13:39. Server port: 21. 220-IPv6 connections are also welcome on this server. 220 You will be disconnected after 15 minutes of inactivity. Name (localhost:username):Now log in with a user account created as explained above. Commands such as ls, cp, pwd and less work just like in tcsh and bash shells. To quit the FTP session type exit.
To configure Pure-FTPd to start at boot time:
# echo 'pureftpd_enable="YES"' >> /etc/rc.conf
To restart Pure-FTPd and determine if it is running:
# /usr/local/etc/rc.d/pure-ftpd restart
# /usr/local/etc/rc.d/pure-ftpd status
# sockstat | grep :21
Rotate pure-ftpd log file:
# vim /etc/newsyslog.conf
/var/log/pureftpd.log 600 7 100000 * JC /var/run/pure-ftpd.pid
# /etc/rc.d/newsyslog restart
Pure-FTPd provides useful features for personal users as well as hosting providers. I've only touched the tip of the iceberg so do take a look at the project's website for the excellent documentation that is available.
PF supports FTP servers and FTP clients behind NAT
WAN0 IP: 11.11.11.11
WAN1 IP: 22.22.22.22
FTP0 server (behind NAT) IP: 192.168.100.1
FTP1 server (behind NAT) IP: 192.168.100.2
# vim /etc/rc.local
### ftp-proxy for outside world FTP clients accessing internal FTP server behand NAT.
/usr/sbin/setfib 0 /usr/sbin/ftp-proxy -R 192.168.100.1 -p 21 -b 11.11.11.11
/usr/sbin/setfib 1 /usr/sbin/ftp-proxy -R 192.168.100.2 -p 21 -b 22.22.22.22
### Note:
### -b // bind to the external IP.
### -R // redirect to internal FTP server IP.
### ftp-proxy for internal FTP clients behand NAT accessing outside world FTP servers.
/usr/sbin/ftp-proxy -p 8021 -b 127.0.0.1
# vim /etc/pf.conf
### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from $dmz_if/24 to any port 21 -> 127.0.0.1 port 8021
### enabloe log for debugging purpose.
block log all
### [FTP] from ROUTER to ANY
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port 21 rtable 0
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port > 1023 rtable 0
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port 21 rtable 1
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port > 1023 rtable 1
### [FTP] from ANY to $dmz_ftp0_ip0 $dmz_ftp1_ip0
pass in quick log on $wan_if0 proto tcp from any to $wan_if0 port 21 rtable 0
pass in quick log on $wan_if0 proto tcp from any to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass in quick log on $wan_if1 proto tcp from any to $wan_if1 port 21 rtable 1
pass in quick log on $wan_if1 proto tcp from any to $dmz_ftp1_ip0 port 30000:30900 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 21 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 21 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 30000:30900 rtable 1
### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"
### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto tcp from $dmz_if/24 to any port 21 rtable 0
Flush the pf.conf setting:
# pfctl -f /etc/pf.conf
Try to connect to ftp:
# ftp ftp.freebsd.org
Name: anonymous
Password: anonymous
Reference:
Setup a Pure-FTPd FTP server with virtual users
http://gala4th.blogspot.com/2012/01/setup-pure-ftpd-ftp-server-with-virtual.html
http://gala4th.blogspot.com/2012/01/pf-supports-ftp-servers-and-ftp-clients.html
http://forums.freebsd.org/showthread.php?t=591
Saturday, January 28, 2012
Resolutions for 2012
[] C
[] gcc, gdb, automake, autoconf
[] LLVM, clang, LLDB, OpenCL, KLEE
[] Git
[] Python
[] PostgreSQL
[] Dtrace
[] ZFS
[] RAID
[] English
[] NFS
[] gcc, gdb, automake, autoconf
[] LLVM, clang, LLDB, OpenCL, KLEE
[] Git
[] Python
[] PostgreSQL
[] Dtrace
[] ZFS
[] RAID
[] English
[] NFS
Thursday, January 26, 2012
remove old svn revisons history
Say you have 1000 revisions and you want to shrink to only have the revisions from r950-r1000. You can do the following:
svnadmin dump /path/to/current/repo -r950:1000 > small_svn.dump
svnadmin create /path/to/new/repo
svnadmin load /path/to/new/repo < small_svn.dump Reference:
http://stackoverflow.com/questions/4350145/how-to-remove-old-svn-revisons
svnadmin dump /path/to/current/repo -r950:1000 > small_svn.dump
svnadmin create /path/to/new/repo
svnadmin load /path/to/new/repo < small_svn.dump Reference:
http://stackoverflow.com/questions/4350145/how-to-remove-old-svn-revisons
Wednesday, January 25, 2012
disable PHP script execution in upload attachment directory
disable PHP script execution in upload attachment directory
// Apache
// Apache
<Directory /website/attachments> php_flag engine off </Directory>// Nginx
location /sites/default/files/ { location ~ .*\.(php)?$ { deny all; } }
Tuesday, January 24, 2012
Multiple default routes gateways with two different ISP ipfw PF setfib load balancing
Multiple default routes gateways with two different ISP ipfw PF setfib load balancing
Thanks to phoenix help I was able to setup multiple default routes, or a default route per network/interface to be precise, in Debian/Linux it is as simple as that:
# cat /etc/network/interfaces
That would be example topology (but more than 2 interfaces is also possible).
Now, You can not use the 'casual' defaultrouter="X" cause it will be only for one network.
We will have to use setfib(1) to create two (or more) separete routing tables per network/interface.
Note: FIB (Forward Information Base, synonym for a routing table here)
Add these lines to /boot/loader.conf file:
# vim /boot/loader.conf
### select one of following firewalls to enable.
#ipfw_load="YES" # Firewall
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log
### set number of routing tables to support.
net.fibs=2
Note: to view a list of options: cat /boot/defaults/loader.conf.
It will unfortunately require kernel recompile, but its not as that hard:
# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2
# vim MYKERNEL8.2
options ROUTETABLES=2 # max 16. 1 is back compatible.
Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
# cat /usr/src/sys/`uname -m`/conf/DEFAULTS
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2
# sync ; reboot
Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html
Remove the files generated during recompiling the kernel.
# rm -rf /usr/obj/usr/src/sys/MYKERNEL8.2
After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.
After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:
# setfib 0 /sbin/route add -net default 10.0.0.1
# setfib 1 /sbin/route add -net default 20.0.0.1
Note: with packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output.
# vi /etc/rc.conf
### WAN interfaces. You do not need to set defaultrouter="XXX" here.
ifconfig_wan0="inet 10.0.0.2/32"
ifconfig_wan1="inet 20.0.0.2/32"
### Note: commented out for multiple WAN routes setup. Routes will be added in /etc/rc.local file.
#defaultrouter="10.0.0.1"
### LAN static route
#static_routes="lan"
#route_lan="-net 192.168.50.0/24 192.168.0.11"
### PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""
### Enable as LAN gateway
gateway_enable="YES"
All the rest configuration resides in /etc/rc.local file:
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 10.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 20.0.0.1
### assing route tables to interfaces
### Note: uncomment following lines if you are using ipfw. Leave it if you are using PF.
###
#ipfw -f flush
#ipfw add allow ip from any to any via lo0
#ipfw add setfib 0 ip from any to any via wan0
#ipfw add setfib 1 ip from any to any via wan1
#ipfw add allow ip from any to any
These would be handy for restarting:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# /etc/rc.d/local restart
# pfctl -f /etc/pf.conf
View the fib:
# sysctl -a | grep fib
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 2
View the default route of each routing table:
# setfib 0 netstat -rn
# setfib 1 netstat -rn
Try to ping:
# setfib 0 ping google.com
# setfib 1 ping google.com
===============================================================
pf.conf
Looking for something like this?
# man 1 setfib
# man 2 setfib
And pf.conf(5) has the route-to and reply-to directives, of course.
# man 5 pf.conf | less +/rtable
# man 5 pf.conf | less +/route-to
# man 5 pf.conf | less +/reply-to
# vim /etc/pf.conf
### [VARIABLES]
wan_if0 = "wan0"
wan_if1 = "wan1"
dmz_if0 = "dmz0"
www0_ip0 = "192.168.0.3"
www0_ip1 = "192.168.0.4"
##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"
### [NAT]
nat on $wan_if0 from $dmz_if0/24 to any -> $wan_if0
nat on $wan_if1 from $dmz_if0/24 to any -> $wan_if1
### [HTTP] RDR Outside to DMZ
rdr on $wan_if0 proto tcp from any to $wan_if0 port 80 -> $www0_ip0
rdr on $wan_if1 proto tcp from any to $wan_if1 port 80 -> $www0_ip1
### [HTTP] outside to router
pass in quick on $wan_if0 proto {tcp} from any to $wan_if0 port 80 rtable 0
pass in quick on $wan_if1 proto {tcp} from any to $wan_if1 port 80 rtable 1
### [HTTP] router to dmz
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip0 port 80 rtable 0
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip1 port 80 rtable 1
Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53
You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp
Or just to/from a certain host:
# tcpdump -ni re0 icmp host 172.16.70.12
Reading a PF (packet filter) log file
The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.
To view the log file:
# tcpdump -n -e -ttt -r /var/log/pf.log
Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:
# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0
Filtering Log Output
Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:
# tcpdump -n -e -ttt -r /var/log/pf.log port 80
set up PF (packet filter) firewall on FreeBSD 8.1
Other Solution:
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html
http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
===============================================================
Just a quick note, if doing this on a router with a single internal interface.
Traffic originating on the local network will go out the default route of FIB 0.
Only traffic coming in on the second public interface will go out the same interface.
IOW, the setup above is really only useful for incoming traffic, to make sure that it goes back out the correct interface.
However, a few more IPFW rules can be added to classify traffic on the internal NIC.
It all depends on what you want to accomplish.
Yes, its mainly for that purpose, to assign proper gateways to networks/interfaces (which is a lot more easier @ Debian in /etc/network/interfaces)
We may even use FIB# 1 and 2 for these networks, and use FIB# 0 with other NIC as a default interface, posibilities are of course big, but I wanted to point posibility of having a gateway per network.
http://www.daemonforums.org/showthread.php?t=4610
===============================================================
setfib selects the routing table for locally originated
outgoing packets. Besides locally originated packets, there
are packets arriving from the network and need to be forwarded.
These packets can be classified in a specific routing table
with the aid of ipfw. That's all there is. I can't think
of something else that needs to be thought with regard to
multiple routing tables.
HTH, Nikos
http://lists.freebsd.org/pipermail/freebsd-questions/2009-July/202462.html
===============================================================
Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I've never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.
I have finally, at long last, found the solution.
The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn't route it's responses to the interface that they were received on, it uses the routing table to determine where to send it's responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It's really that simple. There is no logic for having multiple paths to non-local networks.
Frequently Offered Solution: Since I use pf for my firewall, I'm frequently told to use pf's route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.
The Real Solution: FreeBSD has support for multiple routing tables. It's little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support ("options ROUTETABLES=2"), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.
So here are the steps to solving this problem:
Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.
# cat /usr/src/sys/`uname -m`/conf/MYKERNEL8.2 | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
Step 2: Disable the SSH daemon in your rc.conf:
# cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.local
Step 3: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1
# vim /etc/ssh/sshd_config.rtable0
ListenAddress 20.0.0.2
ListenAddress 192.168.0.1
# vim /etc/ssh/sshd_config.rtable1
ListenAddress 30.0.0.2
ListenAddress 192.168.0.2
Note: Multiple ListenAddress options are permitted.
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1
### Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1
Step 4: Add rtable awareness to your pf.conf file:
[root@router]~ # cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on wan0 inet proto tcp from wan0:network to (wan0) port 22 rtable 0
Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.
In my above config it's worth pointing out that it doesn't actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.
High Availability HA cluster solution for FreeBSD
HAST + CARP + ZFS
HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.
CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.
# man carp
# man ng_one2many
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
===============================================================
Reference:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html
High Availability HA cluster solution for freebsd
http://www.daemonforums.org/showthread.php?t=4610
http://freebsd-forums.liquidneon.com/showthread.php?t=20166
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
http://www.melen.org/u/jan/wp/?p=102
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
Running PF & IPFW Together
Prevent default gateway route and resolv.conf being overwritten by DHCP
Thanks to phoenix help I was able to setup multiple default routes, or a default route per network/interface to be precise, in Debian/Linux it is as simple as that:
# cat /etc/network/interfaces
iface eth0 inet static address 10.0.0.2 netmask 255.255.255.252 gateway 10.0.0.1 iface eth1 inet static address 20.0.0.2 netmask 255.255.255.252 gateway 20.0.0.1
That would be example topology (but more than 2 interfaces is also possible).
ISP NETWORK 0 ISP NETWORK 1 \ / \ / \ / ROUTER 0 ROUTER 1 10.0.0.1/30 20.0.0.1/30 \ / +------\----------------/------+ | \ / | | wan0 wan1 | | 10.0.0.2 20.0.0.2 | | | | FREEBSD (Firewall Router) | | | | dmz0 | | 192.168.0.1/24 | | 192.168.0.2/24 | +------------------------------+ | +--------+ | switch + +--------+ | / ______/ / +--------/-----+ | / | | lan0 | | 192.168.0.3 | | 192.168.0.4 | | | | Web Server 0 | +--------------+
Now, You can not use the 'casual' defaultrouter="X" cause it will be only for one network.
We will have to use setfib(1) to create two (or more) separete routing tables per network/interface.
Note: FIB (Forward Information Base, synonym for a routing table here)
Add these lines to /boot/loader.conf file:
# vim /boot/loader.conf
### select one of following firewalls to enable.
#ipfw_load="YES" # Firewall
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log
### set number of routing tables to support.
net.fibs=2
Note: to view a list of options: cat /boot/defaults/loader.conf.
It will unfortunately require kernel recompile, but its not as that hard:
# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2
# vim MYKERNEL8.2
options ROUTETABLES=2 # max 16. 1 is back compatible.
Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
# cat /usr/src/sys/`uname -m`/conf/DEFAULTS
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2
# sync ; reboot
Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html
Remove the files generated during recompiling the kernel.
# rm -rf /usr/obj/usr/src/sys/MYKERNEL8.2
After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.
After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:
# setfib 0 /sbin/route add -net default 10.0.0.1
# setfib 1 /sbin/route add -net default 20.0.0.1
Note: with packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output.
# vi /etc/rc.conf
### WAN interfaces. You do not need to set defaultrouter="XXX" here.
ifconfig_wan0="inet 10.0.0.2/32"
ifconfig_wan1="inet 20.0.0.2/32"
### Note: commented out for multiple WAN routes setup. Routes will be added in /etc/rc.local file.
#defaultrouter="10.0.0.1"
### LAN static route
#static_routes="lan"
#route_lan="-net 192.168.50.0/24 192.168.0.11"
### PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""
### Enable as LAN gateway
gateway_enable="YES"
All the rest configuration resides in /etc/rc.local file:
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 10.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 20.0.0.1
### assing route tables to interfaces
### Note: uncomment following lines if you are using ipfw. Leave it if you are using PF.
###
#ipfw -f flush
#ipfw add allow ip from any to any via lo0
#ipfw add setfib 0 ip from any to any via wan0
#ipfw add setfib 1 ip from any to any via wan1
#ipfw add allow ip from any to any
These would be handy for restarting:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# /etc/rc.d/local restart
# pfctl -f /etc/pf.conf
View the fib:
# sysctl -a | grep fib
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 2
View the default route of each routing table:
# setfib 0 netstat -rn
# setfib 1 netstat -rn
Try to ping:
# setfib 0 ping google.com
# setfib 1 ping google.com
===============================================================
pf.conf
Looking for something like this?
# man 1 setfib
# man 2 setfib
And pf.conf(5) has the route-to and reply-to directives, of course.
# man 5 pf.conf | less +/rtable
# man 5 pf.conf | less +/route-to
# man 5 pf.conf | less +/reply-to
# vim /etc/pf.conf
### [VARIABLES]
wan_if0 = "wan0"
wan_if1 = "wan1"
dmz_if0 = "dmz0"
www0_ip0 = "192.168.0.3"
www0_ip1 = "192.168.0.4"
##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"
### [NAT]
nat on $wan_if0 from $dmz_if0/24 to any -> $wan_if0
nat on $wan_if1 from $dmz_if0/24 to any -> $wan_if1
### [HTTP] RDR Outside to DMZ
rdr on $wan_if0 proto tcp from any to $wan_if0 port 80 -> $www0_ip0
rdr on $wan_if1 proto tcp from any to $wan_if1 port 80 -> $www0_ip1
### [HTTP] outside to router
pass in quick on $wan_if0 proto {tcp} from any to $wan_if0 port 80 rtable 0
pass in quick on $wan_if1 proto {tcp} from any to $wan_if1 port 80 rtable 1
### [HTTP] router to dmz
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip0 port 80 rtable 0
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip1 port 80 rtable 1
Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53
You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp
Or just to/from a certain host:
# tcpdump -ni re0 icmp host 172.16.70.12
Reading a PF (packet filter) log file
The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.
To view the log file:
# tcpdump -n -e -ttt -r /var/log/pf.log
Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:
# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0
Filtering Log Output
Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:
# tcpdump -n -e -ttt -r /var/log/pf.log port 80
set up PF (packet filter) firewall on FreeBSD 8.1
Other Solution:
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html
http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
===============================================================
Just a quick note, if doing this on a router with a single internal interface.
Traffic originating on the local network will go out the default route of FIB 0.
Only traffic coming in on the second public interface will go out the same interface.
IOW, the setup above is really only useful for incoming traffic, to make sure that it goes back out the correct interface.
However, a few more IPFW rules can be added to classify traffic on the internal NIC.
It all depends on what you want to accomplish.
Yes, its mainly for that purpose, to assign proper gateways to networks/interfaces (which is a lot more easier @ Debian in /etc/network/interfaces)
We may even use FIB# 1 and 2 for these networks, and use FIB# 0 with other NIC as a default interface, posibilities are of course big, but I wanted to point posibility of having a gateway per network.
http://www.daemonforums.org/showthread.php?t=4610
===============================================================
setfib selects the routing table for locally originated
outgoing packets. Besides locally originated packets, there
are packets arriving from the network and need to be forwarded.
These packets can be classified in a specific routing table
with the aid of ipfw. That's all there is. I can't think
of something else that needs to be thought with regard to
multiple routing tables.
HTH, Nikos
http://lists.freebsd.org/pipermail/freebsd-questions/2009-July/202462.html
===============================================================
Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I've never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.
I have finally, at long last, found the solution.
The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn't route it's responses to the interface that they were received on, it uses the routing table to determine where to send it's responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It's really that simple. There is no logic for having multiple paths to non-local networks.
Frequently Offered Solution: Since I use pf for my firewall, I'm frequently told to use pf's route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.
The Real Solution: FreeBSD has support for multiple routing tables. It's little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support ("options ROUTETABLES=2"), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.
So here are the steps to solving this problem:
Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.
# cat /usr/src/sys/`uname -m`/conf/MYKERNEL8.2 | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.
Step 2: Disable the SSH daemon in your rc.conf:
# cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.local
Step 3: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1
# vim /etc/ssh/sshd_config.rtable0
ListenAddress 20.0.0.2
ListenAddress 192.168.0.1
# vim /etc/ssh/sshd_config.rtable1
ListenAddress 30.0.0.2
ListenAddress 192.168.0.2
Note: Multiple ListenAddress options are permitted.
# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1
### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1
### Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1
Step 4: Add rtable awareness to your pf.conf file:
[root@router]~ # cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on wan0 inet proto tcp from wan0:network to (wan0) port 22 rtable 0
Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.
In my above config it's worth pointing out that it doesn't actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.
High Availability HA cluster solution for FreeBSD
HAST + CARP + ZFS
HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.
CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.
# man carp
# man ng_one2many
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.
http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
===============================================================
Reference:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html
High Availability HA cluster solution for freebsd
http://www.daemonforums.org/showthread.php?t=4610
http://freebsd-forums.liquidneon.com/showthread.php?t=20166
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
http://www.melen.org/u/jan/wp/?p=102
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
Running PF & IPFW Together
Prevent default gateway route and resolv.conf being overwritten by DHCP
kill sendmail processes and clean up mail queue mailq
kill sendmail processes and clean up mail queue mailq
# ps auxww | grep 'sendmail: ./' | awk '{print $2}' | xargs kill
# rm /var/spool/mqueue/*
# ps auxww | grep 'sendmail: ./' | awk '{print $2}' | xargs kill
# rm /var/spool/mqueue/*
Monday, January 23, 2012
Check if a file is opened in use or locked by another process
The PHP flock() function is good when the file locking method is used the same way by all programs that will lock the file.
However, you can't check if any process in the system is actually using the file or not. If you for example are monitoring an incoming ftp folder, you don't want to start processing a file until the file is completely tranferred (i.e. wait until the ftp daemon is no longer having the file open).
The following code illustrates how the command lsof (Linux) or fstat (FreeBSD) can be used for the purpose:
lsof on Linux:
fstat on FreeBSD:
test.sh:
Reference:
http://nerdia.net/2011/01/17/check-if-a-file-is-open-by-another-process-in-php-linuxunix/
However, you can't check if any process in the system is actually using the file or not. If you for example are monitoring an incoming ftp folder, you don't want to start processing a file until the file is completely tranferred (i.e. wait until the ftp daemon is no longer having the file open).
The following code illustrates how the command lsof (Linux) or fstat (FreeBSD) can be used for the purpose:
lsof on Linux:
<?php $filename = "locktest.txt"; $directory = "."; while (1) { $result = exec("lsof +D $directory | grep -c -i $filename"); if ($result == "0") { echo "$directory/$filename is NOT open ($result)\n"; } else { echo "$directory/$filename IS OPEN ($result)\n"; } sleep(1); }; ?>
fstat on FreeBSD:
<?php $out = exec('fstat test.pdf | wc -l'); $out = trim($out); if ($out == 1) { echo 'not in use' . PHP_EOL; } else { echo 'in use' . PHP_EOL; } ?>
test.sh:
#!/bin/sh dir="/home/backup/" find "${dir}" -type f | while read file do isFileInUse=`fstat "${file}" | grep -c "${file}"` #echo "${isFileInUse}" if [ "${isFileInUse}" = "0" ]; then #echo ${isFileInUse} echo "File is Not in use. ${file}" else echo "File is in use. ${file}" fi done
Reference:
http://nerdia.net/2011/01/17/check-if-a-file-is-open-by-another-process-in-php-linuxunix/
Install Samba on FreeBSD 8.2
Install Samba on FreeBSD 8.2
root@bmw0 [/root] # cd /usr/ports/net/samba36; make install clean
Edit smb.conf
root@bmw0 [/root] # vi /usr/local/etc/smb.conf
security = user
hosts allow = 192.168.100. 192.168.200.168 127.
load printers = no
[syncdb]
comment = hmm
path = /usr/local/www/apache22/data/syncdb
valid users = smb_user0
public = no
writable = yes
printable = no
create mask = 0600
directory mask = 0700
; write list = @staff
Add a user called "smb_user0"
root@bmw0 [/root] # pw useradd -n smb_user0 -s /sbin/nologin -w no -d /home/smb_user0 -c "Samaba User0" -m
Adding users to samba
root@bmw0 [/root] # /usr/local/bin/pdbedit -a smb_user0
or
root@bmw0 [/root] # smbpasswd -a smb_user0
List user
root@bmw0 [/root] # /usr/local/bin/pdbedit -L
Edit rc.conf
root@bmw0 [/root] # vi /etc/rc.conf
### Samba
samba_enable="YES"
#nmbd_enable="YES"
#smbd_enable="YES"
winbindd_enable="YES"
Start Samba
root@bmw0 [/root] # /usr/local/etc/rc.d/samba start
root 56804 0.2 0.7 39668 14484 ?? Ss 5:15PM 0:00.04 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root 56807 0.2 0.7 39752 14564 ?? S 5:15PM 0:00.00 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root 56800 0.0 0.4 29696 7464 ?? Ss 5:15PM 0:00.01 /usr/local/sbin/nmbd -D -s /usr/local/etc/smb.conf
Check to see if samba is running
root@bmw0 [/root] # ps -auwxx | egrep '[sn]mbd|winbindd'
Test samba using smbclient
root@bmw0 [/root] # /usr/local/bin/smbclient -U smb_user0 \\\\localhost\\smb_user0
or
root@bmw0 [/root] # /usr/local/bin/smbclient -U smb_user0 //localhost/smb_user0
Edit ipfw.rules
root@bmw0 [/root] # vi /usr/local/etc/ipfw.rules
### samba
$IPF 260 allow tcp from 192.168.100.0/24 to any 139 in
Reference:
http://gala4th.blogspot.com/2012/01/install-samba-on-freebsd-82.html
root@bmw0 [/root] # cd /usr/ports/net/samba36; make install clean
Edit smb.conf
root@bmw0 [/root] # vi /usr/local/etc/smb.conf
security = user
hosts allow = 192.168.100. 192.168.200.168 127.
load printers = no
[syncdb]
comment = hmm
path = /usr/local/www/apache22/data/syncdb
valid users = smb_user0
public = no
writable = yes
printable = no
create mask = 0600
directory mask = 0700
; write list = @staff
Add a user called "smb_user0"
root@bmw0 [/root] # pw useradd -n smb_user0 -s /sbin/nologin -w no -d /home/smb_user0 -c "Samaba User0" -m
Adding users to samba
root@bmw0 [/root] # /usr/local/bin/pdbedit -a smb_user0
or
root@bmw0 [/root] # smbpasswd -a smb_user0
List user
root@bmw0 [/root] # /usr/local/bin/pdbedit -L
Edit rc.conf
root@bmw0 [/root] # vi /etc/rc.conf
### Samba
samba_enable="YES"
#nmbd_enable="YES"
#smbd_enable="YES"
winbindd_enable="YES"
Start Samba
root@bmw0 [/root] # /usr/local/etc/rc.d/samba start
root 56804 0.2 0.7 39668 14484 ?? Ss 5:15PM 0:00.04 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root 56807 0.2 0.7 39752 14564 ?? S 5:15PM 0:00.00 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root 56800 0.0 0.4 29696 7464 ?? Ss 5:15PM 0:00.01 /usr/local/sbin/nmbd -D -s /usr/local/etc/smb.conf
Check to see if samba is running
root@bmw0 [/root] # ps -auwxx | egrep '[sn]mbd|winbindd'
Test samba using smbclient
root@bmw0 [/root] # /usr/local/bin/smbclient -U smb_user0 \\\\localhost\\smb_user0
or
root@bmw0 [/root] # /usr/local/bin/smbclient -U smb_user0 //localhost/smb_user0
Edit ipfw.rules
root@bmw0 [/root] # vi /usr/local/etc/ipfw.rules
### samba
$IPF 260 allow tcp from 192.168.100.0/24 to any 139 in
Reference:
http://gala4th.blogspot.com/2012/01/install-samba-on-freebsd-82.html
Friday, January 20, 2012
set up PF (packet filter) firewall on FreeBSD 8.2
set up PF (packet filter) firewall on FreeBSD 8.2
Network Structure
Outside World <-----> PF on FreeBSD 8.1 as a router <-----> DMZ network (more FreeBSD boxes)
Quick Notes
[] "pfctl -e" or "pfctl -d" can be used to enable and disable PF respectively. Note that this just enables or disables PF, it doesn't actually load a ruleset. The ruleset must be loaded separately, either before or after PF is enabled. To load the pf.conf ruleset file, run "pfctl -f /etc/pf.conf".
[] Filter rules are evaluated in sequential order, first to last. Unless the packet matches a rule containing the quick keyword, the packet will be evaluated against all filter rules before the final action is taken.
[] The last rule to match is the "winner" and will dictate what action to take on the packet.
[] There is an implicit pass all at the beginning of a filtering ruleset meaning that if a packet does not match any filter rule the resulting action will be pass.
[] The keyword any meaning all addresses.
[] The keyword all which is short for from any to any.
Edit /boot/loader.conf:
# vim /boot/loader.conf
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log
Edit /etc/rc.conf
### LAN
ifconfig_xl0="inet 192.168.3.1 netmask 255.255.255.0"
### WAN
ifconfig_re0="inet 219.17.112.243 netmask 255.255.255.192"
### WAN - extra IP addresses
### Note: the netmask of extra IP addresses must be 255.255.255.255
ifconfig_re0_alias0="inet 219.17.112.244 netmask 255.255.255.255"
ifconfig_re0_alias1="inet 219.17.112.245 netmask 255.255.255.255"
### Default Route
defaultrouter="219.17.112.193"
### LAN static route
static_routes="lan"
route_lan="-net 192.168.5.0/24 192.168.100.9"
### enable PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""
### enable as LAN gateway.
### You must enable this if you want to do NAT (network address translation).
gateway_enable="YES"
### FTP-Proxy.
### Note: add these two lines to allow people to connect to your FTP server (192.168.100.8) behind NAT.
ftpproxy_enable="YES"
ftpproxy_flags="-R 192.168.100.8"
Note: Since NAT is almost always used on routers and network gateways, it will probably be necessary to enable IP forwarding so that packets can travel between network interfaces on the OpenBSD machine. IP forwarding is enabled using the sysctl(3) mechanism
On FreeBSD, setting gateway_enable="YES" in /etc/rc.conf will automatically turn on net.inet.ip.forwarding=1. You can check this by running:
# sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1
If you are using OpenBSD, you can run follow command:
# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1 (if using IPv6)
To make this change permanent, the following lines should be added to /etc/sysctl.conf (on OpenBSD. No need on FreeBSD):
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
These lines are present but commented out (prefixed with a #) in the default install. Remove the # and save the file. IP forwarding will be enabled when the machine is rebooted.
Create a file for later use:
# touch /etc/pf-blocked_ip
Edit /etc/pf.conf
root@fw1 [/root] # cat /etc/pf.conf
##########################
##### MACROS - User-defined variables that can hold IP addresses, interface names, etc.
##########################
wan_if="re0"
dmz_if="xl0"
#lan_if=""
icmp_types = "{echoreq echorep unreach}"
#icmp_types = "{echoreq}"
tcp_services = "{ssh smtp domain http https ntp nicname}"
udp_services = "{domain ntp}"
allow_ssh_ip = "{219.17.152.199 224.1.1.1}"
dmz_www_ip = "192.168.100.5"
dmz_sql_ip = "192.168.100.6"
dmz_svn_ip = "192.168.100.7"
dmz_ftp0_ip0 = "192.168.100.8"
cn_srv_ip = "222.144.87.188"
ca_pub_ip = "219.17.152.196"
allow_sql_ip = "{" $cn_srv_ip $ca_pub_ip "}"
##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"
##########################
##### OPTIONS - Various options to control how PF works.
##########################
### Sets the default behavior for filter rules that specify the block action.
### Return error codes for ports that are blocked. Allows faster error recovery.
#set block-policy return
### Set pf's debugging level.
#set debug misc
### Skip all PF processing on specified interface. This can be useful on loopback interfaces where filtering, normalization, queueing, etc, are not required.
set skip on lo0
##########################
##### NORMALIZATION
##########################
### Traffic normalization for incoming packets - scrub provides a measure of protection against certain kinds of attacks based on incorrect handling of packet fragments.
### One reason not to scrub on an interface is if one is passing NFS through PF. Some non-OpenBSD platforms send (and expect) strange packets -- fragmented packets with the "do not fragment" bit set, which are (properly) rejected by scrub. This can be resolved by use of the no-df option. Another reason is some multi-player games have connection problems passing through PF with scrub enabled. Other than these somewhat unusual cases, scrubbing all packets is a highly recommended practice.
### The scrub directive syntax is very similar to the filtering syntax which makes it easy to selectively scrub certain packets and not others. The no keyword can be used in front of scrub to specify packets that will not be scrubbed. Just as with nat rules, the first matching rule wins.
scrub in all
##########################
##### QUEUEING - Provides bandwidth control and packet prioritization.
##########################
##########################
##### TRANSLATION
##########################
### NAT
nat on $wan_if from $dmz_if/24 to any -> $wan_if
### [HTTP] RDR Outside to DMZ
rdr on $wan_if proto tcp from any to $wan_if port 80 -> $dmz_www_ip
### [MYSQL] RDR Outside to DMZ
rdr on $wan_if proto tcp from $allow_sql_ip to $wan_if port 3306 -> $dmz_sql_ip
### [RSYNCD] RDR Outside to DMZ
rdr on $wan_if proto tcp from $cn_srv_ip to $wan_if port 873 -> $dmz_www_ip
### [MAIL] RDR Outside to DMZ
#rdr on $wan_if proto tcp from any to $wan_if/32 port {25 110} -> 192.168.10.2
### [FTP] RDR Outside to DMZ
#rdr on $wan_if proto tcp from $allow_ftp_ip to $wan_if port 21 -> 192.168.10.8 port 21
### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from any to $wan_if port 21 -> 127.0.0.1 port 8021
### [RAID-3WARE]
#rdr on $wan_if proto tcp from $wan_if/26 to $wan_if port 888 -> $dmz_sql_ip
##########################
##### FILTERING RULES - Allows the selective filtering or blocking of packets as they pass through any of the interfaces.
##### Filter rules can be given parameters to specify network address translation (NAT) and packet redirection.
##########################
### setup a default deny policy - block everything (inbound and outbound on all interfaces).
### Note: if you want to see the log file to debug your filter rules, change following line to "block log all", otherwise use "block all" instead..
block log all
block in quick on $wan_if from <blocked_ip> to any
### activate spoofing protection for all interfaces.
block in quick from urpf-failed
### [DNS] Outside to Router
pass in quick on $wan_if proto {tcp udp} from any to $wan_if port 53
### [DNS] DMZ to Router
pass in quick on $dmz_if proto {tcp udp} from $dmz_if/24 to any port 53
### [DNS] Router to Outside
pass out quick on $wan_if proto {tcp udp} from $wan_if to any port 53
### [DNS] Router to DMZ
pass out quick on $dmz_if proto {tcp udp} from $dmz_if to $dmz_if/24 port 53
### [HTTP] Outside to DMZ
pass in quick on $wan_if proto tcp from any to $dmz_www_ip port 80
pass out quick on $dmz_if proto tcp from any to $dmz_www_ip port 80
### [HTTP] DMZ to Router
### Note: FreeBSD's portsnap command uses port 80 to grab latest FreeBSD ports/patches.
pass in quick on $dmz_if proto tcp from $dmz_if/24 to any port 80
### [HTTP] Router to Outside
pass out quick on $wan_if proto tcp from $wan_if to any port 80
### [MYSQL] DMZ to Outside
pass in quick on $dmz_if proto tcp from $dmz_sql_ip to $cn_srv_ip port 3306
pass out quick on $wan_if proto tcp from $wan_if to $cn_srv_ip port 3306
### [MYSQL] Outside to DMZ
pass in quick on $wan_if proto tcp from $allow_sql_ip to $dmz_sql_ip port 3306
pass out quick on $dmz_if proto tcp from $allow_sql_ip to $dmz_sql_ip port 3306
### [SMTP] DMZ to Outside
pass in quick on $dmz_if proto tcp from $dmz_www_ip to any port 25
pass out quick on $wan_if proto tcp from $wan_if to any port 25
### [RSYNCD] Outside to DMZ
pass in quick on $wan_if proto tcp from $cn_srv_ip to $dmz_www_ip port 873
pass out quick on $dmz_if proto tcp from $cn_srv_ip to $dmz_www_ip port 873
### [SSH] Outside to Router
pass in quick on $wan_if proto tcp from $allow_ssh_ip to $wan_if port 22 flags S/SA keep state
### [SSH] Router to DMZ
pass out quick on $dmz_if proto tcp from $dmz_if to $dmz_if/24 port 22 flags S/SA keep state
### [ICMP] DMZ/LAN to ANY
pass in quick on $dmz_if proto icmp from {$dmz_if/24 $lan_net} to any icmp-type $icmp_types
### [ICMP] Router to Outside
pass out quick on $wan_if proto icmp from $wan_if to any icmp-type $icmp_types
### [ICMP] DMZ to DMZ/LAN
pass out quick on $dmz_if proto icmp from $dmz_if/24 to {$dmz_if/24 $lan_net} icmp-type $icmp_types
### [NTP] DMZ to Router
pass in quick on $dmz_if proto {tcp udp} from $dmz_if/24 to any port 123
### [NTP] Router to Outside
pass out quick on $wan_if proto {tcp udp} from $wan_if to any port 123
### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"
### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto {tcp} from $dmz_if/24 to any port 21
pass in quick log on $dmz_if proto {tcp} from $dmz_if/24 to any port > 1023
### [FTP] from ROUTER to ANY
pass out quick log on $wan_if proto {tcp} from $wan_if to any port 21
pass out quick log on $wan_if proto {tcp} from $wan_if to any port > 1023
### [FTP] from ANY to $dmz_ftp0_ip
pass out quick on $dmz_if proto {tcp} from $dmz_if to $dmz_ftp0_ip0 port 21
### [NICNAME] DMZ to Router
pass in quick on $dmz_if proto tcp from $dmz_if/24 to any port 43
### [NICNAME] Router to Outside
pass out quick on $wan_if proto tcp from $wan_if to any port 43
### [SVN] Router to DMZ
pass out quick on $dmz_if proto tcp from $dmz_if to $dmz_svn_ip port 3690
### [RAID-3WARE]
#pass in quick on $wan_if proto tcp from $wan_if/26 to $dmz_sql_ip port 888
#pass out quick on $dmz_if proto tcp from $wan_if/26 to $dmz_sql_ip port 888
### [TRACEROUTE] Allow outgoing Trace route
#pass out on $ext_if inet proto udp from any to any port 33433 >< 33626 keep state
Solution to No ALTQ support in kernel ALTQ related functions disabled
Recompile your kernel:
# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2
# vim MYKERNEL8.2
Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES
# cat /usr/src/sys/`uname -m`/conf/DEFAULTS
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2
# sync ; reboot
Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html
Useful Commands
Make sure the pf kernel module has been loaded:
# kldstat | grep pf
2 2 0xc0f8c000 32e98 pf.ko
3 1 0xc61db000 3000 pflog.ko
Diable the packet filter
# pfctl -d
Enable the packet filter
# pfctl -e
Flush all (nat, filter, queue, state, info, table) rules and reload from the file /etc/pf.conf
# pfctl -F all -f /etc/pf.conf
If you are setting up pf remotely, and you are not confident enough with your PF rules, you might be blocked out by the wrong rules. Following command allows you to try the rules, and disable PF after 20 seconds:
# pfctl -f /etc/pf.conf; sleep 20; pfctl -d
This does not actually load any rules, but allows you to check for syntax errors in the file before you do load the ruleset. This is obviously good for testing.
# pfctl -vnf /etc/pf.conf
Report on the currently loaded filter ruleset.
# pfctl -s rules
Report on the currently loaded nat ruleset.
# pfctl -s nat
Report on the currently running state table (very useful).
# pfctl -s state
Tables can be manipulated on the fly by using pfctl(8). For instance, to add entries to the <blocked_ip> table:
# pfctl -t blocked_ip -T add 218.70.0.0/16
This will also create the <blocked_ip> table if it doesn't already exist.
To list the addresses in a table:
# pfctl -t blocked_ip -T show
Note: The -v argument can also be used with -T show to display statistics for each table entry.
To remove addresses from a table:
# pfctl -t blocked_ip -T delete 218.70.0.0/16
Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53
You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp
Or just to/from a certain host:
# tcpdump -ni re0 icmp and host 172.16.70.12
Reading a PF (packet filter) log file
The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.
To view the log file:
# tcpdump -n -e -ttt -r /var/log/pf.log
Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:
# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0
Filtering Log Output
Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:
# tcpdump -n -e -ttt -r /var/log/pf.log port 80
Reference:
set up PF (packet filter) firewall on FreeBSD 8.2
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
PF supports FTP servers and FTP clients behind NAT
http://www.openbsd.org/faq/pf/index.html
http://www.openbsd.org/faq/pf/nat.html
http://www.weithenn.org/cgi-bin/wiki.pl?PF-%E5%88%A9%E7%94%A8_PF_%E8%BC%95%E9%AC%86%E9%81%94%E6%88%90_NAT
http://www.thedeepsky.com/howto/newbie_pf_guide.php
http://home.nuug.no/~peter/pf/en/ftpproblem.html
Active FTP vs. Passive FTP, a Definitive Explanation
Active FTP vs. Passive FTP, a Definitive Explanation
Network Structure
Outside World <-----> PF on FreeBSD 8.1 as a router <-----> DMZ network (more FreeBSD boxes)
Quick Notes
[] "pfctl -e" or "pfctl -d" can be used to enable and disable PF respectively. Note that this just enables or disables PF, it doesn't actually load a ruleset. The ruleset must be loaded separately, either before or after PF is enabled. To load the pf.conf ruleset file, run "pfctl -f /etc/pf.conf".
[] Filter rules are evaluated in sequential order, first to last. Unless the packet matches a rule containing the quick keyword, the packet will be evaluated against all filter rules before the final action is taken.
[] The last rule to match is the "winner" and will dictate what action to take on the packet.
[] There is an implicit pass all at the beginning of a filtering ruleset meaning that if a packet does not match any filter rule the resulting action will be pass.
[] The keyword any meaning all addresses.
[] The keyword all which is short for from any to any.
Edit /boot/loader.conf:
# vim /boot/loader.conf
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log
Edit /etc/rc.conf
### LAN
ifconfig_xl0="inet 192.168.3.1 netmask 255.255.255.0"
### WAN
ifconfig_re0="inet 219.17.112.243 netmask 255.255.255.192"
### WAN - extra IP addresses
### Note: the netmask of extra IP addresses must be 255.255.255.255
ifconfig_re0_alias0="inet 219.17.112.244 netmask 255.255.255.255"
ifconfig_re0_alias1="inet 219.17.112.245 netmask 255.255.255.255"
### Default Route
defaultrouter="219.17.112.193"
### LAN static route
static_routes="lan"
route_lan="-net 192.168.5.0/24 192.168.100.9"
### enable PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""
### enable as LAN gateway.
### You must enable this if you want to do NAT (network address translation).
gateway_enable="YES"
### FTP-Proxy.
### Note: add these two lines to allow people to connect to your FTP server (192.168.100.8) behind NAT.
ftpproxy_enable="YES"
ftpproxy_flags="-R 192.168.100.8"
Note: Since NAT is almost always used on routers and network gateways, it will probably be necessary to enable IP forwarding so that packets can travel between network interfaces on the OpenBSD machine. IP forwarding is enabled using the sysctl(3) mechanism
On FreeBSD, setting gateway_enable="YES" in /etc/rc.conf will automatically turn on net.inet.ip.forwarding=1. You can check this by running:
# sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1
If you are using OpenBSD, you can run follow command:
# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1 (if using IPv6)
To make this change permanent, the following lines should be added to /etc/sysctl.conf (on OpenBSD. No need on FreeBSD):
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
These lines are present but commented out (prefixed with a #) in the default install. Remove the # and save the file. IP forwarding will be enabled when the machine is rebooted.
Create a file for later use:
# touch /etc/pf-blocked_ip
Edit /etc/pf.conf
root@fw1 [/root] # cat /etc/pf.conf
##########################
##### MACROS - User-defined variables that can hold IP addresses, interface names, etc.
##########################
wan_if="re0"
dmz_if="xl0"
#lan_if=""
icmp_types = "{echoreq echorep unreach}"
#icmp_types = "{echoreq}"
tcp_services = "{ssh smtp domain http https ntp nicname}"
udp_services = "{domain ntp}"
allow_ssh_ip = "{219.17.152.199 224.1.1.1}"
dmz_www_ip = "192.168.100.5"
dmz_sql_ip = "192.168.100.6"
dmz_svn_ip = "192.168.100.7"
dmz_ftp0_ip0 = "192.168.100.8"
cn_srv_ip = "222.144.87.188"
ca_pub_ip = "219.17.152.196"
allow_sql_ip = "{" $cn_srv_ip $ca_pub_ip "}"
##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"
##########################
##### OPTIONS - Various options to control how PF works.
##########################
### Sets the default behavior for filter rules that specify the block action.
### Return error codes for ports that are blocked. Allows faster error recovery.
#set block-policy return
### Set pf's debugging level.
#set debug misc
### Skip all PF processing on specified interface. This can be useful on loopback interfaces where filtering, normalization, queueing, etc, are not required.
set skip on lo0
##########################
##### NORMALIZATION
##########################
### Traffic normalization for incoming packets - scrub provides a measure of protection against certain kinds of attacks based on incorrect handling of packet fragments.
### One reason not to scrub on an interface is if one is passing NFS through PF. Some non-OpenBSD platforms send (and expect) strange packets -- fragmented packets with the "do not fragment" bit set, which are (properly) rejected by scrub. This can be resolved by use of the no-df option. Another reason is some multi-player games have connection problems passing through PF with scrub enabled. Other than these somewhat unusual cases, scrubbing all packets is a highly recommended practice.
### The scrub directive syntax is very similar to the filtering syntax which makes it easy to selectively scrub certain packets and not others. The no keyword can be used in front of scrub to specify packets that will not be scrubbed. Just as with nat rules, the first matching rule wins.
scrub in all
##########################
##### QUEUEING - Provides bandwidth control and packet prioritization.
##########################
##########################
##### TRANSLATION
##########################
### NAT
nat on $wan_if from $dmz_if/24 to any -> $wan_if
### [HTTP] RDR Outside to DMZ
rdr on $wan_if proto tcp from any to $wan_if port 80 -> $dmz_www_ip
### [MYSQL] RDR Outside to DMZ
rdr on $wan_if proto tcp from $allow_sql_ip to $wan_if port 3306 -> $dmz_sql_ip
### [RSYNCD] RDR Outside to DMZ
rdr on $wan_if proto tcp from $cn_srv_ip to $wan_if port 873 -> $dmz_www_ip
### [MAIL] RDR Outside to DMZ
#rdr on $wan_if proto tcp from any to $wan_if/32 port {25 110} -> 192.168.10.2
### [FTP] RDR Outside to DMZ
#rdr on $wan_if proto tcp from $allow_ftp_ip to $wan_if port 21 -> 192.168.10.8 port 21
### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from any to $wan_if port 21 -> 127.0.0.1 port 8021
### [RAID-3WARE]
#rdr on $wan_if proto tcp from $wan_if/26 to $wan_if port 888 -> $dmz_sql_ip
##########################
##### FILTERING RULES - Allows the selective filtering or blocking of packets as they pass through any of the interfaces.
##### Filter rules can be given parameters to specify network address translation (NAT) and packet redirection.
##########################
### setup a default deny policy - block everything (inbound and outbound on all interfaces).
### Note: if you want to see the log file to debug your filter rules, change following line to "block log all", otherwise use "block all" instead..
block log all
block in quick on $wan_if from <blocked_ip> to any
### activate spoofing protection for all interfaces.
block in quick from urpf-failed
### [DNS] Outside to Router
pass in quick on $wan_if proto {tcp udp} from any to $wan_if port 53
### [DNS] DMZ to Router
pass in quick on $dmz_if proto {tcp udp} from $dmz_if/24 to any port 53
### [DNS] Router to Outside
pass out quick on $wan_if proto {tcp udp} from $wan_if to any port 53
### [DNS] Router to DMZ
pass out quick on $dmz_if proto {tcp udp} from $dmz_if to $dmz_if/24 port 53
### [HTTP] Outside to DMZ
pass in quick on $wan_if proto tcp from any to $dmz_www_ip port 80
pass out quick on $dmz_if proto tcp from any to $dmz_www_ip port 80
### [HTTP] DMZ to Router
### Note: FreeBSD's portsnap command uses port 80 to grab latest FreeBSD ports/patches.
pass in quick on $dmz_if proto tcp from $dmz_if/24 to any port 80
### [HTTP] Router to Outside
pass out quick on $wan_if proto tcp from $wan_if to any port 80
### [MYSQL] DMZ to Outside
pass in quick on $dmz_if proto tcp from $dmz_sql_ip to $cn_srv_ip port 3306
pass out quick on $wan_if proto tcp from $wan_if to $cn_srv_ip port 3306
### [MYSQL] Outside to DMZ
pass in quick on $wan_if proto tcp from $allow_sql_ip to $dmz_sql_ip port 3306
pass out quick on $dmz_if proto tcp from $allow_sql_ip to $dmz_sql_ip port 3306
### [SMTP] DMZ to Outside
pass in quick on $dmz_if proto tcp from $dmz_www_ip to any port 25
pass out quick on $wan_if proto tcp from $wan_if to any port 25
### [RSYNCD] Outside to DMZ
pass in quick on $wan_if proto tcp from $cn_srv_ip to $dmz_www_ip port 873
pass out quick on $dmz_if proto tcp from $cn_srv_ip to $dmz_www_ip port 873
### [SSH] Outside to Router
pass in quick on $wan_if proto tcp from $allow_ssh_ip to $wan_if port 22 flags S/SA keep state
### [SSH] Router to DMZ
pass out quick on $dmz_if proto tcp from $dmz_if to $dmz_if/24 port 22 flags S/SA keep state
### [ICMP] DMZ/LAN to ANY
pass in quick on $dmz_if proto icmp from {$dmz_if/24 $lan_net} to any icmp-type $icmp_types
### [ICMP] Router to Outside
pass out quick on $wan_if proto icmp from $wan_if to any icmp-type $icmp_types
### [ICMP] DMZ to DMZ/LAN
pass out quick on $dmz_if proto icmp from $dmz_if/24 to {$dmz_if/24 $lan_net} icmp-type $icmp_types
### [NTP] DMZ to Router
pass in quick on $dmz_if proto {tcp udp} from $dmz_if/24 to any port 123
### [NTP] Router to Outside
pass out quick on $wan_if proto {tcp udp} from $wan_if to any port 123
### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"
### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto {tcp} from $dmz_if/24 to any port 21
pass in quick log on $dmz_if proto {tcp} from $dmz_if/24 to any port > 1023
### [FTP] from ROUTER to ANY
pass out quick log on $wan_if proto {tcp} from $wan_if to any port 21
pass out quick log on $wan_if proto {tcp} from $wan_if to any port > 1023
### [FTP] from ANY to $dmz_ftp0_ip
pass out quick on $dmz_if proto {tcp} from $dmz_if to $dmz_ftp0_ip0 port 21
### [NICNAME] DMZ to Router
pass in quick on $dmz_if proto tcp from $dmz_if/24 to any port 43
### [NICNAME] Router to Outside
pass out quick on $wan_if proto tcp from $wan_if to any port 43
### [SVN] Router to DMZ
pass out quick on $dmz_if proto tcp from $dmz_if to $dmz_svn_ip port 3690
### [RAID-3WARE]
#pass in quick on $wan_if proto tcp from $wan_if/26 to $dmz_sql_ip port 888
#pass out quick on $dmz_if proto tcp from $wan_if/26 to $dmz_sql_ip port 888
### [TRACEROUTE] Allow outgoing Trace route
#pass out on $ext_if inet proto udp from any to any port 33433 >< 33626 keep state
Solution to No ALTQ support in kernel ALTQ related functions disabled
Recompile your kernel:
# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2
# vim MYKERNEL8.2
### PF Packet Filter Related settings device pf device pflog device pfsync options ALTQ options ALTQ_CBQ # Class Bases Queuing (CBQ) options ALTQ_RED # Random Early Detection (RED) options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC) options ALTQ_PRIQ # Priority Queuing (PRIQ) options ALTQ_NOPCC # Required for SMP build
Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES
# cat /usr/src/sys/`uname -m`/conf/DEFAULTS
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2
# sync ; reboot
Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html
Useful Commands
Make sure the pf kernel module has been loaded:
# kldstat | grep pf
2 2 0xc0f8c000 32e98 pf.ko
3 1 0xc61db000 3000 pflog.ko
Diable the packet filter
# pfctl -d
Enable the packet filter
# pfctl -e
Flush all (nat, filter, queue, state, info, table) rules and reload from the file /etc/pf.conf
# pfctl -F all -f /etc/pf.conf
If you are setting up pf remotely, and you are not confident enough with your PF rules, you might be blocked out by the wrong rules. Following command allows you to try the rules, and disable PF after 20 seconds:
# pfctl -f /etc/pf.conf; sleep 20; pfctl -d
This does not actually load any rules, but allows you to check for syntax errors in the file before you do load the ruleset. This is obviously good for testing.
# pfctl -vnf /etc/pf.conf
Report on the currently loaded filter ruleset.
# pfctl -s rules
Report on the currently loaded nat ruleset.
# pfctl -s nat
Report on the currently running state table (very useful).
# pfctl -s state
Tables can be manipulated on the fly by using pfctl(8). For instance, to add entries to the <blocked_ip> table:
# pfctl -t blocked_ip -T add 218.70.0.0/16
This will also create the <blocked_ip> table if it doesn't already exist.
To list the addresses in a table:
# pfctl -t blocked_ip -T show
Note: The -v argument can also be used with -T show to display statistics for each table entry.
To remove addresses from a table:
# pfctl -t blocked_ip -T delete 218.70.0.0/16
Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53
You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp
Or just to/from a certain host:
# tcpdump -ni re0 icmp and host 172.16.70.12
Reading a PF (packet filter) log file
The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.
To view the log file:
# tcpdump -n -e -ttt -r /var/log/pf.log
Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:
# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0
Filtering Log Output
Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:
# tcpdump -n -e -ttt -r /var/log/pf.log port 80
Reference:
set up PF (packet filter) firewall on FreeBSD 8.2
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
PF supports FTP servers and FTP clients behind NAT
http://www.openbsd.org/faq/pf/index.html
http://www.openbsd.org/faq/pf/nat.html
http://www.weithenn.org/cgi-bin/wiki.pl?PF-%E5%88%A9%E7%94%A8_PF_%E8%BC%95%E9%AC%86%E9%81%94%E6%88%90_NAT
http://www.thedeepsky.com/howto/newbie_pf_guide.php
http://home.nuug.no/~peter/pf/en/ftpproblem.html
Active FTP vs. Passive FTP, a Definitive Explanation
Active FTP vs. Passive FTP, a Definitive Explanation
Install Jails on FreeBSD 8.2
Install Jails on FreeBSD 8.2
Jails, sometimes referred to as an enhanced replacement of chroot environments, are a very powerful tool for system administrators, but their basic usage can also be useful for advanced users.
Setting up the Host Environment
First, you will want to set up your real system's environment to be "jail-friendly".
For consistency, we will refer to the parent box as the "host environment", and to the jailed virtual machine as the "jail environment". Since jail is implemented using IP aliases, one of the first things to do is to disable IP services on the host system that listen on all local IP addresses for a service. If a network service is present in the host environment that binds all available IP addresses rather than specific IP addresses, it may service requests sent to jail IP addresses if the jail did not bind the port. This means changing inetd(8) to only listen on the appropriate IP address, and so forth. Add the following to /etc/rc.conf in the host environment:
sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"
man inetd for more information.
192.168.0.1 is the native IP address for the host system, in this example. Daemons that run out of inetd(8) can be easily set to use only the specified host IP address. Other daemons will need to be manually configured for some this is possible through the rc.conf(5) flags entries; for others it is necessary to modify per-application configuration files, or to recompile the applications. The following frequently deployed services must have their individual configuration files modified to limit the application to listening to a specific IP address:
To configure sshd(8), it is necessary to modify /etc/ssh/sshd_config.
To configure sendmail(8), it is necessary to modify
/etc/mail/sendmail.cf.
For named(8), it is necessary to modify /etc/namedb/named.conf.
In addition, a number of services must be recompiled in order to run them in the host environment. This includes most applications providing services using rpc(3), such as rpcbind(8), nfsd(8), and mountd(8). In general, applications for which it is not possible to specify which IP address to bind should not be run in the host environment unless they should also service requests sent to jail IP addresses. Attempting to serve NFS from the host environment may also cause confusion, and cannot be easily reconfigured to use only specific IPs, as some NFS services are hosted directly from the kernel. Any third-party network software running in the host environment should also be checked and configured so that it does not bind all IP addresses, which would result in those services' also appearing to be offered by the jail environments.
Once these daemons have been disabled or fixed in the host environment, it is best to reboot so that all daemons are in a known state, to reduce the potential for confusion later (such as finding that when you send mail to a jail, and its sendmail is down, the mail is delivered to the host, etc.).
Getting services to not listen to *
First off, we should make sure we get the system so that we have nothing listening on *, to check what what we need to modify issue this command
# sockstat | grep "\*:[0-9]"
This should give you a synopsys of all the processes and ports you need to trim down. Here are some hints with your ipv4 addr being 10.0.0.1 and your ipv6 addr being 2002::7ea9
# vi /etc/rc.conf
sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"
sshd:
# vim /etc/ssh/sshd_config
change ListenAddress derivative
ListenAddress 192.168.0.1
ListenAddress 192.168.0.111
ListenAddress 2002::7ea9
syslogd:
# vim /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"
rsyncd:
# vim /usr/local/etc/rsyncd.conf
address = 192.168.0.1
MySQL:
# vim /etc/my.cnf
bind-address=10.0.0.1
Samba: (this will get you most of the way there)
edit /usr/local/etc/smb.conf
change the following:
interfaces = 10.0.0.242/24 127.0.0.1
socket address = 10.0.0.242
bind interfaces only = yes
note: if you don't need wins lookups and netbios name translation
you can safely disable nmbd. There doesn't seem to be a way
for nmb to not listen to *:138 anyhow.
To disable nmb go to /etc/rc.conf and replace samba_enable="YES" with smbd_enable="YES"
open ntpd (xntpd listens on all and cannot be changed)
# vim /usr/local/etc/ntpd.conf
listen on 10.0.0.1
listen on 2002::7ea9
bind
edit your named.conf (may be in /var/named/etc/named.conf)
In the options section:
listen-on { 10.0.0.242; };
listen-on-v6 port 53 { 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9; };
query-source address 10.0.0.242 port *;
query-source-v6 address 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9 port *;
The file system layout is described in the following list:
- Each jail will be mounted under the /home/jail/THE_JAIL_NAME directory.
- /home/jail/read-only/root # is the template for each jail and the read-only partition for all of the jails.
- A blank directory will be created for each jail under the /home/jail directory.
- Each jail shall have its own read-write system that is based upon /home/jail/read-only/skel.
- Each jailspace (read-write portion of each jail) shall be created in /home/jail/read-write/THE_JAIL_NAME.
- Each jail will have a /s directory, that will be linked to the read-write portion of the system.
Creating the Template
This section will describe the steps needed to create the master template that will be the read-only portion for the jails to use.
It is always a good idea to update the FreeBSD system to the latest -RELEASE branch. Check the corresponding Handbook Chapter to accomplish this task. In the case the update is not feasible, the buildworld will be required in order to be able to proceed. Additionally, the sysutils/cpdup package will be required. We will use the portsnap(8) utility to download the FreeBSD Ports Collection. The Handbook Portsnap Chapter is always good reading for newcomers.
Install cpdup
# cd /usr/ports/sysutils/cpdup ; make install clean
Create directories
# mkdir /home/jail
# mkdir /home/jail/read-only
# mkdir /home/jail/read-write
Create a directory structure for the read-only file system which will contain the FreeBSD binaries for our jails:
# mkdir /home/jail/read-only/root
Change directory to the FreeBSD source tree and install the read-only file system to the jail template:
# cd /usr/src
If you have already rebuilt your userland using make world or make buildworld, you can skip following step and install your existing userland into the new jail.
# make buildworld
This command will populate the directory subtree chosen as jail's physical location on the file system with the necessary binaries, libraries, manual pages and so on.
# make installworld DESTDIR=/home/jail/read-only/root
The distribution target for make installs every needed configuration file. In simple words, it installs every installable file of /usr/src/etc/ to the /etc directory of the jail environment: /home/jail/THE_JAIL_NAME/etc/.
# make distribution DESTDIR=/home/jail/read-only/root
Next, prepare a FreeBSD Ports Collection for the jails as well as a FreeBSD source tree, which is required for mergemaster:
# mkdir /home/jail/read-only/root/usr/ports
# portsnap -p /home/jail/read-only/root/usr/ports fetch extract
# portsnap -p /home/jail/read-only/root/usr/ports fetch update
# cpdup /usr/src /home/jail/read-only/root/usr/src
Setup for Drupal:
# cd /home/jail/read-only/root
# mkdir www
# cd www
# fetch http://ftp.drupal.org/files/projects/drupal-6.22.tar.gz
# tar zxvf drupal-6.22.tar.gz
# mv drupal-6.22 drupal6
# chown -R root:wheel drupal6
Create a skeleton for the read-write portion of the system:
# mkdir /home/jail/read-only/skel
# mkdir /home/jail/read-only/skel/home
# mkdir /home/jail/read-only/skel/usr-X11R6
# mkdir /home/jail/read-only/skel/distfiles
??? # mkdir /home/jail/read-only/skel/drupal-sites
??? # mkdir /home/jail/read-only/skel/drupal-files
# cd /home/jail/read-only/root
# mv etc /home/jail/read-only/skel
# mv usr/local /home/jail/read-only/skel/usr-local
# mv tmp /home/jail/read-only/skel
# mv var /home/jail/read-only/skel
# mv root /home/jail/read-only/skel
# mv www/drupal6/sites /home/jail/read-only/skel/drupal-sites
Use mergemaster to install missing configuration files. Then get rid of the extra directories that mergemaster creates:
# mergemaster -t /home/jail/read-only/skel/var/tmp/temproot -D /home/jail/read-only/skel -i
How should I handle ./.cshrc? [Leave it to install later] enter
How should I handle ./.profile? [Leave it to install later] enter
...
keep pressing space until you see end : q
*** You installed a login.conf file, so make sure that you run
'/usr/bin/cap_mkdb /home/jail/read-only/skel/etc/login.conf'
to rebuild your login.conf database
Would you like to run it now? y or n [n] y
*** You installed a services file, so make sure that you run
'/usr/sbin/services_mkdb -q -o /home/jail/read-only/skel/var/db/services.db /home/jail/read-only/skel/etc/services'
to rebuild your services database
Would you like to run it now? y or n [n] y
*** You installed a new master.passwd file, so make sure that you run
'/usr/sbin/pwd_mkdb -d /home/jail/read-only/skel/etc -p /home/jail/read-only/skel/etc/master.passwd'
to rebuild your password files
Would you like to run it now? y or n [n] y
# cd /home/jail/read-only/skel
# rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev
Now, symlink the read-write file system to the read-only file system. Please make sure that the symlinks are created in the correct s/ locations. Real directories or the creation of directories in the wrong locations will cause the installation to fail.
# cd /home/jail/read-only/root
# mkdir s
# ln -s s/etc etc
# ln -s s/home home
# ln -s s/root root
# ln -s ../s/usr-local usr/local
# ln -s ../s/usr-X11R6 usr/X11R6
# ln -s ../../s/distfiles usr/ports/distfiles
# ln -s s/tmp tmp
# ln -s s/var var
# ln -s s/home/cwn_www/nginx www
# ln -s s/home/cwn_www/scripts scripts
??? # ln -s ../../s/drupal-sites www/drupal6/sites
??? # ln -s ../../../../s/drupal-files www/drupal6/sites/default/files
??? Changing the directory permission
??? # find /home/jail -type d -print0 | xargs -0 -I @ sh -c 'chmod 700 @ ; chown root:wheel @'
As a last step, create a generic /home/jail/read-only/skel/etc/make.conf with its contents as shown below:
# echo 'WRKDIRPREFIX?= /s/portbuild' >> /home/jail/read-only/skel/etc/make.conf
Having WRKDIRPREFIX set up this way will make it possible to compile FreeBSD ports inside each jail. Remember that the ports directory is part of the read-only system. The custom path for WRKDIRPREFIX allows builds to be done in the read-write portion of every jail.
Creating Jails:
Now that we have a complete FreeBSD jail template, we can setup and configure the jails in /etc/rc.conf. This example demonstrates the creation of 3 jails: "NS", "MAIL" and "WWW".
Put the following lines into the /etc/fstab file, so that the read-only template for the jails and the read-write space will be available in the respective jails:
# vim /etc/fstab
### read-only partitions
/home/jail/read-only/root /home/jail/read-only/ns nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/mail nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/www nullfs ro 0 0
### read-write partitions
/home/jail/read-write/ns /home/jail/read-only/ns/s nullfs rw 0 0
/home/jail/read-write/mail /home/jail/read-only/mail/s nullfs rw 0 0
/home/jail/read-write/www /home/jail/read-only/www/s nullfs rw 0 0
Note: Partitions marked with a 0 pass number are not checked by fsck(8) during boot, and partitions marked with a 0 dump number are not backed up by dump(8). We do not want fsck to check nullfs mounts or dump to back up the read-only nullfs mounts of the jails. This is why they are marked with "0 0" in the last two columns of each fstab entry above.
# vim /etc/rc.conf
Note: The jail_name_rootdir variable must not be set to a path which includes a symbolic link, otherwise the jails will refuse to start. Use the realpath(1) utility to determine a value which should be set to this variable. Please see the FreeBSD-SA-07:01.jail Security Advisory for more information.
Note: for devfs ruleset, cat /etc/defaults/devfs.rules.
Make sure your jail host has all the IP addresses for jail guests:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# ifconfig
Create the required mount points for the read-only file system of each jail:
# cd /home/jail/read-only
# mkdir ns mail www
??? # chmod 700 ns mail www
??? # chown root:wheel ns mail www
Install the read-write template into each jail. Note the use of sysutils/cpdup, which helps to ensure that a correct copy is done of each directory:
# cpdup /home/jail/read-only/skel /home/jail/read-write/ns
# cpdup /home/jail/read-only/skel /home/jail/read-write/mail
# cpdup /home/jail/read-only/skel /home/jail/read-write/www
In this phase, the jails are built and prepared to run. First, mount the required file systems for each jail, and then start them using the /etc/rc.d/jail script:
# mount -a
# df -h
Start a specific jail:
# /etc/rc.d/jail start ns
# /etc/rc.d/jail start mail
# /etc/rc.d/jail start www
Start all jails:
# /etc/rc.d/jail start
Stop a specific jail:
# /etc/rc.d/jail stop www
A clean way to shut down a jail(8) is not available at the moment. This is because commands normally used to accomplish a clean system shutdown cannot be used inside a jail. The best way to shut down a jail is to run the following command from within the jail itself or using the jexec(8) utility from outside the jail:
j-www # sh /etc/rc.shutdown
or
# jexec 2 shutdown -p now
The jails should be running now. To check if they have started correctly, use the jls(8) command. Its output should be similar to the following:
# jls
JID IP Address Hostname Path
7 192.168.0.2 j-ns /home/jail/read-only/ns
8 192.168.0.3 j-mail /home/jail/read-only/mail
9 192.168.0.4 j-www /home/jail/read-only/www
At this point, it should be possible to log onto each jail, add new users or configure daemons. The JID column indicates the jail identification number of each running jail. Use the following command in order to perform administrative tasks in the jail whose JID is 9:
# jexec 9 tcsh
Use google's public DNS server as a DNS resolver:
j-www # vi /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
Modify /etc/rc.conf:
j-www # vi /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"
### SendMail
sendmail_enable="NO"
Modify /etc/hosts:
j-www # vim /etc/hosts
::1 localhost localhost.j-www.local
127.0.0.1 localhost localhost.j-www.local
192.168.100.34 j-www.local j-www
To fix this error "hash map "Alias0": missing map file /etc/mail/aliases.db in FreeBSD jail":
# sendmail -bi
# ls -l /etc/mail/aliases.db
Note: you must edit /etc/hosts before running sendmail -bi command.
Comment out adjkerntz in crontab:
j-www # vim /etc/crontab
#1,31 0-5 * * * root adjkerntz -a
SVN server setup
# cd /usr/ports/devel/subversion
# make install clean
Make sure you DO uncheck this option:
[] BDB=off "db4 repository backend"
# mv /usr/local/bin/svn /usr/local/bin/svn.orig
# vi /usr/local/bin/svn
Don't forget to make your wrapper script executable:
# chmod +x /usr/local/bin/svn
# mkdir /home/jail/svn-rep
# cd /home/jail/svn-rep
# svnadmin create ns
# svnadmin create mail
# svnadmin create www
# vi www/conf/svnserve.conf
[general]
anon-access = none
auth-access = write
password-db = passwd
# vi www/conf/passwd
[users]
danny = mypassword
Start SVN server as a stand-alone daemon
# /usr/local/bin/svnserve -d --listen-port=3690 --listen-host=0.0.0.0 -r /home/jail/svn-rep
# vi /etc/rc.conf
### SVN
svnserve_enable="YES"
svnserve_flags="-d --listen-port=3690 --listen-host=0.0.0.0"
svnserve_data="/home/jail/svn-rep"
svnserve_user="root"
svnserve_group="wheel"
# svn import /home/jail/read-only/skel/drupal-sites/all file:///home/jail/svn-rep/www -m "initial import"
SVN client setup
# rm -r /www/drupal6/sites/all/*
# svn checkout svn://192.168.0.1/www /www/drupal6/sites/all
sending mail through sendmail inside a jail:
# cat mail.php
<?php
mail('test@example.com', 'My Subject', 'my msg');
?>
# php mail.php
collect: Cannot write ./dfpA4NVE5H097195 (bfcommit, uid=25, gid=25): Permission denied
queueup: cannot create queue file ./qfpA4NVE5H097195, euid=25, fd=-1, fp=0x0: Permission denied
# chown smmsp:smmsp /var/spool/clientmqueue
# chmod 770 /var/spool/clientmqueue
Fine Tuning and Administration
There are several options which can be set for any jail, and various ways of combining a host FreeBSD system with jails, to produce higher level applications. This section presents:
Some of the options available for tuning the behavior and security restrictions implemented by a jail installation.
Some of the high-level applications for jail management, which are available through the FreeBSD Ports Collection, and can be used to implement overall jail-based solutions.
Fine tuning of a jail's configuration is mostly done by setting sysctl(8) variables. A special subtree of sysctl exists as a basis for organizing all the relevant options: the security.jail.* hierarchy of FreeBSD kernel options. Here is a list of the main jail-related sysctls, complete with their default value. Names should be self-explanatory, but for more information about them, please refer to the jail(8) and sysctl(8) manual pages.
security.jail.set_hostname_allowed: 1
security.jail.socket_unixiproute_only: 1
security.jail.sysvipc_allowed: 0
security.jail.enforce_statfs: 2
security.jail.allow_raw_sockets: 0
security.jail.chflags_allowed: 0
security.jail.jailed: 0
These variables can be used by the system administrator of the host system to add or remove some of the limitations imposed by default on the root user. Note that there are some limitations which cannot be removed. The root user is not allowed to mount or unmount file systems from within a jail(8). The root inside a jail may not load or unload devfs(8) rulesets, set firewall rules, or do many other administrative tasks which require modifications of in-kernel data, such as setting the securelevel of the kernel.
Ping from jail not permitted error
ICMP is disallowed by defaut for jails, see the sysctl :
# echo security.jail.allow_raw_sockets=1 >> /etc/sysctl.conf
Now restart your jails and the problem should be fixed.
There are good reasons for this default, so if you test remember to set it back when you are done.
Also, on a point of style, jails in their current form (see VIMAGE) do not get a network stack of their own so they don't have a firewall but share the hosts' network and firewall, etc.
Cannot remove files inside a jail directory
override r-sr-xr-x root/wheel schg for j/mroot/bin/rcp? y
rm: j/mroot/bin/rcp: Operation not permitted
rm: j/mroot/bin: Directory not empty
override r--r--r-- root/wheel schg for j/mroot/lib/libc.so.7? y
rm: j/mroot/lib/libc.so.7: Operation not permitted
# chflags -R noschg /home/jail/read-write/ns
# rm -r /home/jail/read-write/ns
High-level administrative tools in FreeBSD Ports Collection
# cd /usr/ports/sysutils/jailutils ; make install clean
# cd /usr/ports/sysutils/ezjail ; make install clean
Reference:
http://gala4th.blogspot.com/2012/01/install-jails-on-freebsd-82.html
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
VIMAGE - Better virtualization in FreeBSD 8
http://gala4th.blogspot.com/2011/10/vimage-better-virtualization-in-freebsd.html
FreeBSD 8 VIMAGE + epair howto
http://gala4th.blogspot.com/2011/10/freebsd-8-vimage-epair-howto.html
Creating a FreeBSD Jail
http://www.section6.net/wiki/index.php/Creating_a_FreeBSD_Jail
Jails, sometimes referred to as an enhanced replacement of chroot environments, are a very powerful tool for system administrators, but their basic usage can also be useful for advanced users.
Setting up the Host Environment
First, you will want to set up your real system's environment to be "jail-friendly".
For consistency, we will refer to the parent box as the "host environment", and to the jailed virtual machine as the "jail environment". Since jail is implemented using IP aliases, one of the first things to do is to disable IP services on the host system that listen on all local IP addresses for a service. If a network service is present in the host environment that binds all available IP addresses rather than specific IP addresses, it may service requests sent to jail IP addresses if the jail did not bind the port. This means changing inetd(8) to only listen on the appropriate IP address, and so forth. Add the following to /etc/rc.conf in the host environment:
sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"
man inetd for more information.
192.168.0.1 is the native IP address for the host system, in this example. Daemons that run out of inetd(8) can be easily set to use only the specified host IP address. Other daemons will need to be manually configured for some this is possible through the rc.conf(5) flags entries; for others it is necessary to modify per-application configuration files, or to recompile the applications. The following frequently deployed services must have their individual configuration files modified to limit the application to listening to a specific IP address:
To configure sshd(8), it is necessary to modify /etc/ssh/sshd_config.
To configure sendmail(8), it is necessary to modify
/etc/mail/sendmail.cf.
For named(8), it is necessary to modify /etc/namedb/named.conf.
In addition, a number of services must be recompiled in order to run them in the host environment. This includes most applications providing services using rpc(3), such as rpcbind(8), nfsd(8), and mountd(8). In general, applications for which it is not possible to specify which IP address to bind should not be run in the host environment unless they should also service requests sent to jail IP addresses. Attempting to serve NFS from the host environment may also cause confusion, and cannot be easily reconfigured to use only specific IPs, as some NFS services are hosted directly from the kernel. Any third-party network software running in the host environment should also be checked and configured so that it does not bind all IP addresses, which would result in those services' also appearing to be offered by the jail environments.
Once these daemons have been disabled or fixed in the host environment, it is best to reboot so that all daemons are in a known state, to reduce the potential for confusion later (such as finding that when you send mail to a jail, and its sendmail is down, the mail is delivered to the host, etc.).
Getting services to not listen to *
First off, we should make sure we get the system so that we have nothing listening on *, to check what what we need to modify issue this command
# sockstat | grep "\*:[0-9]"
This should give you a synopsys of all the processes and ports you need to trim down. Here are some hints with your ipv4 addr being 10.0.0.1 and your ipv6 addr being 2002::7ea9
# vi /etc/rc.conf
sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"
sshd:
# vim /etc/ssh/sshd_config
change ListenAddress derivative
ListenAddress 192.168.0.1
ListenAddress 192.168.0.111
ListenAddress 2002::7ea9
syslogd:
# vim /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"
rsyncd:
# vim /usr/local/etc/rsyncd.conf
address = 192.168.0.1
MySQL:
# vim /etc/my.cnf
bind-address=10.0.0.1
Samba: (this will get you most of the way there)
edit /usr/local/etc/smb.conf
change the following:
interfaces = 10.0.0.242/24 127.0.0.1
socket address = 10.0.0.242
bind interfaces only = yes
note: if you don't need wins lookups and netbios name translation
you can safely disable nmbd. There doesn't seem to be a way
for nmb to not listen to *:138 anyhow.
To disable nmb go to /etc/rc.conf and replace samba_enable="YES" with smbd_enable="YES"
open ntpd (xntpd listens on all and cannot be changed)
# vim /usr/local/etc/ntpd.conf
listen on 10.0.0.1
listen on 2002::7ea9
bind
edit your named.conf (may be in /var/named/etc/named.conf)
In the options section:
listen-on { 10.0.0.242; };
listen-on-v6 port 53 { 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9; };
query-source address 10.0.0.242 port *;
query-source-v6 address 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9 port *;
The file system layout is described in the following list:
- Each jail will be mounted under the /home/jail/THE_JAIL_NAME directory.
- /home/jail/read-only/root # is the template for each jail and the read-only partition for all of the jails.
- A blank directory will be created for each jail under the /home/jail directory.
- Each jail shall have its own read-write system that is based upon /home/jail/read-only/skel.
- Each jailspace (read-write portion of each jail) shall be created in /home/jail/read-write/THE_JAIL_NAME.
- Each jail will have a /s directory, that will be linked to the read-write portion of the system.
Creating the Template
This section will describe the steps needed to create the master template that will be the read-only portion for the jails to use.
It is always a good idea to update the FreeBSD system to the latest -RELEASE branch. Check the corresponding Handbook Chapter to accomplish this task. In the case the update is not feasible, the buildworld will be required in order to be able to proceed. Additionally, the sysutils/cpdup package will be required. We will use the portsnap(8) utility to download the FreeBSD Ports Collection. The Handbook Portsnap Chapter is always good reading for newcomers.
Install cpdup
# cd /usr/ports/sysutils/cpdup ; make install clean
Create directories
# mkdir /home/jail
# mkdir /home/jail/read-only
# mkdir /home/jail/read-write
Create a directory structure for the read-only file system which will contain the FreeBSD binaries for our jails:
# mkdir /home/jail/read-only/root
Change directory to the FreeBSD source tree and install the read-only file system to the jail template:
# cd /usr/src
If you have already rebuilt your userland using make world or make buildworld, you can skip following step and install your existing userland into the new jail.
# make buildworld
This command will populate the directory subtree chosen as jail's physical location on the file system with the necessary binaries, libraries, manual pages and so on.
# make installworld DESTDIR=/home/jail/read-only/root
The distribution target for make installs every needed configuration file. In simple words, it installs every installable file of /usr/src/etc/ to the /etc directory of the jail environment: /home/jail/THE_JAIL_NAME/etc/.
# make distribution DESTDIR=/home/jail/read-only/root
Next, prepare a FreeBSD Ports Collection for the jails as well as a FreeBSD source tree, which is required for mergemaster:
# mkdir /home/jail/read-only/root/usr/ports
# portsnap -p /home/jail/read-only/root/usr/ports fetch extract
# portsnap -p /home/jail/read-only/root/usr/ports fetch update
# cpdup /usr/src /home/jail/read-only/root/usr/src
Setup for Drupal:
# cd /home/jail/read-only/root
# mkdir www
# cd www
# fetch http://ftp.drupal.org/files/projects/drupal-6.22.tar.gz
# tar zxvf drupal-6.22.tar.gz
# mv drupal-6.22 drupal6
# chown -R root:wheel drupal6
Create a skeleton for the read-write portion of the system:
# mkdir /home/jail/read-only/skel
# mkdir /home/jail/read-only/skel/home
# mkdir /home/jail/read-only/skel/usr-X11R6
# mkdir /home/jail/read-only/skel/distfiles
??? # mkdir /home/jail/read-only/skel/drupal-sites
??? # mkdir /home/jail/read-only/skel/drupal-files
# cd /home/jail/read-only/root
# mv etc /home/jail/read-only/skel
# mv usr/local /home/jail/read-only/skel/usr-local
# mv tmp /home/jail/read-only/skel
# mv var /home/jail/read-only/skel
# mv root /home/jail/read-only/skel
# mv www/drupal6/sites /home/jail/read-only/skel/drupal-sites
Use mergemaster to install missing configuration files. Then get rid of the extra directories that mergemaster creates:
# mergemaster -t /home/jail/read-only/skel/var/tmp/temproot -D /home/jail/read-only/skel -i
How should I handle ./.cshrc? [Leave it to install later] enter
How should I handle ./.profile? [Leave it to install later] enter
...
keep pressing space until you see end : q
*** You installed a login.conf file, so make sure that you run
'/usr/bin/cap_mkdb /home/jail/read-only/skel/etc/login.conf'
to rebuild your login.conf database
Would you like to run it now? y or n [n] y
*** You installed a services file, so make sure that you run
'/usr/sbin/services_mkdb -q -o /home/jail/read-only/skel/var/db/services.db /home/jail/read-only/skel/etc/services'
to rebuild your services database
Would you like to run it now? y or n [n] y
*** You installed a new master.passwd file, so make sure that you run
'/usr/sbin/pwd_mkdb -d /home/jail/read-only/skel/etc -p /home/jail/read-only/skel/etc/master.passwd'
to rebuild your password files
Would you like to run it now? y or n [n] y
# cd /home/jail/read-only/skel
# rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev
Now, symlink the read-write file system to the read-only file system. Please make sure that the symlinks are created in the correct s/ locations. Real directories or the creation of directories in the wrong locations will cause the installation to fail.
# cd /home/jail/read-only/root
# mkdir s
# ln -s s/etc etc
# ln -s s/home home
# ln -s s/root root
# ln -s ../s/usr-local usr/local
# ln -s ../s/usr-X11R6 usr/X11R6
# ln -s ../../s/distfiles usr/ports/distfiles
# ln -s s/tmp tmp
# ln -s s/var var
# ln -s s/home/cwn_www/nginx www
# ln -s s/home/cwn_www/scripts scripts
??? # ln -s ../../s/drupal-sites www/drupal6/sites
??? # ln -s ../../../../s/drupal-files www/drupal6/sites/default/files
??? Changing the directory permission
??? # find /home/jail -type d -print0 | xargs -0 -I @ sh -c 'chmod 700 @ ; chown root:wheel @'
As a last step, create a generic /home/jail/read-only/skel/etc/make.conf with its contents as shown below:
# echo 'WRKDIRPREFIX?= /s/portbuild' >> /home/jail/read-only/skel/etc/make.conf
Having WRKDIRPREFIX set up this way will make it possible to compile FreeBSD ports inside each jail. Remember that the ports directory is part of the read-only system. The custom path for WRKDIRPREFIX allows builds to be done in the read-write portion of every jail.
Creating Jails:
Now that we have a complete FreeBSD jail template, we can setup and configure the jails in /etc/rc.conf. This example demonstrates the creation of 3 jails: "NS", "MAIL" and "WWW".
Put the following lines into the /etc/fstab file, so that the read-only template for the jails and the read-write space will be available in the respective jails:
# vim /etc/fstab
### read-only partitions
/home/jail/read-only/root /home/jail/read-only/ns nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/mail nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/www nullfs ro 0 0
### read-write partitions
/home/jail/read-write/ns /home/jail/read-only/ns/s nullfs rw 0 0
/home/jail/read-write/mail /home/jail/read-only/mail/s nullfs rw 0 0
/home/jail/read-write/www /home/jail/read-only/www/s nullfs rw 0 0
Note: Partitions marked with a 0 pass number are not checked by fsck(8) during boot, and partitions marked with a 0 dump number are not backed up by dump(8). We do not want fsck to check nullfs mounts or dump to back up the read-only nullfs mounts of the jails. This is why they are marked with "0 0" in the last two columns of each fstab entry above.
# vim /etc/rc.conf
### for jail host ifconfig_re0="inet 192.168.0.1 netmask 255.255.255.0" ### for jail guests ifconfig_re0_alias0="inet 192.168.0.2 netmask 255.255.255.255" ifconfig_re0_alias1="inet 192.168.0.3 netmask 255.255.255.255" ifconfig_re0_alias2="inet 192.168.0.4 netmask 255.255.255.255" ### jail settings jail_enable="YES" # Set to NO to disable starting of any jails jail_set_hostname_allow="NO" # Allow root user in a jail to change its hostname #jail_socket_unixiproute_only="YES" # Route only TCP/IP within a jail jail_list="ns mail www" # Space separated list of names of jails # Note: Jail names in jail_list should contain alphanumeric characters only. ### j-ns jail_ns_hostname="j-ns.local" # jail's hostname jail_ns_ip="192.168.0.2" # jail's IP address jail_ns_rootdir="/home/jail/read-only/ns" # jail's root directory jail_ns_devfs_enable="YES" # mount devfs in the jail jail_ns_devfs_ruleset="ns_ruleset" # devfs ruleset to apply to jail ### j-mail jail_mail_hostname="j-mail.local" # jail's hostname jail_mail_ip="192.168.0.3" # jail's IP address jail_mail_rootdir="/home/jail/read-only/mail" # jail's root directory jail_mail_devfs_enable="YES" # mount devfs in the jail jail_mail_devfs_ruleset="mail_ruleset" # devfs ruleset to apply to jail ### j-www jail_www_hostname="j-www.local" # jail's hostname jail_www_ip="192.168.0.4" # jail's IP address jail_www_rootdir="/home/jail/read-only/www" # jail's root directory jail_www_devfs_enable="YES" # mount devfs in the jail jail_www_devfs_ruleset="www_ruleset" # devfs ruleset to apply to jail
Note: The jail_name_rootdir variable must not be set to a path which includes a symbolic link, otherwise the jails will refuse to start. Use the realpath(1) utility to determine a value which should be set to this variable. Please see the FreeBSD-SA-07:01.jail Security Advisory for more information.
Note: for devfs ruleset, cat /etc/defaults/devfs.rules.
Make sure your jail host has all the IP addresses for jail guests:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# ifconfig
Create the required mount points for the read-only file system of each jail:
# cd /home/jail/read-only
# mkdir ns mail www
??? # chmod 700 ns mail www
??? # chown root:wheel ns mail www
Install the read-write template into each jail. Note the use of sysutils/cpdup, which helps to ensure that a correct copy is done of each directory:
# cpdup /home/jail/read-only/skel /home/jail/read-write/ns
# cpdup /home/jail/read-only/skel /home/jail/read-write/mail
# cpdup /home/jail/read-only/skel /home/jail/read-write/www
In this phase, the jails are built and prepared to run. First, mount the required file systems for each jail, and then start them using the /etc/rc.d/jail script:
# mount -a
# df -h
Start a specific jail:
# /etc/rc.d/jail start ns
# /etc/rc.d/jail start mail
# /etc/rc.d/jail start www
Start all jails:
# /etc/rc.d/jail start
Stop a specific jail:
# /etc/rc.d/jail stop www
A clean way to shut down a jail(8) is not available at the moment. This is because commands normally used to accomplish a clean system shutdown cannot be used inside a jail. The best way to shut down a jail is to run the following command from within the jail itself or using the jexec(8) utility from outside the jail:
j-www # sh /etc/rc.shutdown
or
# jexec 2 shutdown -p now
The jails should be running now. To check if they have started correctly, use the jls(8) command. Its output should be similar to the following:
# jls
JID IP Address Hostname Path
7 192.168.0.2 j-ns /home/jail/read-only/ns
8 192.168.0.3 j-mail /home/jail/read-only/mail
9 192.168.0.4 j-www /home/jail/read-only/www
At this point, it should be possible to log onto each jail, add new users or configure daemons. The JID column indicates the jail identification number of each running jail. Use the following command in order to perform administrative tasks in the jail whose JID is 9:
# jexec 9 tcsh
Use google's public DNS server as a DNS resolver:
j-www # vi /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
Modify /etc/rc.conf:
j-www # vi /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"
### SendMail
sendmail_enable="NO"
Modify /etc/hosts:
j-www # vim /etc/hosts
::1 localhost localhost.j-www.local
127.0.0.1 localhost localhost.j-www.local
192.168.100.34 j-www.local j-www
To fix this error "hash map "Alias0": missing map file /etc/mail/aliases.db in FreeBSD jail":
# sendmail -bi
# ls -l /etc/mail/aliases.db
Note: you must edit /etc/hosts before running sendmail -bi command.
Comment out adjkerntz in crontab:
j-www # vim /etc/crontab
#1,31 0-5 * * * root adjkerntz -a
SVN server setup
# cd /usr/ports/devel/subversion
# make install clean
Make sure you DO uncheck this option:
[] BDB=off "db4 repository backend"
# mv /usr/local/bin/svn /usr/local/bin/svn.orig
# vi /usr/local/bin/svn
#!/bin/sh
### initialize
svnarg=""
### use encoding utf-8 as default if run "svn ci" or "svn commit".
if [ "$1" != "help" ]; then
for myarg in "$@"; do
if [ "${myarg}" = "commit" ] || [ "${myarg}" = "ci" ]; then
svnarg="--encoding utf-8"
break
fi
done
fi
### wrapper script to set umask to 027 on subversion binaries
### Note: the meaning of each umask:
### umask 002 // File permission 644. Owner can read/write. Group and Others can only read.
### umask 007 // File permission 660. Owner and Group can read/write. Others can not read or write.
### umask 027 // File permission 640. Owner can read/write. Group can read. Others can not read or write.
umask 027
### svn command
/usr/local/bin/svn.orig ${svnarg} "$@"
Don't forget to make your wrapper script executable:
# chmod +x /usr/local/bin/svn
# mkdir /home/jail/svn-rep
# cd /home/jail/svn-rep
# svnadmin create ns
# svnadmin create mail
# svnadmin create www
# vi www/conf/svnserve.conf
[general]
anon-access = none
auth-access = write
password-db = passwd
# vi www/conf/passwd
[users]
danny = mypassword
Start SVN server as a stand-alone daemon
# /usr/local/bin/svnserve -d --listen-port=3690 --listen-host=0.0.0.0 -r /home/jail/svn-rep
# vi /etc/rc.conf
### SVN
svnserve_enable="YES"
svnserve_flags="-d --listen-port=3690 --listen-host=0.0.0.0"
svnserve_data="/home/jail/svn-rep"
svnserve_user="root"
svnserve_group="wheel"
# svn import /home/jail/read-only/skel/drupal-sites/all file:///home/jail/svn-rep/www -m "initial import"
SVN client setup
# rm -r /www/drupal6/sites/all/*
# svn checkout svn://192.168.0.1/www /www/drupal6/sites/all
sending mail through sendmail inside a jail:
# cat mail.php
<?php
mail('test@example.com', 'My Subject', 'my msg');
?>
# php mail.php
collect: Cannot write ./dfpA4NVE5H097195 (bfcommit, uid=25, gid=25): Permission denied
queueup: cannot create queue file ./qfpA4NVE5H097195, euid=25, fd=-1, fp=0x0: Permission denied
# chown smmsp:smmsp /var/spool/clientmqueue
# chmod 770 /var/spool/clientmqueue
Fine Tuning and Administration
There are several options which can be set for any jail, and various ways of combining a host FreeBSD system with jails, to produce higher level applications. This section presents:
Some of the options available for tuning the behavior and security restrictions implemented by a jail installation.
Some of the high-level applications for jail management, which are available through the FreeBSD Ports Collection, and can be used to implement overall jail-based solutions.
Fine tuning of a jail's configuration is mostly done by setting sysctl(8) variables. A special subtree of sysctl exists as a basis for organizing all the relevant options: the security.jail.* hierarchy of FreeBSD kernel options. Here is a list of the main jail-related sysctls, complete with their default value. Names should be self-explanatory, but for more information about them, please refer to the jail(8) and sysctl(8) manual pages.
security.jail.set_hostname_allowed: 1
security.jail.socket_unixiproute_only: 1
security.jail.sysvipc_allowed: 0
security.jail.enforce_statfs: 2
security.jail.allow_raw_sockets: 0
security.jail.chflags_allowed: 0
security.jail.jailed: 0
These variables can be used by the system administrator of the host system to add or remove some of the limitations imposed by default on the root user. Note that there are some limitations which cannot be removed. The root user is not allowed to mount or unmount file systems from within a jail(8). The root inside a jail may not load or unload devfs(8) rulesets, set firewall rules, or do many other administrative tasks which require modifications of in-kernel data, such as setting the securelevel of the kernel.
Ping from jail not permitted error
ICMP is disallowed by defaut for jails, see the sysctl :
# echo security.jail.allow_raw_sockets=1 >> /etc/sysctl.conf
Now restart your jails and the problem should be fixed.
There are good reasons for this default, so if you test remember to set it back when you are done.
Also, on a point of style, jails in their current form (see VIMAGE) do not get a network stack of their own so they don't have a firewall but share the hosts' network and firewall, etc.
Cannot remove files inside a jail directory
override r-sr-xr-x root/wheel schg for j/mroot/bin/rcp? y
rm: j/mroot/bin/rcp: Operation not permitted
rm: j/mroot/bin: Directory not empty
override r--r--r-- root/wheel schg for j/mroot/lib/libc.so.7? y
rm: j/mroot/lib/libc.so.7: Operation not permitted
# chflags -R noschg /home/jail/read-write/ns
# rm -r /home/jail/read-write/ns
High-level administrative tools in FreeBSD Ports Collection
# cd /usr/ports/sysutils/jailutils ; make install clean
# cd /usr/ports/sysutils/ezjail ; make install clean
Reference:
http://gala4th.blogspot.com/2012/01/install-jails-on-freebsd-82.html
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
VIMAGE - Better virtualization in FreeBSD 8
http://gala4th.blogspot.com/2011/10/vimage-better-virtualization-in-freebsd.html
FreeBSD 8 VIMAGE + epair howto
http://gala4th.blogspot.com/2011/10/freebsd-8-vimage-epair-howto.html
Creating a FreeBSD Jail
http://www.section6.net/wiki/index.php/Creating_a_FreeBSD_Jail
Thursday, January 19, 2012
Change to a different IP address of SVN server
Change to a different IP address of SVN server
use TortoiseSVN's "Relocate" menu
or
# svn relocate svn://old-ip-address/YOUR_REPOSITORY_NAME svn://new-ip-address/YOUR_REPOSITORY_NAME
use TortoiseSVN's "Relocate" menu
or
# svn relocate svn://old-ip-address/YOUR_REPOSITORY_NAME svn://new-ip-address/YOUR_REPOSITORY_NAME
PF supports FTP servers and FTP clients behind NAT
PF supports FTP servers and FTP clients behind NAT
WAN0 IP: 11.11.11.11
WAN1 IP: 22.22.22.22
FTP0 server (behind NAT) IP: 192.168.100.1
FTP1 server (behind NAT) IP: 192.168.100.2
# vim /etc/rc.local
### ftp-proxy for outside world FTP clients accessing internal FTP server behand NAT.
/usr/sbin/setfib 0 /usr/sbin/ftp-proxy -R 192.168.100.1 -p 21 -b 11.11.11.11
/usr/sbin/setfib 1 /usr/sbin/ftp-proxy -R 192.168.100.2 -p 21 -b 22.22.22.22
### Note:
### -b // bind to the external IP.
### -R // redirect to internal FTP server IP.
### ftp-proxy for internal FTP clients behand NAT accessing outside world FTP servers.
/usr/sbin/ftp-proxy -p 8021 -b 127.0.0.1
# vim /etc/pf.conf
### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from $dmz_if/24 to any port 21 -> 127.0.0.1 port 8021
### enabloe log for debugging purpose.
block log all
### [FTP] from ROUTER to ANY
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port 21 rtable 0
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port > 1023 rtable 0
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port 21 rtable 1
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port > 1023 rtable 1
### [FTP] from ANY to $dmz_ftp0_ip0 $dmz_ftp1_ip0
pass in quick log on $wan_if0 proto tcp from any to $wan_if0 port 21 rtable 0
pass in quick log on $wan_if0 proto tcp from any to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass in quick log on $wan_if1 proto tcp from any to $wan_if1 port 21 rtable 1
pass in quick log on $wan_if1 proto tcp from any to $dmz_ftp1_ip0 port 30000:30900 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 21 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 21 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 30000:30900 rtable 1
### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"
### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto tcp from $dmz_if/24 to any port 21 rtable 0
Flush the pf.conf setting:
# pfctl -f /etc/pf.conf
Try to connect to ftp:
# ftp ftp.freebsd.org
Name: anonymous
Password: anonymous
WAN0 IP: 11.11.11.11
WAN1 IP: 22.22.22.22
FTP0 server (behind NAT) IP: 192.168.100.1
FTP1 server (behind NAT) IP: 192.168.100.2
# vim /etc/rc.local
### ftp-proxy for outside world FTP clients accessing internal FTP server behand NAT.
/usr/sbin/setfib 0 /usr/sbin/ftp-proxy -R 192.168.100.1 -p 21 -b 11.11.11.11
/usr/sbin/setfib 1 /usr/sbin/ftp-proxy -R 192.168.100.2 -p 21 -b 22.22.22.22
### Note:
### -b // bind to the external IP.
### -R // redirect to internal FTP server IP.
### ftp-proxy for internal FTP clients behand NAT accessing outside world FTP servers.
/usr/sbin/ftp-proxy -p 8021 -b 127.0.0.1
# vim /etc/pf.conf
### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from $dmz_if/24 to any port 21 -> 127.0.0.1 port 8021
### enabloe log for debugging purpose.
block log all
### [FTP] from ROUTER to ANY
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port 21 rtable 0
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port > 1023 rtable 0
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port 21 rtable 1
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port > 1023 rtable 1
### [FTP] from ANY to $dmz_ftp0_ip0 $dmz_ftp1_ip0
pass in quick log on $wan_if0 proto tcp from any to $wan_if0 port 21 rtable 0
pass in quick log on $wan_if0 proto tcp from any to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass in quick log on $wan_if1 proto tcp from any to $wan_if1 port 21 rtable 1
pass in quick log on $wan_if1 proto tcp from any to $dmz_ftp1_ip0 port 30000:30900 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 21 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 21 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 30000:30900 rtable 1
### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"
### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto tcp from $dmz_if/24 to any port 21 rtable 0
Flush the pf.conf setting:
# pfctl -f /etc/pf.conf
Try to connect to ftp:
# ftp ftp.freebsd.org
Name: anonymous
Password: anonymous
Active FTP vs. Passive FTP, a Definitive Explanation
Active FTP vs. Passive FTP, a Definitive Explanation
Contents:
Introduction
The Basics
Active FTP
Active FTP Example
Passive FTP
Passive FTP Example
Other Notes
Summary
References
Appendix 1: Configuration of Common FTP Servers
Introduction
One of the most commonly seen questions when dealing with firewalls and other Internet connectivity issues is the difference between active and passive FTP and how best to support either or both of them. Hopefully the following text will help to clear up some of the confusion over how to support FTP in a firewalled environment.
This may not be the definitive explanation, as the title claims, however, I've heard enough good feedback and seen this document linked in enough places to know that quite a few people have found it to be useful. I am always looking for ways to improve things though, and if you find something that is not quite clear or needs more explanation, please let me know! Recent additions to this document include the examples of both active and passive command line FTP sessions. These session examples should help make things a bit clearer. They also provide a nice picture into what goes on behind the scenes during an FTP session. Now, on to the information...
The Basics
FTP is a TCP based service exclusively. There is no UDP component to FTP. FTP is an unusual service in that it utilizes two ports, a 'data' port and a 'command' port (also known as the control port). Traditionally these are port 21 for the command port and port 20 for the data port. The confusion begins however, when we find that depending on the mode, the data port is not always on port 20.
Active FTP
In active mode FTP the client connects from a random unprivileged port (N > 1023) to the FTP server's command port, port 21. Then, the client starts listening to port N+1 and sends the FTP command PORT N+1 to the FTP server. The server will then connect back to the client's specified data port from its local data port, which is port 20.
From the server-side firewall's standpoint, to support active mode FTP the following communication channels need to be opened:
FTP server's port 21 from anywhere (Client initiates connection)
FTP server's port 21 to ports > 1023 (Server responds to client's control port)
FTP server's port 20 to ports > 1023 (Server initiates data connection to client's data port)
FTP server's port 20 from ports > 1023 (Client sends ACKs to server's data port)
When drawn out, the connection appears as follows:
In step 1, the client's command port contacts the server's command port and sends the command PORT 1027. The server then sends an ACK back to the client's command port in step 2. In step 3 the server initiates a connection on its local data port to the data port the client specified earlier. Finally, the client sends an ACK back as shown in step 4.
The main problem with active mode FTP actually falls on the client side. The FTP client doesn't make the actual connection to the data port of the server--it simply tells the server what port it is listening on and the server connects back to the specified port on the client. From the client side firewall this appears to be an outside system initiating a connection to an internal client--something that is usually blocked.
Active FTP Example
Below is an actual example of an active FTP session. The only things that have been changed are the server names, IP addresses, and user names. In this example an FTP session is initiated from testbox1.slacksite.com (192.168.150.80), a linux box running the standard FTP command line client, to testbox2.slacksite.com (192.168.150.90), a linux box running ProFTPd 1.2.2RC2. The debugging (-d) flag is used with the FTP client to show what is going on behind the scenes. Everything in red is the debugging output which shows the actual FTP commands being sent to the server and the responses generated from those commands. Normal server output is shown in black, and user input is in bold.
There are a few interesting things to consider about this dialog. Notice that when the PORT command is issued, it specifies a port on the client (192.168.150.80) system, rather than the server. We will see the opposite behavior when we use passive FTP. While we are on the subject, a quick note about the format of the PORT command. As you can see in the example below it is formatted as a series of six numbers separated by commas. The first four octets are the IP address while the last two octets comprise the port that will be used for the data connection. To find the actual port multiply the fifth octet by 256 and then add the sixth octet to the total. Thus in the example below the port number is ( (14*256) + 178), or 3762. A quick check with netstat should confirm this information.
testbox1: {/home/p-t/slacker/public_html} % ftp -d testbox2
Connected to testbox2.slacksite.com.
220 testbox2.slacksite.com FTP server ready.
Name (testbox2:slacker): slacker
---> USER slacker
331 Password required for slacker.
Password: TmpPass
---> PASS XXXX
230 User slacker logged in.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,150,80,14,178
200 PORT command successful.
---> LIST
150 Opening ASCII mode data connection for file list.
drwx------ 3 slacker users 104 Jul 27 01:45 public_html
226 Transfer complete.
ftp> quit
---> QUIT
221 Goodbye.
Passive FTP
In order to resolve the issue of the server initiating the connection to the client a different method for FTP connections was developed. This was known as passive mode, or PASV, after the command used by the client to tell the server it is in passive mode.
In passive mode FTP the client initiates both connections to the server, solving the problem of firewalls filtering the incoming data port connection to the client from the server. When opening an FTP connection, the client opens two random unprivileged ports locally (N > 1023 and N+1). The first port contacts the server on port 21, but instead of then issuing a PORT command and allowing the server to connect back to its data port, the client will issue the PASV command. The result of this is that the server then opens a random unprivileged port (P > 1023) and sends the PORT P command back to the client. The client then initiates the connection from port N+1 to port P on the server to transfer data.
From the server-side firewall's standpoint, to support passive mode FTP the following communication channels need to be opened:
FTP server's port 21 from anywhere (Client initiates connection)
FTP server's port 21 to ports > 1023 (Server responds to client's control port)
FTP server's ports > 1023 from anywhere (Client initiates data connection to random port specified by server)
FTP server's ports > 1023 to remote ports > 1023 (Server sends ACKs (and data) to client's data port)
When drawn, a passive mode FTP connection looks like this:
In step 1, the client contacts the server on the command port and issues the PASV command. The server then replies in step 2 with PORT 2024, telling the client which port it is listening to for the data connection. In step 3 the client then initiates the data connection from its data port to the specified server data port. Finally, the server sends back an ACK in step 4 to the client's data port.
While passive mode FTP solves many of the problems from the client side, it opens up a whole range of problems on the server side. The biggest issue is the need to allow any remote connection to high numbered ports on the server. Fortunately, many FTP daemons, including the popular WU-FTPD allow the administrator to specify a range of ports which the FTP server will use. See Appendix 1 for more information.
The second issue involves supporting and troubleshooting clients which do (or do not) support passive mode. As an example, the command line FTP utility provided with Solaris does not support passive mode, necessitating a third-party FTP client, such as ncftp.
With the massive popularity of the World Wide Web, many people prefer to use their web browser as an FTP client. Most browsers only support passive mode when accessing ftp:// URLs. This can either be good or bad depending on what the servers and firewalls are configured to support.
Passive FTP Example
Below is an actual example of a passive FTP session. The only things that have been changed are the server names, IP addresses, and user names. In this example an FTP session is initiated from testbox1.slacksite.com (192.168.150.80), a linux box running the standard FTP command line client, to testbox2.slacksite.com (192.168.150.90), a linux box running ProFTPd 1.2.2RC2. The debugging (-d) flag is used with the FTP client to show what is going on behind the scenes. Everything in red is the debugging output which shows the actual FTP commands being sent to the server and the responses generated from those commands. Normal server output is shown in black, and user input is in bold.
Notice the difference in the PORT command in this example as opposed to the active FTP example. Here, we see a port being opened on the server (192.168.150.90) system, rather than the client. See the discussion about the format of the PORT command above, in the Active FTP Example section.
testbox1: {/home/p-t/slacker/public_html} % ftp -d testbox2
Connected to testbox2.slacksite.com.
220 testbox2.slacksite.com FTP server ready.
Name (testbox2:slacker): slacker
---> USER slacker
331 Password required for slacker.
Password: TmpPass
---> PASS XXXX
230 User slacker logged in.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> passive
Passive mode on.
ftp> ls
ftp: setsockopt (ignored): Permission denied
---> PASV
227 Entering Passive Mode (192,168,150,90,195,149).
---> LIST
150 Opening ASCII mode data connection for file list
drwx------ 3 slacker users 104 Jul 27 01:45 public_html
226 Transfer complete.
ftp> quit
---> QUIT
221 Goodbye.
Other Notes
A reader, Maarten Sjouw, pointed out that active FTP will not function when used in conjunction with a client-side NAT (Network Address Translation) device which is not smart enough to alter the IP address info in FTP packets.
Summary
The following chart should help admins remember how each FTP mode works:
Active FTP :
command : client >1023 -> server 21
data : client >1023 <- server 20
Passive FTP :
command : client >1023 -> server 21
data : client >1023 -> server >1023
A quick summary of the pros and cons of active vs. passive FTP is also in order:
Active FTP is beneficial to the FTP server admin, but detrimental to the client side admin. The FTP server attempts to make connections to random high ports on the client, which would almost certainly be blocked by a firewall on the client side. Passive FTP is beneficial to the client, but detrimental to the FTP server admin. The client will make both connections to the server, but one of them will be to a random high port, which would almost certainly be blocked by a firewall on the server side.
Luckily, there is somewhat of a compromise. Since admins running FTP servers will need to make their servers accessible to the greatest number of clients, they will almost certainly need to support passive FTP. The exposure of high level ports on the server can be minimized by specifying a limited port range for the FTP server to use. Thus, everything except for this range of ports can be firewalled on the server side. While this doesn't eliminate all risk to the server, it decreases it tremendously. See Appendix 1 for more information.
References
An excellent reference on how various internet protocols work and the issues involved in firewalling them can be found in the O'Reilly and Associates book, Building Internet Firewalls, 2nd Ed, by Brent Chapman and Elizabeth Zwicky.
Finally, the definitive reference on FTP would be RFC 959, which sets forth the official specifications of the FTP protocol. RFCs can be downloaded from numerous locations, including http://www.faqs.org/rfcs/rfc959.html.
Reference:
http://slacksite.com/other/ftp.html
Contents:
Introduction
The Basics
Active FTP
Active FTP Example
Passive FTP
Passive FTP Example
Other Notes
Summary
References
Appendix 1: Configuration of Common FTP Servers
Introduction
One of the most commonly seen questions when dealing with firewalls and other Internet connectivity issues is the difference between active and passive FTP and how best to support either or both of them. Hopefully the following text will help to clear up some of the confusion over how to support FTP in a firewalled environment.
This may not be the definitive explanation, as the title claims, however, I've heard enough good feedback and seen this document linked in enough places to know that quite a few people have found it to be useful. I am always looking for ways to improve things though, and if you find something that is not quite clear or needs more explanation, please let me know! Recent additions to this document include the examples of both active and passive command line FTP sessions. These session examples should help make things a bit clearer. They also provide a nice picture into what goes on behind the scenes during an FTP session. Now, on to the information...
The Basics
FTP is a TCP based service exclusively. There is no UDP component to FTP. FTP is an unusual service in that it utilizes two ports, a 'data' port and a 'command' port (also known as the control port). Traditionally these are port 21 for the command port and port 20 for the data port. The confusion begins however, when we find that depending on the mode, the data port is not always on port 20.
Active FTP
In active mode FTP the client connects from a random unprivileged port (N > 1023) to the FTP server's command port, port 21. Then, the client starts listening to port N+1 and sends the FTP command PORT N+1 to the FTP server. The server will then connect back to the client's specified data port from its local data port, which is port 20.
From the server-side firewall's standpoint, to support active mode FTP the following communication channels need to be opened:
FTP server's port 21 from anywhere (Client initiates connection)
FTP server's port 21 to ports > 1023 (Server responds to client's control port)
FTP server's port 20 to ports > 1023 (Server initiates data connection to client's data port)
FTP server's port 20 from ports > 1023 (Client sends ACKs to server's data port)
When drawn out, the connection appears as follows:
In step 1, the client's command port contacts the server's command port and sends the command PORT 1027. The server then sends an ACK back to the client's command port in step 2. In step 3 the server initiates a connection on its local data port to the data port the client specified earlier. Finally, the client sends an ACK back as shown in step 4.
The main problem with active mode FTP actually falls on the client side. The FTP client doesn't make the actual connection to the data port of the server--it simply tells the server what port it is listening on and the server connects back to the specified port on the client. From the client side firewall this appears to be an outside system initiating a connection to an internal client--something that is usually blocked.
Active FTP Example
Below is an actual example of an active FTP session. The only things that have been changed are the server names, IP addresses, and user names. In this example an FTP session is initiated from testbox1.slacksite.com (192.168.150.80), a linux box running the standard FTP command line client, to testbox2.slacksite.com (192.168.150.90), a linux box running ProFTPd 1.2.2RC2. The debugging (-d) flag is used with the FTP client to show what is going on behind the scenes. Everything in red is the debugging output which shows the actual FTP commands being sent to the server and the responses generated from those commands. Normal server output is shown in black, and user input is in bold.
There are a few interesting things to consider about this dialog. Notice that when the PORT command is issued, it specifies a port on the client (192.168.150.80) system, rather than the server. We will see the opposite behavior when we use passive FTP. While we are on the subject, a quick note about the format of the PORT command. As you can see in the example below it is formatted as a series of six numbers separated by commas. The first four octets are the IP address while the last two octets comprise the port that will be used for the data connection. To find the actual port multiply the fifth octet by 256 and then add the sixth octet to the total. Thus in the example below the port number is ( (14*256) + 178), or 3762. A quick check with netstat should confirm this information.
testbox1: {/home/p-t/slacker/public_html} % ftp -d testbox2
Connected to testbox2.slacksite.com.
220 testbox2.slacksite.com FTP server ready.
Name (testbox2:slacker): slacker
---> USER slacker
331 Password required for slacker.
Password: TmpPass
---> PASS XXXX
230 User slacker logged in.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,150,80,14,178
200 PORT command successful.
---> LIST
150 Opening ASCII mode data connection for file list.
drwx------ 3 slacker users 104 Jul 27 01:45 public_html
226 Transfer complete.
ftp> quit
---> QUIT
221 Goodbye.
Passive FTP
In order to resolve the issue of the server initiating the connection to the client a different method for FTP connections was developed. This was known as passive mode, or PASV, after the command used by the client to tell the server it is in passive mode.
In passive mode FTP the client initiates both connections to the server, solving the problem of firewalls filtering the incoming data port connection to the client from the server. When opening an FTP connection, the client opens two random unprivileged ports locally (N > 1023 and N+1). The first port contacts the server on port 21, but instead of then issuing a PORT command and allowing the server to connect back to its data port, the client will issue the PASV command. The result of this is that the server then opens a random unprivileged port (P > 1023) and sends the PORT P command back to the client. The client then initiates the connection from port N+1 to port P on the server to transfer data.
From the server-side firewall's standpoint, to support passive mode FTP the following communication channels need to be opened:
FTP server's port 21 from anywhere (Client initiates connection)
FTP server's port 21 to ports > 1023 (Server responds to client's control port)
FTP server's ports > 1023 from anywhere (Client initiates data connection to random port specified by server)
FTP server's ports > 1023 to remote ports > 1023 (Server sends ACKs (and data) to client's data port)
When drawn, a passive mode FTP connection looks like this:
In step 1, the client contacts the server on the command port and issues the PASV command. The server then replies in step 2 with PORT 2024, telling the client which port it is listening to for the data connection. In step 3 the client then initiates the data connection from its data port to the specified server data port. Finally, the server sends back an ACK in step 4 to the client's data port.
While passive mode FTP solves many of the problems from the client side, it opens up a whole range of problems on the server side. The biggest issue is the need to allow any remote connection to high numbered ports on the server. Fortunately, many FTP daemons, including the popular WU-FTPD allow the administrator to specify a range of ports which the FTP server will use. See Appendix 1 for more information.
The second issue involves supporting and troubleshooting clients which do (or do not) support passive mode. As an example, the command line FTP utility provided with Solaris does not support passive mode, necessitating a third-party FTP client, such as ncftp.
With the massive popularity of the World Wide Web, many people prefer to use their web browser as an FTP client. Most browsers only support passive mode when accessing ftp:// URLs. This can either be good or bad depending on what the servers and firewalls are configured to support.
Passive FTP Example
Below is an actual example of a passive FTP session. The only things that have been changed are the server names, IP addresses, and user names. In this example an FTP session is initiated from testbox1.slacksite.com (192.168.150.80), a linux box running the standard FTP command line client, to testbox2.slacksite.com (192.168.150.90), a linux box running ProFTPd 1.2.2RC2. The debugging (-d) flag is used with the FTP client to show what is going on behind the scenes. Everything in red is the debugging output which shows the actual FTP commands being sent to the server and the responses generated from those commands. Normal server output is shown in black, and user input is in bold.
Notice the difference in the PORT command in this example as opposed to the active FTP example. Here, we see a port being opened on the server (192.168.150.90) system, rather than the client. See the discussion about the format of the PORT command above, in the Active FTP Example section.
testbox1: {/home/p-t/slacker/public_html} % ftp -d testbox2
Connected to testbox2.slacksite.com.
220 testbox2.slacksite.com FTP server ready.
Name (testbox2:slacker): slacker
---> USER slacker
331 Password required for slacker.
Password: TmpPass
---> PASS XXXX
230 User slacker logged in.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> passive
Passive mode on.
ftp> ls
ftp: setsockopt (ignored): Permission denied
---> PASV
227 Entering Passive Mode (192,168,150,90,195,149).
---> LIST
150 Opening ASCII mode data connection for file list
drwx------ 3 slacker users 104 Jul 27 01:45 public_html
226 Transfer complete.
ftp> quit
---> QUIT
221 Goodbye.
Other Notes
A reader, Maarten Sjouw, pointed out that active FTP will not function when used in conjunction with a client-side NAT (Network Address Translation) device which is not smart enough to alter the IP address info in FTP packets.
Summary
The following chart should help admins remember how each FTP mode works:
Active FTP :
command : client >1023 -> server 21
data : client >1023 <- server 20
Passive FTP :
command : client >1023 -> server 21
data : client >1023 -> server >1023
A quick summary of the pros and cons of active vs. passive FTP is also in order:
Active FTP is beneficial to the FTP server admin, but detrimental to the client side admin. The FTP server attempts to make connections to random high ports on the client, which would almost certainly be blocked by a firewall on the client side. Passive FTP is beneficial to the client, but detrimental to the FTP server admin. The client will make both connections to the server, but one of them will be to a random high port, which would almost certainly be blocked by a firewall on the server side.
Luckily, there is somewhat of a compromise. Since admins running FTP servers will need to make their servers accessible to the greatest number of clients, they will almost certainly need to support passive FTP. The exposure of high level ports on the server can be minimized by specifying a limited port range for the FTP server to use. Thus, everything except for this range of ports can be firewalled on the server side. While this doesn't eliminate all risk to the server, it decreases it tremendously. See Appendix 1 for more information.
References
An excellent reference on how various internet protocols work and the issues involved in firewalling them can be found in the O'Reilly and Associates book, Building Internet Firewalls, 2nd Ed, by Brent Chapman and Elizabeth Zwicky.
Finally, the definitive reference on FTP would be RFC 959, which sets forth the official specifications of the FTP protocol. RFCs can be downloaded from numerous locations, including http://www.faqs.org/rfcs/rfc959.html.
Reference:
http://slacksite.com/other/ftp.html
Subscribe to:
Posts (Atom)