Jails, sometimes referred to as an enhanced replacement of chroot environments, are a very powerful tool for system administrators, but their basic usage can also be useful for advanced users.
Setting up the Host Environment
First, you will want to set up your real system's environment to be "jail-friendly".
For consistency, we will refer to the parent box as the "host environment", and to the jailed virtual machine as the "jail environment". Since jail is implemented using IP aliases, one of the first things to do is to disable IP services on the host system that listen on all local IP addresses for a service. If a network service is present in the host environment that binds all available IP addresses rather than specific IP addresses, it may service requests sent to jail IP addresses if the jail did not bind the port. This means changing inetd(8) to only listen on the appropriate IP address, and so forth. Add the following to /etc/rc.conf in the host environment:
sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"
man inetd for more information.
192.168.0.1 is the native IP address for the host system, in this example. Daemons that run out of inetd(8) can be easily set to use only the specified host IP address. Other daemons will need to be manually configured for some this is possible through the rc.conf(5) flags entries; for others it is necessary to modify per-application configuration files, or to recompile the applications. The following frequently deployed services must have their individual configuration files modified to limit the application to listening to a specific IP address:
To configure sshd(8), it is necessary to modify /etc/ssh/sshd_config.
To configure sendmail(8), it is necessary to modify
/etc/mail/sendmail.cf.
For named(8), it is necessary to modify /etc/namedb/named.conf.
In addition, a number of services must be recompiled in order to run them in the host environment. This includes most applications providing services using rpc(3), such as rpcbind(8), nfsd(8), and mountd(8). In general, applications for which it is not possible to specify which IP address to bind should not be run in the host environment unless they should also service requests sent to jail IP addresses. Attempting to serve NFS from the host environment may also cause confusion, and cannot be easily reconfigured to use only specific IPs, as some NFS services are hosted directly from the kernel. Any third-party network software running in the host environment should also be checked and configured so that it does not bind all IP addresses, which would result in those services' also appearing to be offered by the jail environments.
Once these daemons have been disabled or fixed in the host environment, it is best to reboot so that all daemons are in a known state, to reduce the potential for confusion later (such as finding that when you send mail to a jail, and its sendmail is down, the mail is delivered to the host, etc.).
Getting services to not listen to *
First off, we should make sure we get the system so that we have nothing listening on *, to check what what we need to modify issue this command
# sockstat | grep "\*:[0-9]"
This should give you a synopsys of all the processes and ports you need to trim down. Here are some hints with your ipv4 addr being 10.0.0.1 and your ipv6 addr being 2002::7ea9
# vi /etc/rc.conf
sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"
sshd:
# vim /etc/ssh/sshd_config
change ListenAddress derivative
ListenAddress 192.168.0.1
ListenAddress 192.168.0.111
ListenAddress 2002::7ea9
syslogd:
# vim /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"
rsyncd:
# vim /usr/local/etc/rsyncd.conf
address = 192.168.0.1
MySQL:
# vim /etc/my.cnf
bind-address=10.0.0.1
Samba: (this will get you most of the way there)
edit /usr/local/etc/smb.conf
change the following:
interfaces = 10.0.0.242/24 127.0.0.1
socket address = 10.0.0.242
bind interfaces only = yes
note: if you don't need wins lookups and netbios name translation
you can safely disable nmbd. There doesn't seem to be a way
for nmb to not listen to *:138 anyhow.
To disable nmb go to /etc/rc.conf and replace samba_enable="YES" with smbd_enable="YES"
open ntpd (xntpd listens on all and cannot be changed)
# vim /usr/local/etc/ntpd.conf
listen on 10.0.0.1
listen on 2002::7ea9
bind
edit your named.conf (may be in /var/named/etc/named.conf)
In the options section:
listen-on { 10.0.0.242; };
listen-on-v6 port 53 { 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9; };
query-source address 10.0.0.242 port *;
query-source-v6 address 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9 port *;
The file system layout is described in the following list:
- Each jail will be mounted under the /home/jail/THE_JAIL_NAME directory.
- /home/jail/read-only/root # is the template for each jail and the read-only partition for all of the jails.
- A blank directory will be created for each jail under the /home/jail directory.
- Each jail shall have its own read-write system that is based upon /home/jail/read-only/skel.
- Each jailspace (read-write portion of each jail) shall be created in /home/jail/read-write/THE_JAIL_NAME.
- Each jail will have a /s directory, that will be linked to the read-write portion of the system.
Creating the Template
This section will describe the steps needed to create the master template that will be the read-only portion for the jails to use.
It is always a good idea to update the FreeBSD system to the latest -RELEASE branch. Check the corresponding Handbook Chapter to accomplish this task. In the case the update is not feasible, the buildworld will be required in order to be able to proceed. Additionally, the sysutils/cpdup package will be required. We will use the portsnap(8) utility to download the FreeBSD Ports Collection. The Handbook Portsnap Chapter is always good reading for newcomers.
Install cpdup
# cd /usr/ports/sysutils/cpdup ; make install clean
Create directories
# mkdir /home/jail
# mkdir /home/jail/read-only
# mkdir /home/jail/read-write
Create a directory structure for the read-only file system which will contain the FreeBSD binaries for our jails:
# mkdir /home/jail/read-only/root
Change directory to the FreeBSD source tree and install the read-only file system to the jail template:
# cd /usr/src
If you have already rebuilt your userland using make world or make buildworld, you can skip following step and install your existing userland into the new jail.
# make buildworld
This command will populate the directory subtree chosen as jail's physical location on the file system with the necessary binaries, libraries, manual pages and so on.
# make installworld DESTDIR=/home/jail/read-only/root
The distribution target for make installs every needed configuration file. In simple words, it installs every installable file of /usr/src/etc/ to the /etc directory of the jail environment: /home/jail/THE_JAIL_NAME/etc/.
# make distribution DESTDIR=/home/jail/read-only/root
Next, prepare a FreeBSD Ports Collection for the jails as well as a FreeBSD source tree, which is required for mergemaster:
# mkdir /home/jail/read-only/root/usr/ports
# portsnap -p /home/jail/read-only/root/usr/ports fetch extract
# portsnap -p /home/jail/read-only/root/usr/ports fetch update
# cpdup /usr/src /home/jail/read-only/root/usr/src
Setup for Drupal:
# cd /home/jail/read-only/root
# mkdir www
# cd www
# fetch http://ftp.drupal.org/files/projects/drupal-6.22.tar.gz
# tar zxvf drupal-6.22.tar.gz
# mv drupal-6.22 drupal6
# chown -R root:wheel drupal6
Create a skeleton for the read-write portion of the system:
# mkdir /home/jail/read-only/skel
# mkdir /home/jail/read-only/skel/home
# mkdir /home/jail/read-only/skel/usr-X11R6
# mkdir /home/jail/read-only/skel/distfiles
??? # mkdir /home/jail/read-only/skel/drupal-sites
??? # mkdir /home/jail/read-only/skel/drupal-files
# cd /home/jail/read-only/root
# mv etc /home/jail/read-only/skel
# mv usr/local /home/jail/read-only/skel/usr-local
# mv tmp /home/jail/read-only/skel
# mv var /home/jail/read-only/skel
# mv root /home/jail/read-only/skel
# mv www/drupal6/sites /home/jail/read-only/skel/drupal-sites
Use mergemaster to install missing configuration files. Then get rid of the extra directories that mergemaster creates:
# mergemaster -t /home/jail/read-only/skel/var/tmp/temproot -D /home/jail/read-only/skel -i
How should I handle ./.cshrc? [Leave it to install later] enter
How should I handle ./.profile? [Leave it to install later] enter
...
keep pressing space until you see end : q
*** You installed a login.conf file, so make sure that you run
'/usr/bin/cap_mkdb /home/jail/read-only/skel/etc/login.conf'
to rebuild your login.conf database
Would you like to run it now? y or n [n] y
*** You installed a services file, so make sure that you run
'/usr/sbin/services_mkdb -q -o /home/jail/read-only/skel/var/db/services.db /home/jail/read-only/skel/etc/services'
to rebuild your services database
Would you like to run it now? y or n [n] y
*** You installed a new master.passwd file, so make sure that you run
'/usr/sbin/pwd_mkdb -d /home/jail/read-only/skel/etc -p /home/jail/read-only/skel/etc/master.passwd'
to rebuild your password files
Would you like to run it now? y or n [n] y
# cd /home/jail/read-only/skel
# rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev
Now, symlink the read-write file system to the read-only file system. Please make sure that the symlinks are created in the correct s/ locations. Real directories or the creation of directories in the wrong locations will cause the installation to fail.
# cd /home/jail/read-only/root
# mkdir s
# ln -s s/etc etc
# ln -s s/home home
# ln -s s/root root
# ln -s ../s/usr-local usr/local
# ln -s ../s/usr-X11R6 usr/X11R6
# ln -s ../../s/distfiles usr/ports/distfiles
# ln -s s/tmp tmp
# ln -s s/var var
# ln -s s/home/cwn_www/nginx www
# ln -s s/home/cwn_www/scripts scripts
??? # ln -s ../../s/drupal-sites www/drupal6/sites
??? # ln -s ../../../../s/drupal-files www/drupal6/sites/default/files
??? Changing the directory permission
??? # find /home/jail -type d -print0 | xargs -0 -I @ sh -c 'chmod 700 @ ; chown root:wheel @'
As a last step, create a generic /home/jail/read-only/skel/etc/make.conf with its contents as shown below:
# echo 'WRKDIRPREFIX?= /s/portbuild' >> /home/jail/read-only/skel/etc/make.conf
Having WRKDIRPREFIX set up this way will make it possible to compile FreeBSD ports inside each jail. Remember that the ports directory is part of the read-only system. The custom path for WRKDIRPREFIX allows builds to be done in the read-write portion of every jail.
Creating Jails:
Now that we have a complete FreeBSD jail template, we can setup and configure the jails in /etc/rc.conf. This example demonstrates the creation of 3 jails: "NS", "MAIL" and "WWW".
Put the following lines into the /etc/fstab file, so that the read-only template for the jails and the read-write space will be available in the respective jails:
# vim /etc/fstab
### read-only partitions
/home/jail/read-only/root /home/jail/read-only/ns nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/mail nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/www nullfs ro 0 0
### read-write partitions
/home/jail/read-write/ns /home/jail/read-only/ns/s nullfs rw 0 0
/home/jail/read-write/mail /home/jail/read-only/mail/s nullfs rw 0 0
/home/jail/read-write/www /home/jail/read-only/www/s nullfs rw 0 0
Note: Partitions marked with a 0 pass number are not checked by fsck(8) during boot, and partitions marked with a 0 dump number are not backed up by dump(8). We do not want fsck to check nullfs mounts or dump to back up the read-only nullfs mounts of the jails. This is why they are marked with "0 0" in the last two columns of each fstab entry above.
# vim /etc/rc.conf
### for jail host
ifconfig_re0="inet 192.168.0.1 netmask 255.255.255.0"
### for jail guests
ifconfig_re0_alias0="inet 192.168.0.2 netmask 255.255.255.255"
ifconfig_re0_alias1="inet 192.168.0.3 netmask 255.255.255.255"
ifconfig_re0_alias2="inet 192.168.0.4 netmask 255.255.255.255"
### jail settings
jail_enable="YES" # Set to NO to disable starting of any jails
jail_set_hostname_allow="NO" # Allow root user in a jail to change its hostname
#jail_socket_unixiproute_only="YES" # Route only TCP/IP within a jail
jail_list="ns mail www" # Space separated list of names of jails
# Note: Jail names in jail_list should contain alphanumeric characters only.
### j-ns
jail_ns_hostname="j-ns.local" # jail's hostname
jail_ns_ip="192.168.0.2" # jail's IP address
jail_ns_rootdir="/home/jail/read-only/ns" # jail's root directory
jail_ns_devfs_enable="YES" # mount devfs in the jail
jail_ns_devfs_ruleset="ns_ruleset" # devfs ruleset to apply to jail
### j-mail
jail_mail_hostname="j-mail.local" # jail's hostname
jail_mail_ip="192.168.0.3" # jail's IP address
jail_mail_rootdir="/home/jail/read-only/mail" # jail's root directory
jail_mail_devfs_enable="YES" # mount devfs in the jail
jail_mail_devfs_ruleset="mail_ruleset" # devfs ruleset to apply to jail
### j-www
jail_www_hostname="j-www.local" # jail's hostname
jail_www_ip="192.168.0.4" # jail's IP address
jail_www_rootdir="/home/jail/read-only/www" # jail's root directory
jail_www_devfs_enable="YES" # mount devfs in the jail
jail_www_devfs_ruleset="www_ruleset" # devfs ruleset to apply to jail
Note: The jail_name_rootdir variable must not be set to a path which includes a symbolic link, otherwise the jails will refuse to start. Use the realpath(1) utility to determine a value which should be set to this variable. Please see the FreeBSD-SA-07:01.jail Security Advisory for more information.
Note: for devfs ruleset, cat /etc/defaults/devfs.rules.
Make sure your jail host has all the IP addresses for jail guests:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# ifconfig
Create the required mount points for the read-only file system of each jail:
# cd /home/jail/read-only
# mkdir ns mail www
??? # chmod 700 ns mail www
??? # chown root:wheel ns mail www
Install the read-write template into each jail. Note the use of sysutils/cpdup, which helps to ensure that a correct copy is done of each directory:
# cpdup /home/jail/read-only/skel /home/jail/read-write/ns
# cpdup /home/jail/read-only/skel /home/jail/read-write/mail
# cpdup /home/jail/read-only/skel /home/jail/read-write/www
In this phase, the jails are built and prepared to run. First, mount the required file systems for each jail, and then start them using the /etc/rc.d/jail script:
# mount -a
# df -h
Start a specific jail:
# /etc/rc.d/jail start ns
# /etc/rc.d/jail start mail
# /etc/rc.d/jail start www
Start all jails:
# /etc/rc.d/jail start
Stop a specific jail:
# /etc/rc.d/jail stop www
A clean way to shut down a jail(8) is not available at the moment. This is because commands normally used to accomplish a clean system shutdown cannot be used inside a jail. The best way to shut down a jail is to run the following command from within the jail itself or using the jexec(8) utility from outside the jail:
j-www # sh /etc/rc.shutdown
or
# jexec 2 shutdown -p now
The jails should be running now. To check if they have started correctly, use the jls(8) command. Its output should be similar to the following:
# jls
JID IP Address Hostname Path
7 192.168.0.2 j-ns /home/jail/read-only/ns
8 192.168.0.3 j-mail /home/jail/read-only/mail
9 192.168.0.4 j-www /home/jail/read-only/www
At this point, it should be possible to log onto each jail, add new users or configure daemons. The JID column indicates the jail identification number of each running jail. Use the following command in order to perform administrative tasks in the jail whose JID is 9:
# jexec 9 tcsh
Use google's public DNS server as a DNS resolver:
j-www # vi /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
Modify /etc/rc.conf:
j-www # vi /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"
### SendMail
sendmail_enable="NO"
Modify /etc/hosts:
j-www # vim /etc/hosts
::1 localhost localhost.j-www.local
127.0.0.1 localhost localhost.j-www.local
192.168.100.34 j-www.local j-www
To fix this error "hash map "Alias0": missing map file /etc/mail/aliases.db in FreeBSD jail":
# sendmail -bi
# ls -l /etc/mail/aliases.db
Note: you must edit /etc/hosts before running sendmail -bi command.
Comment out adjkerntz in crontab:
j-www # vim /etc/crontab
#1,31 0-5 * * * root adjkerntz -a
SVN server setup
# cd /usr/ports/devel/subversion
# make install clean
Make sure you DO uncheck this option:
[] BDB=off "db4 repository backend"
# mv /usr/local/bin/svn /usr/local/bin/svn.orig
# vi /usr/local/bin/svn
#!/bin/sh
### initialize
svnarg=""
### use encoding utf-8 as default if run "svn ci" or "svn commit".
if [ "$1" != "help" ]; then
for myarg in "$@"; do
if [ "${myarg}" = "commit" ] || [ "${myarg}" = "ci" ]; then
svnarg="--encoding utf-8"
break
fi
done
fi
### wrapper script to set umask to 027 on subversion binaries
### Note: the meaning of each umask:
### umask 002 // File permission 644. Owner can read/write. Group and Others can only read.
### umask 007 // File permission 660. Owner and Group can read/write. Others can not read or write.
### umask 027 // File permission 640. Owner can read/write. Group can read. Others can not read or write.
umask 027
### svn command
/usr/local/bin/svn.orig ${svnarg} "$@"
Don't forget to make your wrapper script executable:
# chmod +x /usr/local/bin/svn
# mkdir /home/jail/svn-rep
# cd /home/jail/svn-rep
# svnadmin create ns
# svnadmin create mail
# svnadmin create www
# vi www/conf/svnserve.conf
[general]
anon-access = none
auth-access = write
password-db = passwd
# vi www/conf/passwd
[users]
danny = mypassword
Start SVN server as a stand-alone daemon
# /usr/local/bin/svnserve -d --listen-port=3690 --listen-host=0.0.0.0 -r /home/jail/svn-rep
# vi /etc/rc.conf
### SVN
svnserve_enable="YES"
svnserve_flags="-d --listen-port=3690 --listen-host=0.0.0.0"
svnserve_data="/home/jail/svn-rep"
svnserve_user="root"
svnserve_group="wheel"
# svn import /home/jail/read-only/skel/drupal-sites/all file:///home/jail/svn-rep/www -m "initial import"
SVN client setup
# rm -r /www/drupal6/sites/all/*
# svn checkout svn://192.168.0.1/www /www/drupal6/sites/all
sending mail through sendmail inside a jail:
# cat mail.php
<?php
mail('test@example.com', 'My Subject', 'my msg');
?>
# php mail.php
collect: Cannot write ./dfpA4NVE5H097195 (bfcommit, uid=25, gid=25): Permission denied
queueup: cannot create queue file ./qfpA4NVE5H097195, euid=25, fd=-1, fp=0x0: Permission denied
# chown smmsp:smmsp /var/spool/clientmqueue
# chmod 770 /var/spool/clientmqueue
Fine Tuning and Administration
There are several options which can be set for any jail, and various ways of combining a host FreeBSD system with jails, to produce higher level applications. This section presents:
Some of the options available for tuning the behavior and security restrictions implemented by a jail installation.
Some of the high-level applications for jail management, which are available through the FreeBSD Ports Collection, and can be used to implement overall jail-based solutions.
Fine tuning of a jail's configuration is mostly done by setting sysctl(8) variables. A special subtree of sysctl exists as a basis for organizing all the relevant options: the security.jail.* hierarchy of FreeBSD kernel options. Here is a list of the main jail-related sysctls, complete with their default value. Names should be self-explanatory, but for more information about them, please refer to the jail(8) and sysctl(8) manual pages.
security.jail.set_hostname_allowed: 1
security.jail.socket_unixiproute_only: 1
security.jail.sysvipc_allowed: 0
security.jail.enforce_statfs: 2
security.jail.allow_raw_sockets: 0
security.jail.chflags_allowed: 0
security.jail.jailed: 0
These variables can be used by the system administrator of the host system to add or remove some of the limitations imposed by default on the root user. Note that there are some limitations which cannot be removed. The root user is not allowed to mount or unmount file systems from within a jail(8). The root inside a jail may not load or unload devfs(8) rulesets, set firewall rules, or do many other administrative tasks which require modifications of in-kernel data, such as setting the securelevel of the kernel.
Ping from jail not permitted error
ICMP is disallowed by defaut for jails, see the sysctl :
# echo security.jail.allow_raw_sockets=1 >> /etc/sysctl.conf
Now restart your jails and the problem should be fixed.
There are good reasons for this default, so if you test remember to set it back when you are done.
Also, on a point of style, jails in their current form (see VIMAGE) do not get a network stack of their own so they don't have a firewall but share the hosts' network and firewall, etc.
Cannot remove files inside a jail directory
override r-sr-xr-x root/wheel schg for j/mroot/bin/rcp? y
rm: j/mroot/bin/rcp: Operation not permitted
rm: j/mroot/bin: Directory not empty
override r--r--r-- root/wheel schg for j/mroot/lib/libc.so.7? y
rm: j/mroot/lib/libc.so.7: Operation not permitted
# chflags -R noschg /home/jail/read-write/ns
# rm -r /home/jail/read-write/ns
High-level administrative tools in FreeBSD Ports Collection
# cd /usr/ports/sysutils/jailutils ; make install clean
# cd /usr/ports/sysutils/ezjail ; make install clean
Reference:
http://gala4th.blogspot.com/2012/01/install-jails-on-freebsd-82.html
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
VIMAGE - Better virtualization in FreeBSD 8
http://gala4th.blogspot.com/2011/10/vimage-better-virtualization-in-freebsd.html
FreeBSD 8 VIMAGE + epair howto
http://gala4th.blogspot.com/2011/10/freebsd-8-vimage-epair-howto.html
Creating a FreeBSD Jail
http://www.section6.net/wiki/index.php/Creating_a_FreeBSD_Jail
No comments:
Post a Comment