Tuesday, June 4, 2013

Monitor network bandwidth usage with NetFlow, flow-tools, MySQL on FreeBSD

nfdump
nfsen

ng_netflow - a NetGraph-based kernel module for FreeBSD.
flow-tools

pfflowd
softflowd

sFlow - Eric Chou: NFSen 和 NFDump 都是好軟體，謝謝介紹。順帶一提的是有一些產品已不支持 NetFlow v5，例如 Cisco ASA (8.0 後) 還有就是我以前在安裝時曾浪費了一些時間抓 Bug 結果發現 ASA 根本需要用不同的版本，NetFlow Security Event (NSEL)。在高傳送或非 Cisco 的環境下，有時 sFlow 也是一種選擇。

ipcad - is an IP accounting daemon. It uses bpf or pcap to access interfaces and gather IP statistics. Collected numbers are arranged to form an address-to-address flow pairs and than can be accessed via rsh in Cisco fashion, or exported via NetFlow UDP protocol.

fprobe - a NetFlow probe - libpcap-based tool that collects network traffic data and emit it as NetFlow flows towards the specified collector.

Wireshark - Wireshark is a free and open-source packet analyzer.
http://www.wireshark.org/

Adventnet Netflow Analyzer
http://www.manageengine.com/products/netflow

Using "NetFlow" requires:

  • Sensor: netflow export from your network device(s) - e.g. on Cisco IOS "ip flow-export destination x.x.x.x yyyy"
  • Collector: a netflow collector daemon/application to stick the exported flow info into a database
  • Analyzer/Cruncher/Reporter: an analysis tool to report on the netflow information collected.

RusDyr: I usually recommend ng_netflow + flow-tools. I use it on regular basis in ISP's environment.

http://www.gorlani.com/portal/articles/gathering-netflow-data-with-a-freebsd-server
http://forums.freebsd.org/showthread.php?t=31256

NetFlow Packet transport protocol

NetFlow records are traditionally exported using User Datagram Protocol (UDP) and collected using a NetFlow collector. The IP address of the NetFlow collector and the destination UDP port must be configured on the sending router. The standard value is UDP port 2055, but other values like 9555 or 9995 are often used.

For efficiency reasons, the router traditionally does not keep track of flow records already exported, so if a NetFlow packet is dropped due to network congestion or packet corruption, all contained records are lost forever. The UDP protocol does not inform the router of the loss so it can send the packets again. This can be a real problem, especially with NetFlow v8 or v9 that can aggregate a lot of packets or flows into a single record. A single UDP packet loss can cause a huge impact on the statistics of some flows.

That is why some modern implementations of NetFlow use the Stream Control Transmission Protocol (SCTP) to export packets so as to provide some protection against packet loss, and make sure that NetFlow v9 templates are received before any related record is exported. Note that TCP would not be suitable for NetFlow because a strict ordering of packets would cause excessive buffering and delays.

The problem with SCTP is that it requires interaction between each NetFlow collector and each routers exporting NetFlow. There may be performance limitations if a router has to deal with many NetFlow collectors, and a NetFlow collector has to deal with lots of routers, especially when some of them are unavailable due to failure or maintenance.

SCTP may not be efficient if NetFlow must be exported toward several independent collectors, some of which may be test servers that can go down at any moment. UDP allows simple replication of NetFlow packets using Network taps or L2 or L3 Mirroring. Simple stateless equipment can also filter or change the destination address of NetFlow UDP packets if necessary. Since NetFlow export almost only use network backbone links, packet loss will often be negligible. If it happens, it will mostly be on the link between the network and the NetFlow collectors.

On Sensor machine, make sure netgraph is supported:
# ls /boot/kernel/netgraph*
# ls /boot/kernel/ng_ether*
# ls /boot/kernel/ng_one2many*

On Sensor machine:
# vim /boot/loader.conf
ng_ether_load="YES"
ng_one2many_load="YES"

Reboot the system to ensure the kernel modules are loaded:

# sync;sync;reboot
or
# kldload /boot/kernel/ng_ether.ko
# kldload /boot/kernel/ng_one2many.ko

Make sure the kernel modules are loaded:
# kldstat
Id Refs Address Size Name
1 7 0xffffffff80200000 1323388 kernel
2 1 0xffffffff81524000 45c8 ng_ether.ko
3 3 0xffffffff81529000 15330 netgraph.ko
4 1 0xffffffff8153f000 2bc0 ng_one2many.ko

On Sensor machine:
# vim /etc/ng_conf
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/192.168.0.130:2055

Note: em0 is the network interface to listen to.
Note: 192.168.0.130:2055 is the collector's IP address and its port number.

On Sensor machine:
# /usr/sbin/ngctl -f /etc/ng_conf

# sockstat | grep 2055

Sensing the Sensor

It is also useful to determine that the sensor data is reaching your collector's network interface before installing a collector. A simple tcpdump invocation should be sufficient to let you see whether traffic is coming from your sensor's IP address, to the collector's IP address at the specified port.

Run tcpdump on Collector:
# tcpdump -nettti em0 udp and port 2055

Note: you have to wait for few minutes to see the Sensor sending the packets.

Install flow-tools on Collector with MySQL support:
# cd /usr/ports/net-mgmt/flow-tools/
# make config-recursive
# make install clean

MYSQL=on: MySQL database support

# mkdir /var/log/netflows
# chmod 755 /var/log/netflows
# chown root:wheel /var/log/netflows

Note: the configuration files of flow-tools can be found at /usr/local/etc/flow-tools.

Run flow-capture on Collector:
# /usr/local/bin/flow-capture -n 287 -S 5 -w /var/log/netflows/ 0/0/2055

Note: -S stat_interval log a timestamped message every stat_interval minutes indicating counters such as the number of flows received, packets processed, and lost flows.
Note: -n rotations the number of times a new file per day. -n 287 means 288 samples during all day (every 5 minutes).
Note: 0/0/2055 is LocalIP/RemoteIP/port. Use 0 for any IP address.
Note: -w workdir set the workdir to /var/log/netflows/.

Make sure flow-capture is running:
# ps auxww | grep -i flow-capture
# sockstat | grep flow-cap

Make sure the network interface of the Collector is in promiscuous mode:
# ifconfig em0 | grep -i promisc
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

# grep -i promisc /var/log/messages
Jun 1 21:10:21 bsd-netflow kernel: em0: promiscuous mode enabled

If the interface is not in promiscuous mode:
# ifconfig em0 promisc

Log Files
When flow-capture is working correctly, data files will be stored in the specified directory, with data split into date folders, such as:

/var/netflow/sensorXY/YYYY/YYYY-MM/YYYY-MM-DD

the file naming convention for the incremental files are:
- tmp-v05.YYY-MM-DD.HHMMSS+UTCTZ // temporary file
- ft-v05.YY-MM-DD.HHMMSS+UTCTZ // permanent file

flow-capture will generate a tmp-* file once it is running:
# ps auxww | grep -i flow-capture

# find /var/log/netflows -name 'tmp*'
/var/log/netflows/2013/2013-05/2013-05-31/tmp-v05.2013-05-31.224501-0700

Note: file begins with tmp-* is a temporary file, which will be moved to ft-* as a permanent file (every 5 minutes in our setting).
Note: v05 means Netflow version 5.
Note: -0700 means UTC (standard time) -7 hours.

Log messages
flow-capture logs its messages, errors to syslog's /var/log/messages which can be monitored.

Export the netflow data from binary to ASCII text file:
# flow-export -f2 -m0x303000 < ft-v05.2013-05-31.174501-0700 > test.txt

Export the netflow data from binary into MySQL with flow-tools:
Create a new database called "netflow", with following table:

CREATE TABLE `flows` (
  `FLOW_ID` bigint(32) unsigned NOT NULL AUTO_INCREMENT,
  `UNIX_SECS` int(32) unsigned NOT NULL DEFAULT '0',
  `UNIX_NSECS` int(32) unsigned NOT NULL DEFAULT '0',
  `SYSUPTIME` int(20) NOT NULL,
  `EXADDR` varchar(16) NOT NULL,
  `DPKTS` int(32) unsigned NOT NULL DEFAULT '0',
  `DOCTETS` int(32) unsigned NOT NULL DEFAULT '0',
  `FIRST` int(32) unsigned NOT NULL DEFAULT '0',
  `LAST` int(32) unsigned NOT NULL DEFAULT '0',
  `ENGINE_TYPE` int(10) NOT NULL,
  `ENGINE_ID` int(15) NOT NULL,
  `SRCADDR` varchar(16) NOT NULL DEFAULT '0',
  `DSTADDR` varchar(16) NOT NULL DEFAULT '0',
  `NEXTHOP` varchar(16) NOT NULL DEFAULT '0',
  `INPUT` int(16) unsigned NOT NULL DEFAULT '0',
  `OUTPUT` int(16) unsigned NOT NULL DEFAULT '0',
  `SRCPORT` int(16) unsigned NOT NULL DEFAULT '0',
  `DSTPORT` int(16) unsigned NOT NULL DEFAULT '0',
  `PROT` int(8) unsigned NOT NULL DEFAULT '0',
  `TOS` int(2) NOT NULL,
  `TCP_FLAGS` int(8) unsigned NOT NULL DEFAULT '0',
  `SRC_MASK` int(8) unsigned NOT NULL DEFAULT '0',
  `DST_MASK` int(8) unsigned NOT NULL DEFAULT '0',
  `SRC_AS` int(16) unsigned NOT NULL DEFAULT '0',
  `DST_AS` int(16) unsigned NOT NULL DEFAULT '0',
  PRIMARY KEY (`FLOW_ID`),
  KEY `SRCADDR` (`SRCADDR`),
  KEY `DSTADDR` (`DSTADDR`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Create a "rotate program" that will actually enter in the information into mysql:
# cat /etc/myscript/flow-mysql-export.sh
#!/bin/sh
/usr/local/bin/flow-export -f3 -u "username:password:localhost:3306:netflow:flows" < /var/log/netflows/$1

Kill the previous flow-capture process:
# ps auxww | grep -i flow-capture
root 13813 0.0 0.5 22588 9628 ?? Ss 5:39PM 0:00.65 /usr/local/bin/flow-capture -w /var/log/netflows -S 5 0/0/2055

# kill 13813

Run flow-capture with mysql-export enabled:
# /usr/local/bin/flow-capture -n 287 -S 5 -w /var/log/netflows -R /etc/myscript/flow-mysql-export.sh 0/0/2055

Run MySQL Query:
mysql> SELECT * FROM netflow.flows;

Note: IP protocol 1 is for ICMP, 6 is for TCP and 17 is for UDP. Refer to the List of IP protocol numbers.

Print the netflow data in human readable format:
# flow-print -p < /var/log/netflows/2013/2013-06/2013-06-01/ft-v05.2013-06-01.215000-0700 | less

Print the netflow data in human readable format:
# flow-cat /var/log/netflows/2013/2013-06/2013-06-01/ft-v05.2013-06-01.220000-0700 | flow-print -f 5 | less

Note: the octets column, one octet equals to 8 bits (one byte).

Print the netflow data with filter:
# flow-cat ft-v05.2013-06-04.161001-0700 | flow-filter -p 53 | flow-print -f 5 | less

Generate reports from the netflow data:
# flow-cat /var/log/netflows/2013/2013-06/2013-06-01/ft-v05.2013-06-01.214000-0700 | flow-report | less

Install FlowViewer:
# cd /usr/ports/net-mgmt/flowviewer
# make config-recursive
# make install

# cd /usr/local/www/flowviewer
# cp FlowViewer_Configuration.pm.dist FlowViewer_Configuration.pm

You can find additional information in the:
# less /usr/local/share/doc/flowviewer/README

Alternative tool to export the netflow data:
p5-Cflow is a perl module for analyzing raw flow files written by cflowd, a package used to collect Cisco NetFlow data.

# cd /usr/ports/net-mgmt/p5-Cflow/
# make install clean

# flowdumper -s ft-v05.2013-05-31.174151-0700
# flowdumper -v ft-v05.2013-05-31.174151-0700
# flowdumper -V ft-v05.2013-05-31.174151-0700

Install nfsen:
# cd /usr/ports/net-mgmt/nfsen
# make config-recursive

nfdump-1.6.9
check NFTRACK - with PortTracker support.

# make install

/usr/local/etc/nfsen.conf
/usr/local/var/nfsen
/usr/local/libexec/nfsen
/usr/local/www/nfsen

%sources = (
'mysrv' => { 'port' => '2055', 'col' => '#0000ff', 'type' => 'netflow' },
);

Note: mysrv needs to be resolvable to the IP address of the netflow source device (add record in /etc/hosts file).

Note: check directory and file permission!!!

# cd /usr/local/www/apache22/data
# ln -s /usr/local/www/nfsen nfsen

# less /usr/ports/net-mgmt/nfsen/work/nfsen-1.3.6p1/contrib/PortTracker/INSTALL

# mkdir /usr/local/var/nfsen/portdb

# /usr/local/etc/rc.d/nfsen restart

Reference:
http://bbs.chinaunix.net/forum.php?mod=viewthread&tid=667760
http://en.wikipedia.org/wiki/NetFlow
http://lijian366.i.sohu.com/blog/view/170814918.htm
http://meefirst.blogspot.ca/2012/02/installing-nfsen-on-freebsd-9.html
http://www.gorlani.com/portal/articles/gathering-netflow-data-with-a-freebsd-server
http://www.kelvinism.com/2008/12/netflow-into-mysql-with-flow-tools_5439.html
http://www.netadmin.com.tw/article_content.aspx?sn=1111030003
http://www.nomoa.com/bsd/toolkit/monitoring/netflow/collector.html

Posted by Jun Hsieh at 8:53 PM
Labels: ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)