Network statistics and archaeology with SiLK

This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

Have you ever had the need to take a rear-view mirror look at your network traffic, for instance for finding out which of your servers were contacted by some remote IP address a week ago? Or to establish whether or not a recently compromised server was able to further distribute its malware? Perhaps you need to find out which single IP address congested your network yesterday evening? How about finding each day’s bandwidth usage for the VoIP telephone system? Has any system been contacted by Chinese IP addresses?

Netflow

Enter NetFlow. NetFlow makes shallow information about data traffic available, registering general information and some meta data about every connection passing a router or other network device. With proper analysis tools, all the questions above can be easily answered. The bad news is that you will have had to install it before you need the data, since we’re talking about a network statistics tool and not a retroactive crystal ball. The good news is that storing NetFlow data does not take up a lot of disk space, and does not incur any significant load on your network devices, so this should be feasible on any site.

Enterprise network equipment will usually support some kind of traffic flow export format, be it NetFlow (the original Cisco format), Rflow, sFlow, or IPFIX. For smaller office and home networks, traffic flow export can be configured for router software like OpenWRT (by installing and configuring the softflowd package) and DD-WRT (by enabling the RFlow service in the web interface). There’s even an iptables module named ipt-netflow if you want to build your own firewall or simply wish to support traffic flow export on a regular server.

SiLK

In addition to configuring the network device(s), a system must be set up to receive the traffic flow data. On Linux and Unix systems, our NetFlow toolbox includes projects like Argus and NFSen, as well as the fairly recent SiLK (“System for Internet-Level Knowledge”) which will be the tool presented in this article.

By identifying your own IP networks in a configuration file, SiLK will have a notion of inbound and outbound traffic. GeoIP can also be implemented, making it easy to identify network connections by country.

Installation

SiLK is a suite of command-line tools for processing records created by the SiLK packing system. Since it’s a command-line tool, it can easily be scripted, for instance to produce daily usage reports or for giving alerts on weird network behaviour.

Unfortunately, you cannot install SiLK through prepackaged rpm or deb packages. Fortunately, the suite has detailed installation instructions. In short, after satisfying dependency requirements, the suite can be installed by unpacking the source, then run configure, make and make install:

tar -xzvf silk-3.11.0.1.tar.gz
cd silk-3.11.0.1
./configure
make
sudo make install

Configuration

SiLK can run in “live mode” receiving live feeds from network components, or it can be fed different types of packet capture files. For receiving live feeds, optimal sources include edge routers, core routers, VPN tunnel terminators etc. For this kind of setup you will also need to make sure rwflowcapd, the piece of software that accepts the feeds and stores the data, is running. The flow sources are configured in SiLK’s configuration files silk.conf and sensors.conf. Below are some extracts from Redpill Linpro’s installation:

silk.conf

sensor 0 cr "Core router"
sensor 1 cs "Core switch"
class all
  sensors cr cs
end class

sensors.conf

probe cr netflow-v5
  listen-on-port 4739
  protocol udp
  accept-from-host 87.238.a.b
end probe

probe cs sflow
  listen-on-port 6343
  protocol udp
  accept-from-host 87.238.c.d
end probe

group my-network
  ipblocks 87.238.32.0/19 # RL
  ipblocks 2a02:c0::/32  # RL
  ipblocks 10.0.0.0/8     # RFC1918
  ipblocks 172.16.0.0/12  # RFC1918
  ipblocks 192.168.0.0/16 # RFC1918
end group

sensor cr
netflow-v5-probes cr
  internal-ipblock @my-network
  external-ipblock remainder
end sensor

sensor cs
  sflow-probes cs
  internal-ipblock @my-network
  external-ipblock remainder
end sensor

Examples

Below are some examples to illustrate the power of SiLK. The IP addresses have been changed to protect the innocent.

Example: Identify inbound FTP connection attempts. Group by number of attempts, and list up to 20 attempts ordered by number of connections.

$ rwfilter --type=in --syn=1 --ack=0 --fin=0 --dport=21 \
  --pass=stdout | rwstats --top --count=20 \
  --fields=sip,dip,dport --value=flows

INPUT: 29 Records for 13 Bins and 29 Total Records
OUTPUT: Top 20 Bins by Records

            sIP|                dIP|dPort|   Records|  %Records|   cumul_%|
     aaa.bb.d.5|      87.238.aa.bbb|   21|        10| 34.482759| 34.482759|
 ccc.ddd.eee.50|       87.238.dd.ff|   21|         4| 13.793103| 48.275862|
   fff.gg.hhh.4|      87.238.ee.aaa|   21|         2|  6.896552| 55.172414|
 aa.fff.nnn.118|       87.238.dd.bb|   21|         2|  6.896552| 62.068966|
 ff.eee.mmm.128|       87.238.aa.cc|   21|         2|  6.896552| 68.965517|
  ccc.eee.aaa.6|       87.238.gg.dd|   21|         2|  6.896552| 75.862069|
    dd.eee.cc.8|      87.238.ff.ccc|   21|         1|  3.448276| 79.310345|
  ee.aa.ccc.175|       87.238.cc.ff|   21|         1|  3.448276| 82.758621|
  aaa.cc.ddd.23|       87.238.hh.aa|   21|         1|  3.448276| 86.206897|
   dd.sss.zz.70|       87.238.bb.dd|   21|         1|  3.448276| 89.655172|
 hh.ccc.ddd.118|       87.238.nn.mm|   21|         1|  3.448276| 93.103448|
hhh.aaa.aaa.116|      87.238.mm.ddd|   21|         1|  3.448276| 96.551724|
 dd.ggg.fff.246|       87.238.aa.ee|   21|         1|  3.448276|100.000000|

Example: Identify all servers that do outbound DNS request. Group the IP addresses by number or occurrences. The purpose is finding systems that are not configured to use our own DNS resolvers.

$ rwfilter --dport=53 --type=out --pass=stdout | \
  rwuniq --fields=sip --dip-distinct

             sIP|dIP-Distin|
    87.238.aa.bb|         1|
   87.238.ff.hhh|       976|
    87.238.dd.aa|         2|
    87.238.ee.dd|         8|
   87.238.gg.ccc|        63|
    87.238.nn.ff|         1|
    87.238.mm.aa|         3|
   [...]

Example: Find the top 10 busiest server protocols below port 1024.

$ rwfilter --proto=6,17 --type=out,outweb --sport=1-1023 \
  --pass=stdout  | rwstats --fields=sport --count=10 --bytes

INPUT: 6947933 Records for 118 Bins and 8833031076 Total Bytes
OUTPUT: Top 10 Bins by Bytes

sPort|       Bytes|    %Bytes|   cumul_%|
   80|  8104989791| 91.757741| 91.757741|
  443|   720068914|  8.152002| 99.909744|
   53|     3621571|  0.041000| 99.950744|
  873|     1294730|  0.014658| 99.965402|
  993|      957669|  0.010842| 99.976244|
   22|      914727|  0.010356| 99.986599|
  999|      308347|  0.003491| 99.990090|
  444|      199635|  0.002260| 99.992350|
   25|      184706|  0.002091| 99.994441|
   21|      142686|  0.001615| 99.996057|

Example: Find listening servers in a network, but filter out source port 80. Note that the output from rwfilter is fed into a new rwfilter construction, where we negate port 80 using --fail=stdout.

$ rwfilter --proto=6,17 --ip-version=4 --type=out,outweb \
  --scidr=87.238.aa.0/27 --pass=stdout --flags-all=SA/SA | \
  rwfilter --sport=80 --fail=stdout stdin | \
  rwuniq --fields dip,sip,sport

            dIP|              sIP|sPort|   Records|
 aaa.dd.203.237|     87.238.aa.bb|  443|         1|
   bb.cc.172.34|     87.238.aa.bb|  443|         1|
   c.bb.109.234|     87.238.aa.bb|  443|         1|
 dd.aaa.199.102|     87.238.aa.bb|  443|        21|
    aa.dd.183.2|     87.238.aa.bb|  443|         1|
 bb.ccc.160.162|     87.238.aa.bb|  443|         4|