Saturday, July 2, 2016

Dealing with large pcaps ... using netflow

Large PCAP … Netflow to the rescue


There are a lot of times that you have to deal with a large pcap, network troubleshooting, forensics, ... gigabytes of data and millions of packets that can make powerful computers cry and the person who needs to analyze it to get desperate for the times it takes to process it, making this a mission impossible.


In this situation the more important is to try to suppress the unrelated data so we need a quick way to start to filter the noise, and is when netflow comes handy for this, as it was designed to have insights in large networks.


Let’s see an example, there are lot of pcaps out there that you can download but this page has good samples http://www.netresec.com/?page=PcapFiles.


Let’s try to download a large file:


Connecting to download.netresec.com (download.netresec.com)|5.9.12.237|:443... connected.
WARNING: cannot verify download.netresec.com's certificate, issued by '/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3':
 Issued certificate has expired.
HTTP request sent, awaiting response... 206 Partial Content
Length: 331001091 (316M), 260288049 (248M) remaining [application/octet-stream]
Saving to: 'maccdc2012_00000.pcap.gz'


100%[++++++++++++++++++++==========================================================================>] 331,001,091  385KB/s   in 10m 56s


2016-07-02 10:00:36 (387 KB/s) - 'maccdc2012_00000.pcap.gz' saved [331001091/331001091]


after uncompress it we can see the real size, 1G:


root@vagrant-ubuntu-trusty-64:~# gunzip maccdc2012_00000.pcap.gz
root@vagrant-ubuntu-trusty-64:~# ls -altrh
total 1.0G
-rw-r--r--  1 root root 1.0G May  7  2013 maccdc2012_00000.pcap


if we try to use tshark directly with this file we have to give up after a couple of minutes, because we can't obtain any output in a reasanoble time:


Screenshot 2016-07-02 13.09.59.png

To speed up the process we can use softflowd to generate the netflow data and nfdump for the
In ubuntu it’s as easy as:


root@vagrant-ubuntu-trusty-64:~# apt-get -y install softflowd nfdump


we need to launch nfcapd to listen in a port and store the netflows and use softflowd to generate the netflows from the pcap and send to the nfcap process:


root@vagrant-ubuntu-trusty-64:~# nfcapd -D -p 12345 -l .
root@vagrant-ubuntu-trusty-64:~# time softflowd -r maccdc2012_00000.pcap -d -n localhost:12345
softflowd v0.9.9 starting data collection
Exporting flows to [10.0.2.15]:12345


Shutting down after pcap EOF
Shutting down on user request
Number of active flows: 0
Packets processed: 497
Fragments: 0
Ignored packets: 8635446 (8635446 non-IP, 0 too short)
Flows expired: 2 (0 forced)
Flows exported: 2 in 1 packets (0 failures)


Expired flow statistics:  minimum       average       maximum
 Flow bytes:                1450         77789        154128
 Flow packets:                29           248           468
 Duration:               3517.91s      3540.03s      3562.14s


Expired flow reasons:
      tcp =         0   tcp.rst =         0   tcp.fin =         0
      udp =         0      icmp =         0   general =         0
  maxlife =         0
over 2 GiB =         0
 maxflows =         0
  flushed =         2


Per-protocol statistics:     Octets      Packets   Avg Life    Max Life
          igmp (2):           1450           29    3517.91s    3517.91s
          udp (17):         154128          468    3562.14s    3562.14s


real 0m0.697s
user 0m0.501s
sys 0m0.189s


wait, what’s happening Ignored packets: 8635446. Let’s look closer:


root@vagrant-ubuntu-trusty-64:~# tcpdump -evvv -r maccdc2012_00000.pcap | less
reading from file maccdc2012_00000.pcap, link-type EN10MB (Ethernet)
12:30:00.000000 00:16:47:9d:f2:c2 (oui Unknown) > 00:0c:29:41:4b:e7 (oui Unknown), ethertype 802.1Q (0x8100), length 117: vlan 120, p 0,
ethertype IPv4, (tos 0x0, ttl 254, id 36140, offset 0, flags [none], proto TCP (6), length 99)
   192.168.229.254.https > 192.168.202.79.46117: Flags [P.], cksum 0x19da (correct), seq 2162570451:2162570498, ack 4204467708, win 327
68, options [nop,nop,TS val 319644338 ecr 0], length 47


Wow, the packets are tagged and softflowd seems that doesn’t like it. There is a patch for softflowd to process tagged packets, but we will use tcprewrite to strip the vlan header:


root@vagrant-ubuntu-trusty-64:~# tcprewrite --enet-vlan=del --infile=maccdc2012_00000.pcap --outfile=output.pcap


root@vagrant-ubuntu-trusty-64:~# time softflowd -r output.pcap -d -n localhost:12345
softflowd v0.9.9 starting data collection
Exporting flows to [127.0.0.1]:12345
Shutting down after pcap EOF
Shutting down on user request
Number of active flows: 0
Packets processed: 8590643
Fragments: 0
Ignored packets: 45300 (45300 non-IP, 0 too short)
Flows expired: 3973668 (3965235 forced)
Flows exported: 7108230 in 238678 packets (0 failures)


Expired flow statistics:  minimum       average       maximum
 Flow bytes:                  46           196      17027122
 Flow packets:                 1             2         87380
 Duration:                  0.00s         0.12s      2793.81s


Expired flow reasons:
      tcp =         0   tcp.rst =         0   tcp.fin =         0
      udp =         0      icmp =         0   general =         0
  maxlife =         0
over 2 GiB =         0
 maxflows =   3965235
  flushed =      8433


Per-protocol statistics:     Octets      Packets   Avg Life    Max Life
          icmp (1):        3107560        62818       0.39s     774.66s
          igmp (2):           5836          124     174.36s    1130.35s
           tcp (6):      770692525      8481053       0.09s    1207.32s
          udp (17):        2806822        30271       3.06s     769.17s
        eigrp (88):         982620        16377     229.19s    2793.81s


real 0m20.438s
user 0m17.278s
sys 0m1.736s


Ok, this looks better and we have our netflow file:


-rw-r--r--  1 root root 353M Jul  2 11:20 nfcapd.201607021119


that can be used by nfdump to obtain all kind of statistics and filter traffic only in a few seconds:


root@vagrant-ubuntu-trusty-64:~# time nfdump -r nfcapd.201607021119 -s srcip -s ip/flows -s dstport/pps/packets/bytes -s record/bytes
Aggregated flows 7011880
Top 10 flows ordered by bytes:
Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2016-06-04 13:16:52.686     5.810 TCP     192.168.22.254:443   ->   192.168.202.76:52965    20667   16.6 M     1
2016-06-04 13:35:40.586  1246.030 TCP     192.168.28.254:443   ->   192.168.202.76:54282    41674    9.2 M    14
2016-06-04 12:56:26.176    55.200 TCP     192.168.202.79:55173 ->  192.168.229.153:445      43697    8.4 M     1
2016-06-04 13:32:03.176  1470.760 TCP     192.168.202.68:55554 ->   192.168.203.61:36694    39988    6.9 M     5
2016-06-04 13:32:03.176  1470.760 TCP     192.168.203.61:36694 ->   192.168.202.68:55554    39520    6.4 M     5
2016-06-04 13:43:56.646     2.870 TCP     192.168.202.78:80    ->   192.168.203.62:54381     3870    5.8 M     2
2016-06-04 13:34:47.515  1303.391 TCP     192.168.203.45:5432  ->  192.168.202.110:32881    11402    4.7 M    12
2016-06-04 12:56:26.176    55.200 TCP    192.168.229.153:445   ->   192.168.202.79:55173    43683    4.0 M     1
2016-06-04 13:12:25.706  2099.059 TCP    192.168.205.253:902   ->   192.168.202.91:50088     4488    3.8 M     3
2016-06-04 13:32:25.956  1447.980 TCP     192.168.203.61:52409 ->   192.168.202.68:55553    19652    3.7 M     9
Top 10 Src IP Addr ordered by flows:
Date first seen          Duration Proto       Src IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2016-06-04 13:21:50.965  2082.991 any     192.168.202.110    1.3 M(18.5)    1.6 M(19.2)  107.4 M(13.8)      791   412380    65
2016-06-04 13:12:21.896  2651.409 any      192.168.202.83    1.3 M(17.7)    1.3 M(14.8)   76.2 M( 9.8)      479   229948    59
2016-06-04 13:12:21.896  2651.409 any      192.168.206.44    1.3 M(17.7)    1.3 M(14.7)   58.0 M( 7.5)      474   175010    46
2016-06-04 13:18:35.225  2278.731 any      192.168.204.45   637608( 9.0)   668703( 7.8)   33.2 M( 4.3)      293   116516    49
2016-06-04 12:56:26.175  3181.600 any      192.168.202.79   539011( 7.6)   652991( 7.6)   40.1 M( 5.2)      205   100825    61
2016-06-04 13:23:09.375  1960.440 any      192.168.21.100   136633( 1.9)   137153( 1.6)    6.4 M( 0.8)       69    25974    46
2016-06-04 13:04:41.295  3089.060 any       192.168.21.25   136346( 1.9)   137955( 1.6)    6.4 M( 0.8)       44    16681    46
2016-06-04 13:24:16.085  1936.761 any      192.168.21.101   135657( 1.9)   137473( 1.6)    6.7 M( 0.9)       70    27702    48
2016-06-04 12:56:26.815  3606.751 any      192.168.27.203    99196( 1.4)   197790( 2.3)   58.5 M( 7.5)       54   129710   295
2016-06-04 13:24:05.796  1639.650 any        192.168.21.1    81708( 1.1)    81708( 1.0)    3.8 M( 0.5)       49    18344    46


Top 10 IP Addr ordered by flows:
Date first seen          Duration Proto           IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2016-06-04 13:12:21.896  2651.409 any      192.168.202.83    2.5 M(35.5)    2.5 M(29.5)  134.6 M(17.3)      954   406261    53
2016-06-04 13:01:30.765  3302.540 any      192.168.206.44    2.5 M(35.4)    2.5 M(29.4)  134.1 M(17.3)      765   324817    53
2016-06-04 13:21:18.076  2115.880 any     192.168.202.110    2.1 M(29.6)    2.7 M(31.1)  287.7 M(37.0)     1262    1.1 M   107
2016-06-04 13:18:35.225  2278.731 any      192.168.204.45    1.2 M(16.6)    1.2 M(14.3)   62.8 M( 8.1)      537   220490    51
2016-06-04 12:56:26.175  3181.600 any      192.168.202.79   945354(13.3)    1.2 M(13.5)   70.7 M( 9.1)      363   177779    61
2016-06-04 13:23:09.375  1960.440 any      192.168.21.100   273308( 3.8)   274830( 3.2)   12.8 M( 1.6)      140    52234    46
2016-06-04 13:04:41.295  3089.060 any       192.168.21.25   272727( 3.8)   277009( 3.2)   13.0 M( 1.7)       89    33626    46
2016-06-04 13:24:16.085  1936.761 any      192.168.21.101   271342( 3.8)   275553( 3.2)   13.4 M( 1.7)      142    55216    48
2016-06-04 12:56:26.815  3606.751 any      192.168.27.203   222712( 3.1)   466171( 5.4)   84.5 M(10.9)      129   187494   181
2016-06-04 12:56:26.395  3607.341 any      192.168.27.102   196801( 2.8)   269846( 3.1)   33.0 M( 4.2)       74    73203   122


Top 10 Dst Port ordered by packets:
Date first seen          Duration Proto          Dst Port    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2016-06-04 13:09:16.155  2581.721 any               63805   393252( 5.5)   393252( 4.6)   18.1 M( 2.3)      152    56054    46
2016-06-04 13:09:29.186  2824.770 any               41936   321527( 4.5)   321535( 3.7)   14.8 M( 1.9)      113    41890    46
2016-06-04 12:56:26.175  3607.141 any                  80    83000( 1.2)   299840( 3.5)   35.0 M( 4.5)       83    77617   116
2016-06-04 13:10:27.276  2675.259 any               39436   205308( 2.9)   205313( 2.4)    9.4 M( 1.2)       76    28253    46
2016-06-04 12:56:26.176  3607.770 any                 443    49667( 0.7)   163292( 1.9)   13.6 M( 1.8)       45    30208    83
2016-06-04 13:09:16.206  2837.750 any                8080    20083( 0.3)   116806( 1.4)   14.1 M( 1.8)       41    39685   120
2016-06-04 12:56:26.176  3607.099 any                 445    13032( 0.2)   101474( 1.2)   14.5 M( 1.9)       28    32108   142
2016-06-04 13:09:16.956  2823.010 any               45799    59062( 0.8)    59071( 0.7)    2.7 M( 0.4)       20     7711    46
2016-06-04 13:10:47.245  2746.691 any               55554       99( 0.0)    55413( 0.6)    9.0 M( 1.2)       20    26128   161
2016-06-04 13:09:52.365  2773.761 any               34371    53146( 0.7)    53148( 0.6)    2.4 M( 0.3)       19     7053    46


Top 10 Dst Port ordered by bytes:
Date first seen          Duration Proto          Dst Port    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2016-06-04 12:56:26.175  3607.141 any                  80    83000( 1.2)   299840( 3.5)   35.0 M( 4.5)       83    77617   116
2016-06-04 13:09:16.155  2581.721 any               63805   393252( 5.5)   393252( 4.6)   18.1 M( 2.3)      152    56054    46
2016-06-04 13:10:37.066  2653.240 any               52965       78( 0.0)    20747( 0.2)   16.6 M( 2.1)        7    50190   802
2016-06-04 13:09:29.186  2824.770 any               41936   321527( 4.5)   321535( 3.7)   14.8 M( 1.9)      113    41890    46
2016-06-04 12:56:26.176  3607.099 any                 445    13032( 0.2)   101474( 1.2)   14.5 M( 1.9)       28    32108   142
2016-06-04 13:09:16.206  2837.750 any                8080    20083( 0.3)   116806( 1.4)   14.1 M( 1.8)       41    39685   120
2016-06-04 12:56:26.176  3607.770 any                 443    49667( 0.7)   163292( 1.9)   13.6 M( 1.8)       45    30208    83
2016-06-04 13:10:27.276  2675.259 any               39436   205308( 2.9)   205313( 2.4)    9.4 M( 1.2)       76    28253    46
2016-06-04 13:09:31.326  2815.290 any               54282      118( 0.0)    41790( 0.5)    9.2 M( 1.2)       14    26280   221
2016-06-04 13:10:47.245  2746.691 any               55554       99( 0.0)    55413( 0.6)    9.0 M( 1.2)       20    26128   161


Top 10 Dst Port ordered by pps:
Date first seen          Duration Proto          Dst Port    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2016-06-04 13:09:16.155  2581.721 any               63805   393252( 5.5)   393252( 4.6)   18.1 M( 2.3)      152    56054    46
2016-06-04 13:09:29.186  2824.770 any               41936   321527( 4.5)   321535( 3.7)   14.8 M( 1.9)      113    41890    46
2016-06-04 12:56:26.175  3607.141 any                  80    83000( 1.2)   299840( 3.5)   35.0 M( 4.5)       83    77617   116
2016-06-04 13:10:27.276  2675.259 any               39436   205308( 2.9)   205313( 2.4)    9.4 M( 1.2)       76    28253    46
2016-06-04 12:56:26.176  3607.770 any                 443    49667( 0.7)   163292( 1.9)   13.6 M( 1.8)       45    30208    83
2016-06-04 13:09:16.206  2837.750 any                8080    20083( 0.3)   116806( 1.4)   14.1 M( 1.8)       41    39685   120
2016-06-04 12:56:26.176  3607.099 any                 445    13032( 0.2)   101474( 1.2)   14.5 M( 1.9)       28    32108   142
2016-06-04 13:09:16.956  2823.010 any               45799    59062( 0.8)    59071( 0.7)    2.7 M( 0.4)       20     7711    46
2016-06-04 13:10:47.245  2746.691 any               55554       99( 0.0)    55413( 0.6)    9.0 M( 1.2)       20    26128   161
2016-06-04 13:09:52.365  2773.761 any               34371    53146( 0.7)    53148( 0.6)    2.4 M( 0.3)       19     7053    46


Summary: total flows: 7107131, total bytes: 777.1 M, total packets: 8.6 M, avg bps: 1.7 M, avg pps: 2380, avg bpp: 90
Time window: 2016-06-04 12:56:26 - 2016-06-04 13:56:33
Total flows processed: 7107131, Blocks skipped: 0, Bytes read: 369575156
Sys: 9.514s flows/second: 747000.4   Wall: 10.322s flows/second: 688519.3


real 0m12.218s
user 0m10.668s
sys 0m0.706s


don't forget that you can use pcap filters with nfdump that will be used with the original pcap file to supress the noise or you can export the flows to csv and insert them in elasticsearch to analyze with Kibana:


root@vagrant-ubuntu-trusty-64:~# time nfdump -r nfcapd.201607021119 -s dstip/flows 'src ip 192.168.22.254'
Top 10 Dst IP Addr ordered by flows:
Date first seen          Duration Proto       Dst IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2016-06-04 13:36:50.866  1182.079 any      192.168.204.45      239(65.7)      420( 1.2)    41383( 0.2)        0      280    98
2016-06-04 13:16:31.776  2392.870 any      192.168.202.76      101(27.7)    34860(95.5)   20.6 M(99.3)       14    68706   589
2016-06-04 13:37:43.676    28.599 any     192.168.202.110       12( 3.3)       12( 0.0)      574( 0.0)        0      160    47
2016-06-04 13:28:23.416  1540.450 any     192.168.202.109        8( 2.2)     1225( 3.4)   101636( 0.5)        0      527    82
2016-06-04 13:29:24.306   888.420 any     192.168.202.102        3( 0.8)        3( 0.0)      144( 0.0)        0        1    48
2016-06-04 13:48:00.426     0.000 any     192.168.202.108        1( 0.3)        1( 0.0)       46( 0.0)        0        0    46


Summary: total flows: 364, total bytes: 20.7 M, total packets: 36521, avg bps: 68947, avg pps: 15, avg bpp: 566
Time window: 2016-06-04 12:56:26 - 2016-06-04 13:56:33
Total flows processed: 7107131, Blocks skipped: 0, Bytes read: 369575156
Sys: 0.490s flows/second: 14481415.1