Friday, February 20, 2015

Netflow Analysis with Elasticsearch

Netflow Analysis with Elasticsearch

This post http://www.bulutsal.com/2014/03/netflow-analysis-using-elasticsearch.html shows how to use Elasticsearch and Kibana to analyze netflows, so let's try it with the VAST 2013 netflow dataset .

To install the ELK stack I used docker and fig as explained here https://deviantony.wordpress.com/2014/12/06/an-elk-stack-powered-by-docker-and-fig/
itsuugo@itsuugo-HPE ~/nf $ ls -alh
total 5,5G
drwxr-xr-x 2 itsuugo itsuugo 4,0K feb 17 00:19 .
drwxr-xr-x 4 itsuugo itsuugo  12K feb 20 11:00 ..
-rw-r--r-- 1 itsuugo itsuugo 1,8G abr 19  2013 nf-chunk1.csv
-rw-r--r-- 1 itsuugo itsuugo 2,6G abr 19  2013 nf-chunk2.csv
-rw-r--r-- 1 itsuugo itsuugo 1,2G abr 19  2013 nf-chunk3.csv

itsuugo@itsuugo-HPE ~/nf $ wc nf-chunk1.csv 
  15172768   30345535 1908288165 nf-chunk1.csv

itsuugo@itsuugo-HPE ~/nf $ head nf-chunk1.csv 
TimeSeconds,parsedDate,dateTimeStr,ipLayerProtocol,ipLayerProtocolCode,firstSeenSrcIp,firstSeenDestIp,firstSeenSrcPort,firstSeenDestPort,moreFragments,contFragments,durationSeconds,firstSeenSrcPayloadBytes,firstSeenDestPayloadBytes,firstSeenSrcTotalBytes,firstSeenDestTotalBytes,firstSeenSrcPacketCount,firstSeenDestPacketCount,recordForceOut
1364802616.290452,2013-04-01 07:50:16,20130401075016.290452,17,UDP,172.20.0.3,172.255.255.255,137,137,0,0,29,600,0,1104,0,12,0,0
1364802621.0921521,2013-04-01 07:50:21,20130401075021.092152,17,UDP,172.10.0.40,172.255.255.255,137,137,0,0,0,100,0,184,0,2,0,0

We need to modify the script to import the dataset and create the data mappings, but only with the interesting fields.
Time to take a coffee while it loads ;)
itsuugo@itsuugo-HPE ~/nf $ time ./importES.py ~/Descargas/nf/nf-chunk1.csv
...
Indexed 15000000, working on next 100000
Indexed 15100000, working on next 100000
Indexed 15172766, finishing.

real    29m0.041s
user    20m6.394s
sys 0m10.452s

We can build our custom visualizations and dashboards, here you have a good tutorial https://www.timroes.de/2015/02/07/kibana-4-tutorial-part-1-introduction/
The DDOS is easy to visualize, and the search capabilities are fast and awesome.