Flocon 2012: Argus Training Slide

If you are looking for detail information about latest argus development and offering, look no further -

http://www.qosient.com/argus/presentations/Argus.FloCon.2012.Tutorial.pdf

The slide is made by Carter and it contains a lot of information for state of the art flow analysis tool - argus. Though a long time argus user, I still learn something new from the slide.

Cheers (;])

Argus 3: Some hardly used scripts

There are couple of perl scripts come with argus 3 to process argus data, in case you haven't used them, do try them out, I will just show the result generated by those scripts -

shell>perl ./raips -r ~/pcap-repo/anubis.arg3
187.45.196.28
187.45.241.156
192.168.0.1
192.168.0.2

Raips will generate all unique IP addresses that are seen in the argus data.

shell>perl ./rahosts -r ~/pcap-repo/anubis.arg3
192.168.0.2: (3) 187.45.196.28, 187.45.241.156, 192.168.0.1

Rahosts will generate host report, and telling you the hosts that initiate network connection(transmitter) and also destination hosts that are probed(receiver), you may get an array of IP addresses in the same network if it is network scanning or worm outbreak activity.

shell>perl ./raports -r ~/pcap-repo/anubis.arg3
187.45.241.156 tcp: (1) 80
192.168.0.1 udp: (1) 53
187.45.196.28 tcp: (1) 1433

Raports will generate the port report, however only on server side, which means those ports that are probed by any host.

If you are not satisfied with the result generated by those scripts, you are free to modify them to fit your needs, basically Carter is just demonstrating what you can do with argus data using some scripting capabilities.

Cheers (;])

Argus 3: Situational Awareness(ratop)

You need to know the current state of the network, who is probing your network and services, who is consuming your bandwidth, what are the stuffs running in your network, the main question remains - How much you know about your network?

Then people talk about Situational Awareness, in fact Wikipedia has well-versed explanation about it where you can find here.

As network security operator, we look at Network Situational Awareness, in fact you can use Argus 3 for this purpose, I'm going to discuss about it here. There are few argus client tools that can be used for near Real Time Network Situational Awareness -

- ratop
- rasql/rasqlinsert
- ralabel

Ratop works just like top, it can connect to argus monitor and show network flow data in near real time view, it also offers vi-like feature, where you can use / to search for flows, and : as command mode to perform various actions such as network flow record filtering/sorting, flow record field reordering, or even extract flow record based on certain timespan in real time. To run ratop, you must have argus monitor running first -

shell>argus -mAJZRU 128 -P 561

Use ratop to connect to the argus monitor -

shell>ratop -S localhost:561

Here's the ratop screenshot -

To quit ratop, it is similar to exiting vi editor, just type :q and you will disconnect from argus monitor. You can see that ratop is very useful when comes to monitor your network in real time, while it doesn't offer you insightful information, it gives quick view of the layer2/3 network conversation. Other features such as sorting can be toggled on with :s, or filtering with :f.

This is considered part 1 which I have ratop covered, and for part 2 I'm going to discuss about rasql/rasqlinsert, then I will introduce ralabel in part 3. All of them are very effective tools for Network Situational Awareness.

Enjoy (:])

Argus 3: OpenWRT Binary Blob

Here's the argus 3 binary blob that will work on OpenWRT KamiKaze 8.09(Linksys WRT54GL MIPS platform), if you are lazy to compile your own, and want to check it out, please do give it a try. Thanks to guti for hosting it -

http://gutizz.com/scripts/argusbinary/argus3-mips.tar.bz2

http://gutizz.com/scripts/argusbinary/argus3-mips.tar.bz2.md5.txt

All you need to do is download, verify, decompress, upload it to your OpenWRT, and run!

Enjoy (;])

Argus 3: Database Support

If you have followed argus mailing list, you should have known that Carter has implemented argus database client(rasql/rasqlinsert) to read/write/bla network flow records to database. I'm currently testing this feature and here's the preview for you -

Currently it seems to work on my testing machine. I will introduce more about the new argus client tools such as ralabel, rasql, rasqlinsert and etc in my coming posts.

Cheers (;])

Argus 3.x On Linksys WRT54GL

I have bought two units of Linksys WRT54GL wlan router previously so that I can run Linux and getting network security monitoring tools running on it as well. This little device has very limited space but you can't beat linux as router device. One of the unit is currently living in spoonfork's place to serve that Darth Vader, and another one is with me.

Since Carter has argus supported on OpenWRT, I have been thinking of getting argus installed on it(MIPS platform). And after some tinkering, I have successfully loading argus on it and export the network flow to another box in the network. Here's the complete howto that you can follow exactly to get argus compiled for OpenWRT Kamikaze 8.09(MIPS platform) using Ubuntu Linux.

Prepare the environment, my main directory to build this is /home/geek00l/i-Projects -

shell>sudo apt-get install gcc g++ patch binutils \
flex bison make pkg-config unzip zlib1g zlib1g-dev \
libc6 libc6-dev gawk autoconf upslug2 libncurses5-dev

To build OpenWRT Kamikaze 8.09, svn up the source first -

shell>svn co https://svn.openwrt.org/openwrt/branches/8.09 kamikaze-8.09

shell>cd kamikaze-8.09

Start the building process -

shell>make defconfig

shell>make package/symlinks

shell>make menuconfig

shell>make

Take a coffee break when you run make .....

Install libpcap, this is the only dependencies we need to get argus 3 compiled -

shell>make package/libpcap-compile V=99

shell>make package/libpcap-install V=99

Check out the gcc that we need to use -

shell>/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/gcc --version
gcc (GCC) 3.4.6 (OpenWrt-2.0)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Now we need to set the environment variables for this build -

shell>export PATH=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin:/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/bin:$PATH

shell>export AR=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/ar

shell>export AS=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/as

shell>export LD=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/ld

shell>export NM=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/nm

shell>export CC=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/gcc

shell>export CPP=/home/geek00l/i-Projects/kamikaze-8.09/build_dir/toolchain-mipsel_gcc3.4.6/gcc-3.4.6-initial/gcc/cpp

shell>export GCC=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/gcc

shell>export CXX=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/g++

shell>export RANLIB=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir/toolchain-mipsel_gcc3.4.6/mipsel-linux-uclibc/bin/ranlib

shell>export ac_cv_linux_vers=2.4.35

shell>export LDFLAGS="-static"

shell>export CFLAGS="-Os -s"

Time to have fun, doing cross-compile for argus so it works on MIPS platform -

shell>cd /home/geek00l/i-Projects/argus-3.0.1.beta.2

shell>./configure --host=mipsel-linux \
--with-openwrt=/home/geek00l/i-Projects/kamikaze-8.09/staging_dir \
--with-libpcap=/home/geek00l/i-Projects/kamikaze-8.09/build_dir/mipsel/libpcap-0.9.8

shell>make

shell>file bin/argus
bin/argus: ELF 32-bit LSB executable, MIPS, version 1 (SYSV), statically linked, stripped

Transfer it to my OpenWRT -

shell>scp -P 55555 bin/argus root@192.168.1.1:/tmp

To export argus network flow on ppp0 interface -

shell>argus -i ppp0 -B 192.168.1.1 -P 561 -d

To intercept the network flow -

shell>ra -S 192.168.1.1:561 - ip

I have found some good references here to get me going, and I would like to thank to David Watson(UK Honeynet) for his guide on building nepenthes on openwrt too.

Reference:
http://www.frontiernet.net/~beakmyn/CrossCompile.htm
http://www.ukhoneynet.org/research/building-nepenthes-on-the-openwrt-embedded-platform/
http://forum.openwrt.org/viewtopic.php?pid=31794
http://gargoyle-router.com/openwrt-coding.php

Since this embedded device has very limited space, there's no point to run packet logger locally, other tools I would like to run on it so that I can export pcap to other system should be something like packetforward or rpcap. If anyone has experience to get any of these tools installed on OpenWRT, please do share!

Enjoy (;])

Network Flow: Uni-Directional VS Bi-Directional

If you are working on network flow research, you should have heard about Uni-Directional and Bi-Directional Network Flow. I will try to explain what are they here. Lets take the quick look of what network flow is first -

Network Flow is the sequence of packets or a packet that belonged to certain network session(conversation) between two end points but delimited by the setting of flow generation tool. To cut it short, it provides network traffic summarization by metering or accounting certain attributes in the network session.

The endpoints here are defined as below -

Layer 2 Endpoint - Source Mac Address | Destination Mac Address
Layer 3 Endpoint - Source IP Address | Destination IP Address
Layer 4 Endpoint - Source Port | Destination Port

Before we dive into understanding of UniFlow and BiFlow, lets look at the definition of Uni and Bi here -

http://www.yourdictionary.com/uni-prefix

http://www.yourdictionary.com/bi-prefix

Uni - one; having or consisting of one only; regarded as a single entity

Bi - using two or both; joining two, combining or involving two

In the context of Uni/Bi Directional Flow, Uni means single, Bi means both. Now, let make it more clearer.

Uni-Directional = Single Directional

Bi-Direction = Both Directional

I put up the illustration in the diagram below.

Uni-Directional Flow

Bi-Directional Flow

Now I will make a simple example, host A sends 90 bytes to host B and host B replies with 120 bytes. Here's the output -

Uni-Directional Network Flow
Srcaddr Direction Dstaddr Total Bytes
Host A -> Host B 90
Host B -> Host A 120

Bi-Directional Network Flow
Srcaddr Direction Dstaddr Total Bytes Src Bytes Dst Bytes
Host A <-> Host B 210 90 120

The Srcaddr and Dstaddr are the endpoints here. In Uni-Directional Flow, you only see the total bytes that sent by Host A(attribute of Host A) but nothing about Host B in the first flow record. Then the next record shows Host B sends 120 bytes to Host A(attribute of Host B). The total bytes is accounted from single endpoint(either Host A or B) only. But in BiFlow, you can see that Host A sends 90 bytes(Source Bytes) and Host B replies with 120 bytes(Destination Bytes). The total bytes is the accumulation of source and destination bytes. To summarize them -

Uni-Directional Network Flow Model - One direction at a time, every flow record contains the attribute of single endpoint only.

Bi-Directional Network Flow Model - Both direction at a time, every flow record contains the attribute of both endpoints.

Theory is tough sometime, here's the practical sample -

Cisco NetFlow uses Uni-Directional model for flow generation

Argus uses Bi-Directional model for flow generation

To draw good picture of Uni-Directional and Bi-Directional Network Flow, it's best to do comparison of them.

1. Network Flow data which is generated by Argus 3 natively
2. Network Flow data which is generated by Cisco NetFlow version 5

The flow records below are generated from the same network session. You can examine closely by clicking on them.

Cisco NetFlow(UniFlow):

Argus(BiFlow):


Flow record property:
SrcAddr = Source Address
Sport = Source Port
Dir = Direction
DstAddr = Destination Address
Dport = Destination Port
SrcPkts = Source Packets
DstPkets = Destination Packets
TotPkts = Total Packets
SrcBytes = Source Bytes
DstBytes = Destination Bytes
TotBytes = Total Bytes

Sometimes I like to think that UniFlow is stateless and BiFlow is stateful.

I will continue writing this Network Flow series, and I hope you enjoy it. Stay tuned for the next one - Traffic Matrix. And of course the HeX 021 series too.

Argus 3 Tip:
You can convert Argus BiFlow to UniFlow by using -M rmon option.

Peace (;])

Network Flow: TopN

There are a lot of questions popping up on and off in argus mailing list regarding how to generate TopN output from argus data, but frequently you may find the questions are too rough to give complete answer.

I'm going to discuss about TopN this time, TopN is the technique that widely used in many industries, what is it for?

TopN is used to retrieve the first N records from the data based on certain object and ordered by its property. Since I'm talking about Network Flow, I would like to make the example using it.

Data: Network Flow Record
Object: Protocol, Network, IP(host), Port, etc
Object Property: Packet Count, Byte Count, etc

Bear in mind that I'm avoiding the use of Flow terminology but layman one so that this example can be understood easily.

If you want to use TopN technique to generate information from the network flow data, first you need to know what you are looking for. Lets go with a simple one -

I want to find out Top 5 IP ordered by Total Packet Count

Total Packet Per IP(host) = (packet send + packet receive) Per IP(host)

Now you run the argus client command to parse the data and generate exactly the result which looks like this -

shell>racluster -M rmon -m saddr -nr testing.arg3 -w - | \
rasort -m pkts -w - | \
ra -L0 -N 5 -s saddr pkts

SrcAddr TotPkts
172.16.1.108 993
193.231.236.41 824
211.185.125.124 178
172.16.1.103 56
211.180.229.190 36

The command above is to generate Top 5 IP ordered by Packet Count. Don't ask me about the command line, it looks complicated for now but that's not my point here, look at the output instead. Host 172.16.1.108 sends or receives 993 pakcets, followed by 193.231.236.41 and so forth.

Now if you want to locate Top 5 IP ordered by Byte Count. You can just run -

shell>racluster -M rmon -m saddr -nr testing.arg3 -w - | \
rasort -m bytes -w - | \
ra -L0 -N 5 -s saddr bytes

SrcAddr TotBytes
172.16.1.108 599949
193.231.236.41 579050
211.185.125.124 18901
172.16.1.103 4964
216.168.224.69 3458

You want to use TopN, you should draft out the TopN output you are looking for, I have seen questions like this -

1. Which is the most active network?
2. Who is the most active sender?
3. Who is the most active receiver(got ddos?)

Or worse,

How can I find out the top talkers?

These kind of questions are too loose, you should at least specify the property, such as most active sender that is ordered by packet count, or most active network that is ordered by byte count and so forth. You have to bear in mind that packet and byte are not going inline, you can have one host sending many small size packets which won't hit TopN byte count at all.

With this kind of idea in mind, you can build the list of TopN which can draw you a good picture of network activeness to solve different issues.

For the next round, I will introduce Traffic Matrix, stay tuned!

Enjoy (;])

Argus 3 Release

This is nothing new, argus 3 is finally released after long testing period. Thanks to everyone who involves in the argus 3 development and testing cycle especially Carter. You can download it at -

- http://qosient.com/argus/downloads.htm

If you are using FreeBSD, good new is argus 3 port is available now and you can check out the information about it here -

- http://www.freshports.org/net-mgmt/argus3/
- http://www.freshports.org/net-mgmt/argus3-clients/

I will try out the argus 3 ports on FreeBSD and see how it goes, have fun!

Enjoy ;]

Argus 3: Tips & Tricks

I have labeled all my argus 3 blog post to make it easy for people who are interested in it. You can find them at -

http://geek00l.blogspot.com/search/label/Argus3

If you are using argus, and would like to know how to use certain client tools and you think it might be useful, feel free to email me as well and maybe I can put up the tips and tricks for it.

Peace ;]

Argus 3: Audit Trail with HGWL

I have discussed about the Harimau Global Watchlist here, initially one of the reason Spoonfork developes HGWL is to put all the malicious IP addresses in one place so that it can be used with argus. In fact it can be used with any network flow technology but I'm going to show how you can use HGWL and argus 3 to perform trail auditing.

Obtaining the Harimau Global Watchlist -

shell>wget http://watchlist.security.org.my/all.txt

Generate IP addresses list from argus data(shit.arg3) -

shell>racluster -M rmon -m saddr -r shit.arg3 -s saddr - ip > IP.list

I use -M rmon to convert argus biflow data to uniflow, the source and destination doesn't matter now, with that I just need to specify it to show source address by using -s saddr and I will be able to obtain all the IP address. The -m saddr is used here to cluster source address field only so the same IP is only shown once.

Filtering out suspected IP addresses -

shell>for host in `cat IP.list`; \
do egrep $host all.txt >> Suspect.list; done

61.187.72.70, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08
61.193.128.171, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08
65.160.238.180, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08
66.45.254.244, www.emergingthreats.net/rules/bleeding-rbn.rules, ET RBN Known Russian Business Network Monitored Domains (15), rbn, 2008/03/16 00:03:31
195.161.8.1,http://www.senderbase.org/home/detail_spam_source,senderbase-spam-source,2008/03/16 00:28:34
202.58.85.15, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08
217.175.134.217, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08
218.30.19.40, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08
221.6.4.67, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08
221.237.161.186, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2008/03/16 00:00:08

Most of the time we start doing audit trail when IDS is giving us pointer. Now we can start querying the IP addresses in Suspect.list from argus data when there's matching and tracing its network activities. For example -

shell>ra -nr data.arg3 - host 195.161.8.1

StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
13:00:58.918872 e tcp 195.161.8.1.10424 -> 192.168.42.109.25 16 1559 RST
13:00:59.275570 e tcp 192.168.42.109.28457 -> 195.161.8.1.113 1 60 REQ
13:02:03.314641 e d tcp 195.161.8.1.24113 -> 192.168.42.109.25 8 698 RST
13:02:03.815631 e tcp 192.168.42.109.28493 -> 195.161.8.1.113 1 60 REQ

Output truncated ....

End for this.

The new client tool - rastream is included in argus 3 lately as argus data stream block processor, I'm thinking of using it to perform HGWL matching operation automatically once the argus data is generated so that we can catch near real time suspected host to watch possible ongoing malicious actions, maybe I will discuss about rastream in future blog post.

Updates: My friend scholar01 has shown me useful trick, instead of using for host in `cat IP.list`; do egrep $host all.txt >> Suspect.list; done, it is better to use fgrep -f IP.list all.txt as fgrep has advance engine for matching static strings at once. Thanks.

Enjoy (;])

Argus 3: German Article

My friend Stefan has sent me this link about argus 3 but it is in German language, so I think it's good to share with others. You can click the link here -

- The Argus-eye watches(German)

Thanks to Google Translator, you can view the english version here -

- The Argus-eye watches(English)

Basically the article demonstrates the usages of argus client tools, and gives brief explanation about them, the article is written by Ralf Spenneberg and it should be good read for people who just started to adapt to argus 3.

Cheers ;]

Argus 3: Statistics for Major Protocols

Most people would like to have macro view of the network, for example how many bytes have been utilized for protocol such as tcp, udp and icmp, or other things like the amount of packets that have been transmitted or received.

Previously in argus 2.x, argus offers racount -ar to generate the general statistics but the option -a is gone in argus 3.x, so how can you generate the network utilization for major protocols? I have shown the usage of racluster previously here for network session reconstruction and now I will demonstrate another example of using racluster.

Before I move on, I would like to rephrase racluster's functions from the man page -

Racluster reads argus data from an argus-data source, and clusters/merges the records based on the flow key criteria specified either on the command line, or in a racluster configuration file, and outputs a valid argus-stream. This tool is primarily used for data mining, data management and report generation.

Here you go, you can cluster or merge the records based on the flow key and it is suitable for data mining, data management and report generation, let's generate the statistical report using protocol as flow key. Notice I specify -m proto in command line below and using -s to print the field I want -

shell>racluster -L0 -m proto -r data.arg3 -s proto trans pkts bytes appbytes -\
tcp or udp or icmp
Proto Trans TotPkts TotBytes TotAppByte
udp 18115 72665 8488022 5430758
tcp 22996 1291078 969152661 895531494
icmp 1089 1933 424733 346837

This is something simple from racluster but you maybe scratching your head to figure how to do it when you are still new with argus 3(in fact I did), it is considered one of the most powerful tool in argus 3 client suite and maybe sooner, I will talk more about it. Hopefully you find it helpful(hint, hint).

Enjoy (;])

Packets -> Flows -> Session

This is my last post before reaching the milestone 500th, so I try my best to write great post. Since I want to keep this post simple and clear, I will try my best to explain it in details. If you are network flow analysis guru, you can skip this post because I consider this as introductory post but may help others understanding more about network flow because it was me who taking amount of time to learn how to utilize network flow data. My approach will be similar to my previous post here but the topic is totally different. The "Not So Upcoming" Argus 3 will be the main weapon to be discussed here. Lets walk through it now.

Network Packets

For this matter, I need to obtain the network packets, I have logged the network traffic using tcpdump during the time I was downloading wireshark. Here's how I do it -

shell>sudo tcpdump -s 0 -nni lnc0 -w http-download.pcap

After I finished downloading wireshark, I terminated tcpdump and get initial view of the pcap file with capinfos.

shell>capinfos http-download.pcap
File name: http-download.pcap
File type: Wireshark/tcpdump/... - libpcap
Number of packets: 19782
File size: 1981512 bytes
Data size: 18047455 bytes
Capture duration: 405.100833 seconds
Start time: Thu Dec 20 23:43:22 2007
End time: Thu Dec 20 23:50:07 2007
Data rate: 44550.53 bytes/s
Data rate: 356404.20 bits/s
Average packet size: 912.32 bytes

For single file download which is approximately 20MB, it contains 19872 packets. It is painful to look at every single packet if it is not important. What if I don't want to know the payload in the packet but the connection summary such as how many packets have been sent by one host to another, how many bytes have been transferred in this connection? How long is the duration of this particular connection? Packet centric analysis doesn't fit well here. Therefore I introduce you network flow analysis. But before that, lets have fun with packets -

------------------------------------------------------------
Scenario:

Host A(Client) - 192.168.0.102
Host B(Server) - 128.121.50.122

Host A downloads the wireshark source from Host B
-------------------------------------------------------------

To get the count of how many packets have been sent by Host A to Host B -

shell>tcpdump -ttttnnr http-download.pcap \
ip src 192.168.0.102 | wc -l
reading from file http-download.pcap, link-type EN10MB (Ethernet)
7801

To get the count of how many packets have been sent by Host B to Host A -

shell>tcpdump -ttttnnr http-download.pcap \
ip src 128.121.50.122 | wc -l
reading from file http-download.pcap, link-type EN10MB (Ethernet)
11981

What if you want to know how many bytes have been sent by Host A to Host B and the reversal? It would be exhaustive if you have to look into those packets and count. Now this is where network flow kicks in.

Network Flows

Network Flow is really different beast. To give you the idea what is flow, I define it as -

Flow is the sequence of packets or a packet that belonged to certain network session(conversation) between two hosts but delimited by the setting of flow generation tool. To cut it short, it provides network traffic summarization by metering or accounting certain attributes in the network session.

To understand them better, lets convert the packet data(pcap) to argus format flow data -

shell>argus -mAJZRU 512 -r http-download.pcap \
-w http-download.arg3

I run argus with the option -mAJZRU 512 so that it will generate as much data as possible for each flow record. I won't explain each option here since you can find them in the man page or argus -h.

Now I can examine/parse http-download.arg3 with argus client tools for further flow processing. To make it easy to read, I use ra here as it is the most basic argus flow data processing tool. I choose to print the necessary field with -s option such as (start time|src address|src port|direction|dst address|dst port|src packets|dst packets) -

shell>ra -L0 -nnr http-download.arg3 \
-s stime saddr sport dir daddr dport spkts dpkts - ip
StartTime SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts
23:43:22.024899 192.168.0.102.51371 -> 128.121.50.122.80 1165 1800
23:44:22.068631 192.168.0.102.51371 -> 128.121.50.122.80 1186 1807
23:45:22.101391 192.168.0.102.51371 -> 128.121.50.122.80 1246 1919
23:46:22.117747 192.168.0.102.51371 -> 128.121.50.122.80 1125 1751
23:47:22.171437 192.168.0.102.51371 -> 128.121.50.122.80 1160 1759
23:48:22.209375 192.168.0.102.51371 -> 128.121.50.122.80 1080 1664
23:49:22.186030 192.168.0.102.51371 -> 128.121.50.122.80 839 1281

There are totally 7 flow records here for just single network session. Why?

If you read the argus configuration manual page, it mentions -

ARGUS_FLOW_STATUS_INTERVAL
Argus will periodically report on a flow’s activity every ARGUS_FLOW_STATUS_INTERVAL seconds, as long as there is new activity on the flow. This is so that you can get a view into the activity of very long lived flows. The default is 60 seconds, but this number may be too low or too high depending on your uses.

The default value is 60 seconds, but argus does support a minimum value of 1. This is very useful for doing measurements in a controlled experimental environment where the number of flows is <>
Command line equivalent -S

ARGUS_FLOW_STATUS_INTERVAL=60

For better understanding, I print the start time field only to get better interpretation -

shell>ra -nr http-download.arg3 -s stime - ip
23:43:22.024899
23:44:22.068631
23:45:22.101391
23:46:22.117747
23:47:22.171437
23:48:22.209375
23:49:22.186030

With the default setting, you may notice the boundary is 1 minute for each flow record, that's actually what I try to explain above -

Flow is the sequence of packets or a packet that belonged to certain network session(conversation) between two hosts but delimited by the setting of flow generation tool.

If the network session longer than 1 minute(long lived flow), then it will generate another flow(with same attribute/label) which is actually belonged to the same network session though. Of course you can tune this with -S option in argus. Lets try -

shell>argus -S 480 -mAJZRU 512 -r http-download.pcap \
-w http-download-480.arg3

I set 480 seconds here which is 8 minutes as the network session duration falls in that time range. Now we read it again with ra -

shell>ra -L0 -nr http-download-480.arg3 \
-s stime saddr sport dir daddr dport spkts dpkts - ip
StartTime SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts
23:43:22.024899 192.168.0.102.51371 -> 128.121.50.122.80 7801 11981

However in the real world implementation, this is not the right way to construct the network session from multiple flows, especially if your network structure is complex(provides various of network services) and busy(heavy network traffics) and this is really arbitrary. You can't easily identify that multiple flows belong to the same network session as there will be many other flow records inserted in between, another issue is what if the network session duration is longer than 480 seconds(8 minutes). That's where racluster(another argus client tool) comes into rescue.

Network Session

From the racluster partial man page -

Racluster reads argus data from an argus-data source, and clusters/merges the records based on the flow key criteria specified either on the command line, or in a racluster configuration file, and outputs a valid argus-stream. This tool is primarily used for data mining, data management and report generation.
The default action is to merge status records from the same flow and argus probe, providing in some cases huge data reduction with limited loss of flow information.

Racluster is easy to use but hard to master, however here's the simple usage to construct the network session from multiple network flow records.

shell>racluster -L0 -nr http-download.arg3 \
-s stime saddr sport dir daddr dport spkts dpkts
StartTime SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts
23:43:22.024899 192.168.0.102.51371 -> 128.121.50.122.80 7801 11981

It is just really that simple, to explain this network session.

Start Time - 23:43:22.024899
Source Address - 192.168.0.102
Source Port - 51371
Destination Address - 128.121.50.122
Destination Port - 80
Source Packets - 7801
Destination Packets - 11981

Start Time is the time when the network session started, others are pretty self-explained except Source Packets and Destination Packets. Source packets count how many packets have been sent by the Source Address, Destination Packets count how many packets have been sent by Destination Address. To generate summarization of this network session, you can run -

shell>racluster -L0 -nr http-download.arg3 \
-s dur pkts bytes
Dur TotPkts TotBytes
405.100830 19782 18047455

This network session duration is approximately 405 seconds, the total packets in this network session is 19782, and the total bytes is 18047255. Yes, this is where network flow analysis can be useful - traffic accounting but I won't really explain it much here since it will be another topic.

Maybe I should make this post title sounds more interesting with "Network Flow Demystified". There are other topic about network flow where I don't mention here such as Cisco Netflow, Unidirectional vs Bidirectional model, other interesting flow metrics that provided by argus and so forth, I wish I can close the gap in coming posts.

Enjoy (;])

Ubuntu: Argus 3

I'm currently working hard on network flow analysis stuff, and argus is always my best friend. Another wonderful application suite is silktools and I think you should try it out if you are into network flow analysis. Anyway here's the quick installation for upcoming argus 3 on Ubuntu 7.10.

It is pretty straight forward to get argus 3 installed -

shell>sudo apt-get install libpcap0.8 libpcap0.8-dev flex bison rrdtool

Once you have installed all the dependencies of argus 3, lets download argus 3 server and its client suite to install.

shell>wget \
ftp://qosient.com/dev/argus-3.0/argus-3.0.0.tar.gz

shell>wget \
ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.0.rc.63.tar.gz

Once you have downloaded them, you just need to perform usual compilation steps by decompressing them -> configure; make && make install.

And if you still don't know what argus is about, check out this post.

P/S: Both argus and silktools are included in the HeX liveCD.

Enjoy ;]

Packets -> Flows -> CSV -> Graph

Comma-Separated Values(CSV) file format is widely used and it can be easily parsed by lot of graphing tools. Here's the simple trick to generate CSV data from packet dump(pcap) with the used of upcoming argus 3 and the pipe.

Say I downloaded this slammer.pcap that available at wireshark sample capture wiki -

http://wiki.wireshark.org/SampleCaptures

shell>argus -w - -r slammer.pcap | \
ra -nnr - -c ',' -s saddr daddr dport - ip

213.76.212.22,65.165.167.86,1434

There's only one flow but you get the idea of how to generate the CSV ouput from packet dump(pcap). The next thing to do is to generate the graph, I won't show it here but you are free to use any application such as OpenOffice Spreadsheet, afterglow and etc for that purpose.

The good thing about argus is that it provides wide range of useful flow metrics so you can actually generate rich set of data for graphing purpose.

Enjoy (;])

Argus: Development Testing

I think many of you have heard about argus from my previous blog post. Here's how I maintained my argus installation in my testing environment on FreeBSD platform.

Installing argus server -

shell>wget ftp://qosient.com/dev/argus-3.0/argus-3.0.0.tar.gz

shell>tar xvzf argus-3.0.0.tar.gz

shell>cd argus-3.0.0

shell>./configure --prefix=/usr/local/stow/argus3

shell>make && make install

shell>mv /usr/local/stow/argus3/bin/argusbug /usr/local/stow/argus3/

Installing argus client -

shell>wget ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.0.rc.45.tar.gz

shell>tar xvzf argus-clients-3.0.0.rc.45.tar.gz

shell>cd argus-clients-3.0.0.rc.45

shell>./configure --prefix=/usr/local/stow/argusc-3rc45

shell>make && make install

I have mentioned about the usage of stow previously, you can easily handle source installation with it.

shell>cd /usr/local/stow

shell>stow argus3

shell>stow argusc-3rc45

If there's new version of argus released, you can just unstow and remove all the argus file from /usr/local/stow, then install the new version using the same steps above and you are done. If you want to try out argus 3, just download HeXtra here and load it to HeX liveCD will do.

Argus: Anonymize the flows

After l have read the interesting example in secviz.org, I decide to try the argus graphing example using afterglow and graphviz tool, however I'm lazy to change the IPs in my pcap file that contains p2p traffic to protect the privacy, hence I decide to use one of the tool bundled in argus suite called ranonymize(the name tells the story), I converted my pcap file to argus file format so that it can be parsed by argus suite.

shell>argus -r p2p-suspect.pcap -w p2p-suspect.arg

Now I just need to use ranonymize to read the file and pipe it to other tools to generate the graph, I have afterglow source under ~/i-Apps and below is the full command line I use -

shell>ranonymize -r p2p-suspect.arg -w - | \
racluster -r - -m saddr daddr proto dport -c, -s saddr daddr | \ ~/i-Apps/afterglow/src/perl/graph/afterglow.pl -a -t -e 2 -c \ ~/i-Apps/afterglow/src/perl/parsers/color.properties | \
neato -Tgif -o p2p-anonymize.gif

Now you will have p2p-anonymize.gif file in the current directory and here's the simple graph. You can see all the IPs are already been anonymized but you get the idea of the flows.

Instead of reading pcap raw data, sometimes graph helps.

Enjoy ;]

It's all about Network Flow

Argus is my favorite tactical tool when performing flow based analysis and I have mentioned about it quiet a few times in my blog. While it works pretty well and flexible with its powerful client tools to do flow processing, it also offers complex usages. I have found two new weapons lately which can enhance my flow based analysis and complementing.

Undoubtedly argus can eat cisco netflow data, but that's not what argus do best. Argus is more pcap friendly instead of netflow data. Then which tools are great in Open Source arsenal to process netflow? I vote silktools where you can find at -

http://silktools.sourceforge.net/

Silktools is developed as part of NetSA security where you can find at -

http://tools.netsa.cert.org/

It is undeniable that all other projects are interesting too, but that doesn't make my point here and I have no time to check them out yet. The main reason why I'm looking into silktools is because it also offers wide range of analysis tools like argus do. Instead of just doing flow data collection, one can perform in depth analysis on the netflow data using the analysis tools that packed with silktools. But again I found out all these great tools come with complexity and that blow away a lot of new comers.

No fear, silktools comes with great documentation, I greatly appreciate all the hard works that have been put into it not only on the coding section but documentation. You can just learn them from the ground up at -

http://silktools.sourceforge.net/silk_docs.html

Check out the silktools analyst handbook and you may find it interesting, I have read through it and about time to bring them into practice.

The second weapon that worth mentioning is ourmon. You can find that I have installed it on FreeBSD. Ourmon is incredible in doing flow anomaly and heuristic detection that brought to you by Jim Binkley. While it doesn't emphasize on post or offline processing, it is the weapon that concentrating on real time analysis combining it's meaningful network graphs. I'm still learning to utilize ourmon in wide area network environment.

Combining argus, silktools and ourmon, I'm sured you will have a lot of fun to research on the network flow. Except argus 3, silktools and ourmon are available via FreeBSD port. Feel free to try them out. Other network flow processing tools that are great should be flow-tools and nfdump.

Peace ;]

Argus 3.0: Cisco Netflow

Cisco netflow is invented initially to speed up the route with its flow cache, but it appears to be very useful this day where Cisco Netflow is used in different area. One of my interest field is examining the network flow data to track the malicious events but you are free to do any kind of interesting research with the netflow data in hand.

Cisco has improved and add new features to its IOS, I have found few new features for Netflow that looks pretty interesting to me where you can capture more useful information. The most commonly used Netflow version is 5, I would like to try out version 9(shiny?If any of you use version 9, I would like to hear from you) however argus doesn't identify Netflow version 9 yet thus I remain to use the solid Netflow version 5. So here I start to export Cisco Netflow data to argus collector(probe). Login to the Cisco Router, I run the following commands -

ios#config t
ios(config)#ip flow-capture packet-length
ios(config)#ip flow-capture ttl
ios(config)#ip flow-capture icmp
ios(config)#ip flow-capture ip-id

I choose to export the Netflow version 5 data from network interface GigabitEthernet 0/0 to my argus collector(192.168.0.55) port 9996.

ios(config)#ip flow-export source GigabitEthernet0/0
ios(config)#ip flow-export version 5
ios(config)#ip flow-export destination 192.168.0.55 9996
ios(config)#ip flow-top-talkers
ios(config)#interface GigabitEthernet 0/0

Enable it at the interface GigabitEthernet 0/0 for both ingress and egress flows -

ios(config-if)#ip route-cache flow
ios(config-if)#ip flow ingress
ios(config-if)#ip flow egress
Ctrl+z

Save it to survive reboot -

ios#copy run start

Once I have done the Cisco router configuration part, I login to my argus collector and do the following -

shell>rasplit -CS 9996 -M time 60m -n \
-w /nsm/argus/log/Net-DMZ/%Y/%m/%d/argus_%H:%M:%S

rasplit is one of argus client tools that can split resulting output into consecutive sections of records based on different criteria. The options -CS 9996 is to connect to port 9996 and identify the input as Cisco Netflow format. The interesting thing here is it will split the data hourly(-M time 60m) and log it to its respected directory.

To read the Netflow data, what you need to do is just change to directory /nsm/argus/log/Net-DMZ/2007/05/19(as for today) and read them with ra or racluster.

All for now, have fun with the flow!

Cheers (;])