Intel X520

I want this for my Christmas present ;]

http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/ethernet-x520.html

I never thought 10G network adapter can go very cheap, really need to get one for development and testing!

Network Flow: Uni-Directional VS Bi-Directional

If you are working on network flow research, you should have heard about Uni-Directional and Bi-Directional Network Flow. I will try to explain what are they here. Lets take the quick look of what network flow is first -

Network Flow is the sequence of packets or a packet that belonged to certain network session(conversation) between two end points but delimited by the setting of flow generation tool. To cut it short, it provides network traffic summarization by metering or accounting certain attributes in the network session.

The endpoints here are defined as below -

Layer 2 Endpoint - Source Mac Address | Destination Mac Address
Layer 3 Endpoint - Source IP Address | Destination IP Address
Layer 4 Endpoint - Source Port | Destination Port

Before we dive into understanding of UniFlow and BiFlow, lets look at the definition of Uni and Bi here -

http://www.yourdictionary.com/uni-prefix

http://www.yourdictionary.com/bi-prefix

Uni - one; having or consisting of one only; regarded as a single entity

Bi - using two or both; joining two, combining or involving two

In the context of Uni/Bi Directional Flow, Uni means single, Bi means both. Now, let make it more clearer.

Uni-Directional = Single Directional

Bi-Direction = Both Directional

I put up the illustration in the diagram below.

Uni-Directional Flow

Bi-Directional Flow

Now I will make a simple example, host A sends 90 bytes to host B and host B replies with 120 bytes. Here's the output -

Uni-Directional Network Flow
Srcaddr Direction Dstaddr Total Bytes
Host A -> Host B 90
Host B -> Host A 120

Bi-Directional Network Flow
Srcaddr Direction Dstaddr Total Bytes Src Bytes Dst Bytes
Host A <-> Host B 210 90 120

The Srcaddr and Dstaddr are the endpoints here. In Uni-Directional Flow, you only see the total bytes that sent by Host A(attribute of Host A) but nothing about Host B in the first flow record. Then the next record shows Host B sends 120 bytes to Host A(attribute of Host B). The total bytes is accounted from single endpoint(either Host A or B) only. But in BiFlow, you can see that Host A sends 90 bytes(Source Bytes) and Host B replies with 120 bytes(Destination Bytes). The total bytes is the accumulation of source and destination bytes. To summarize them -

Uni-Directional Network Flow Model - One direction at a time, every flow record contains the attribute of single endpoint only.

Bi-Directional Network Flow Model - Both direction at a time, every flow record contains the attribute of both endpoints.

Theory is tough sometime, here's the practical sample -

Cisco NetFlow uses Uni-Directional model for flow generation

Argus uses Bi-Directional model for flow generation

To draw good picture of Uni-Directional and Bi-Directional Network Flow, it's best to do comparison of them.

1. Network Flow data which is generated by Argus 3 natively
2. Network Flow data which is generated by Cisco NetFlow version 5

The flow records below are generated from the same network session. You can examine closely by clicking on them.

Cisco NetFlow(UniFlow):

Argus(BiFlow):


Flow record property:
SrcAddr = Source Address
Sport = Source Port
Dir = Direction
DstAddr = Destination Address
Dport = Destination Port
SrcPkts = Source Packets
DstPkets = Destination Packets
TotPkts = Total Packets
SrcBytes = Source Bytes
DstBytes = Destination Bytes
TotBytes = Total Bytes

Sometimes I like to think that UniFlow is stateless and BiFlow is stateful.

I will continue writing this Network Flow series, and I hope you enjoy it. Stay tuned for the next one - Traffic Matrix. And of course the HeX 021 series too.

Argus 3 Tip:
You can convert Argus BiFlow to UniFlow by using -M rmon option.

Peace (;])

Argus 3: German Article

My friend Stefan has sent me this link about argus 3 but it is in German language, so I think it's good to share with others. You can click the link here -

- The Argus-eye watches(German)

Thanks to Google Translator, you can view the english version here -

- The Argus-eye watches(English)

Basically the article demonstrates the usages of argus client tools, and gives brief explanation about them, the article is written by Ralf Spenneberg and it should be good read for people who just started to adapt to argus 3.

Cheers ;]

Argus 3: Statistics for Major Protocols

Most people would like to have macro view of the network, for example how many bytes have been utilized for protocol such as tcp, udp and icmp, or other things like the amount of packets that have been transmitted or received.

Previously in argus 2.x, argus offers racount -ar to generate the general statistics but the option -a is gone in argus 3.x, so how can you generate the network utilization for major protocols? I have shown the usage of racluster previously here for network session reconstruction and now I will demonstrate another example of using racluster.

Before I move on, I would like to rephrase racluster's functions from the man page -

Racluster reads argus data from an argus-data source, and clusters/merges the records based on the flow key criteria specified either on the command line, or in a racluster configuration file, and outputs a valid argus-stream. This tool is primarily used for data mining, data management and report generation.

Here you go, you can cluster or merge the records based on the flow key and it is suitable for data mining, data management and report generation, let's generate the statistical report using protocol as flow key. Notice I specify -m proto in command line below and using -s to print the field I want -

shell>racluster -L0 -m proto -r data.arg3 -s proto trans pkts bytes appbytes -\
tcp or udp or icmp
Proto Trans TotPkts TotBytes TotAppByte
udp 18115 72665 8488022 5430758
tcp 22996 1291078 969152661 895531494
icmp 1089 1933 424733 346837

This is something simple from racluster but you maybe scratching your head to figure how to do it when you are still new with argus 3(in fact I did), it is considered one of the most powerful tool in argus 3 client suite and maybe sooner, I will talk more about it. Hopefully you find it helpful(hint, hint).

Enjoy (;])

Loopback header

For system and network adminstrator who put loopback interface for good use(local proxy and so forth), have you ever thought of looking at the network traffics that pass through it? I took a closer look at the packet capture lately and it looks interesting though on different OS platform. Here's the result that I get by capturing it from Ubuntu linux box -

shell>sudo tcpdump -c 1 -XXttttnni lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes
2008-01-28 15:39:37.190524 IP 127.0.0.1.55555 > 127.0.0.1.33108: P 1689457265:1689461100(3835) ack 1687382572 win 283
0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E.
0x0010: 0f2f 9c2f 4000 4006 9197 7f00 0001 7f00 ././@.@.........
0x0020: 0001 d903 8154 64b3 1271 6493 6a2c 8018 .....Td..qd.j,..
0x0030: 011b 0d24 0000 0101 080a 01ef 6ceb 01ef ...$........l...
0x0040: 6c47 e938 36b7 4cc3 c0dd d673 2e3a cc65 lG.86.L....s.:.e
0x0050: c257 5cd0 7a0f dc7a d2d7 066d eee3 deb8 .W\.z..z...m....

lo is the loopback interface on my Ubuntu box, and you may notice the link type is ethernet which is 14 bytes.

0000 0000 0000 0000 0000 0000 0800

Since there's no source and destination mac address for loopback interface, they are just padded with 0000 0000 0000 0000 0000 0000(12 bytes) and with another 2 bytes(0800) as next layer protocol) so total up 14 bytes. I think that way it is much easier to read and decode.

However when I capture the packet through the loopback interface on FreeBSD, I got this result -

shell>sudo tcpdump -c 1 -XXttttnni lo0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 96 bytes
2008-01-28 15:40:50.695859 IP 127.0.0.1.65304 > 127.0.0.1.6343: UDP, length 304
0x0000: 0200 0000 4500 014c 73f0 0000 4011 07af ....E..Ls...@...
0x0010: 7f00 0001 7f00 0001 ff18 18c7 0138 7937 .............8y7
0x0020: 0000 0005 0000 0001 7f00 0001 0000 0000 ................
0x0030: 0000 2369 09e4 5558 0000 0002 0000 0001 ..#i..UX........
0x0040: 0000 0090 0000 0c8c 0000 0001 0000 0001 ................
0x0050: 0000 0000 0000 0000 0000 0001 3fff ffff ............?...

The loopback interface is lo0 and the link type is null? If you are familiar with IPv4 header(usually it starts with 45 if the header length is 20 bytes(no ip options enabled)). So you can identify that the loopback has the header of 4 bytes -

0200 0000

This tells us that the implementation of both operating system for loopback interface are different. I'm wondering if other packet analysis tools will have problem parsing the packets that are captured from FreeBSD loopback interface. I haven't taken look on other OS yet and maybe you can tell me more about this. Any thoughts?

Enjoy (;])