Network Flow: TopN

There are a lot of questions popping up on and off in argus mailing list regarding how to generate TopN output from argus data, but frequently you may find the questions are too rough to give complete answer.

I'm going to discuss about TopN this time, TopN is the technique that widely used in many industries, what is it for?

TopN is used to retrieve the first N records from the data based on certain object and ordered by its property. Since I'm talking about Network Flow, I would like to make the example using it.

Data: Network Flow Record
Object: Protocol, Network, IP(host), Port, etc
Object Property: Packet Count, Byte Count, etc

Bear in mind that I'm avoiding the use of Flow terminology but layman one so that this example can be understood easily.

If you want to use TopN technique to generate information from the network flow data, first you need to know what you are looking for. Lets go with a simple one -

I want to find out Top 5 IP ordered by Total Packet Count

Total Packet Per IP(host) = (packet send + packet receive) Per IP(host)

Now you run the argus client command to parse the data and generate exactly the result which looks like this -

shell>racluster -M rmon -m saddr -nr testing.arg3 -w - | \
rasort -m pkts -w - | \
ra -L0 -N 5 -s saddr pkts

SrcAddr TotPkts
172.16.1.108 993
193.231.236.41 824
211.185.125.124 178
172.16.1.103 56
211.180.229.190 36

The command above is to generate Top 5 IP ordered by Packet Count. Don't ask me about the command line, it looks complicated for now but that's not my point here, look at the output instead. Host 172.16.1.108 sends or receives 993 pakcets, followed by 193.231.236.41 and so forth.

Now if you want to locate Top 5 IP ordered by Byte Count. You can just run -

shell>racluster -M rmon -m saddr -nr testing.arg3 -w - | \
rasort -m bytes -w - | \
ra -L0 -N 5 -s saddr bytes

SrcAddr TotBytes
172.16.1.108 599949
193.231.236.41 579050
211.185.125.124 18901
172.16.1.103 4964
216.168.224.69 3458

You want to use TopN, you should draft out the TopN output you are looking for, I have seen questions like this -

1. Which is the most active network?
2. Who is the most active sender?
3. Who is the most active receiver(got ddos?)

Or worse,

How can I find out the top talkers?

These kind of questions are too loose, you should at least specify the property, such as most active sender that is ordered by packet count, or most active network that is ordered by byte count and so forth. You have to bear in mind that packet and byte are not going inline, you can have one host sending many small size packets which won't hit TopN byte count at all.

With this kind of idea in mind, you can build the list of TopN which can draw you a good picture of network activeness to solve different issues.

For the next round, I will introduce Traffic Matrix, stay tuned!

Enjoy (;])

Laptop: Alternatives For Security Road Warrior

Which laptop model are you using? This is my question today. I have been looking for laptop of choice for myself, as a security road warrior, I prefer it to be -

1. Lightweight(small/medium size and compact)

2. Black and solid look

3. Miminum 150G of Disk and 4G of Rams

4. Great keyboard touch build(old thinkpad?)

5. 12-14inch display

6. Good battery life

7. FreeBSD/Linux Compatible or I will use VMware/VirtualBox

Currently there are 3 laptop models I have in my mind -

1. Macbook Black

2. Thinkpad T61

3. Dell Xps m1330

What do you think and what's your favorite laptop if you work in security industry, and you need to travel frequently? I'm looking forward for any good suggestion and sharing. Thanks!

Peace ;]

HeX 021 Series

I will start this HeX Zero To One(021) Series in my blog while HeX 2.0 is in active development, and all of them will be imported to HeX Handbook. In future you will see my post with the title prefix of HeX 021: belongs to the series.

Enjoy ;]

HeX: From Zero To One

These days, I have encountered questions like this,

1. How can I be an efficient network security analyst?

2. Is there a quick path or short cut to be one?

3. I'm just system administrator/programmer and don't know crap about security, I'm interested in it but don't know where to start?

4. There are so many resources in the internet, what's the specific knowledge required to be network security analyst so that I can be more focusing on particular subjects?

If you are a student, or just starting to work as network security analyst, I hope this post will shade some lights for you -

I would like to point out 3 posts that I have written -

http://geek00l.blogspot.com/2008/05/hex-handbook.html

http://geek00l.blogspot.com/2007/07/hex-livecd-analogy.html

http://geek00l.blogspot.com/2007/03/netsecanalyst-handbook.html

The HeX System that we are developing is the key to answer almost all your questions, but you might not be able to know where to look at if I throw you HeX liveCD without giving you hints or tips. The HeX Handbook which is derived from my own Network Security Analyst Handbook is actually designed to lead you to the right path.

If you read my post about HeX liveCD analogy, I mentioned this -

The HeX liveCD can only make up to this part(see below) -

Obtain Network Based Data -> Utilizing NSM Based Tools -> Generate Output

The rest depends on how analyst able to perform it -

Output Interpretation -> Output Analysis -> Output Summarization -> Report

Clearly enough, the HeX itself can't do everything for you, you have to help yourself starting from Output Interpretation process.

If you apply the reverse thinking, what are the obstacle you have encountered during Output Interpretation? You have used the tools to generate the output for you, for example - snort, bro-ids or even simple tcpdump. Apparently if you find yourself can't understand those output, you can't interpret them correctly. Now the important question is "Why can't you understand those output?" There are few answers to it -

1. You may not have enough network protocol knowledge.

2. You may not familiar with the tools because different tools tend to generate the output in different ways or results.

3. You may not update yourself with current security trends(follow bug traq, cve and so forth)

4. You are being lazy

Now I flash back again to my Network Security Analyst Handbook post, I have put the book into four sections -

Sec 1 - Net Sec Analyst: The RoadMap
Sec 2 - Net Sec Analyst: The Workflows
Sec 3 - Net Sec Analyst: The Tools
Sec 4 - Net Sec Analyst: The Case Study

For the Section 1 and Section 2, I have elaborated them as -

Network Security Analyst: The RoadMap
What are good foundations and technical knowledge that should be acquired to become good network security analyst? I hope The RoadMap can answer question like that, until now I haven 't seen any books discussing about this topic yet.

Network Security Analyst: The WorkFlows
What are the methodologies and mechanisms that are used by network security analyst to handle their tasks? The routine daily tasks, the automated and manual process of performing analysis, situation handling and so forth.
This is more of how to think or work like a network security analyst. I will try to standardize the common work flows but you are free to extend it to your own way.

If you have gone through Section 1 and 2, you should be able to do this -

Output Interpretation -> Output Analysis -> Output Summarization -> Report

Unfortunately we don't offer these in HeX version 1.x, but this is going to change, we are currently working on integrating things that are discussed in Section 1 and 2 into HeX version 2.x which will be released sometime around June. As Section 3 is already integrated into HeX, you should be able to complete Section 1-3 with HeX, all you need is discipline!

This is not a myth, the HeX Handbook will guide you to complete Section 1-3 using HeX System itself, you don't need more.

For the Section 4, I already have other plan and maybe you can see them in HeX 3.x, who knows.

If you want to learn to be a competent network security analyst, you can start with HeX. It will take you from 0 to 1.

Now I start to think that University should offer this kind of course for students, as far as I know country like Philippine has their Universities offering malware analysis course and therefore you can see a lot of them working in AntiVirus Industry, if our country want to produce competent network security analyst, they should offer security related courses in University. Not wait until they are out of school and busy with works.

Enjoy ;]

Training: Practical Network Flow Analysis

This time, me and spoonfork will bring you our new and upcoming training which is -

Understanding Network Conversations:
Practical Network Flow Analysis

Here's the description of our training -

Network Flow data represents a summary of conversation between two end points. It provides valuable information to assist investigation and analysis of network and security issues. Unlike deep packet inspection, flow data does not rely on packet payloads. Instead the analyst relies on information gathered from packet headers and its associated metrics. This provides the analyst a neutral view of network traffic flow by tracking network sessions between multiple endpoints simultaneously. In addition, having network flow data will provide a better visibility of network events without having the need to perform payload analysis.

With the implementation and deployment of Network Flow technologies, an analyst can discover different types and classes of network activities, be it normal or abnormal. In this training we will show you how to interpret Network Flow data and perform practical Network Flow Analysis.

While high level theory explanations are extremely useful, hands-on exercises are even better. Each chapter is accompanied by practical hands-on exercises such as exporting network flow data from Unix and Cisco-based routers, performing simple operations such as IP accounting, network baselining, and identifying different kinds of network anomalies and attacks.

Who should attend?

Network Security Analyst
Network Administrator
ISP Network Architect
System Administrator

Bonus

+ First 10 registrants get free seat for HITB Conference Kuala Lumpur in October 2008

+ Human Resources Development Fund(HRDF) Claimable

For more information, check it out at -

http://training.hitb.org/flowanalysis/

Cheers (;])

HeX: Handbook

While we are in the active development of HeX 2.0, we will start a side project mainly for documentation purpose. We call it HeX Handbook, the link is here -

https://trac.security.org.my/hex/wiki/HeXHandbook

Currently there's nothing yet, but I will import all the contents from my incomplete Network Security Analyst Handbook to there, and I'm now trying to design the standard template so that whoever want to contribute can follow the template.

If you are using HeX, and you know different way of doing analysis using the tools in HeX, we would like to hear from you. By the way, if you are good in language translation, please do let me know.

Thanks to scholar who always gives me very fruitful input!

Cheers (;])

Como: Installation on Ubuntu

I found this without intention while searching for tool that can convert pcap format to netflow v5 format, its name is Como, the project which is developed by Intel people. If you want to know more about Como, check out their publications here -

http://como.sourceforge.net/publications.php

Here's the quick way to get it installed on Ubuntu -

shell>sudo apt-get install cmake

shell>wget \
http://como.sourceforge.net/download/como-1.5.tar.gz

shell>tar xvzf como-1.5.tar.gz

shell>mkdir build-como

shell>cd build-como/

shell>cmake -DCMAKE_INSTALL_PREFIX=/usr/local/stow/como-1.5 ../como-1.5

shell>make

shell>sudo make install

Once you have it installed, you can run it via command line interface but make sure you have configured its paths and modules. The configuration file is como.conf which can be found under directory /usr/local/stow/como-1.5/etc/como, you can also enable its module in run time. Here's the available modules -

shell>ls -la /usr/local/stow/como-1.5/libexec/como-1.0/

total 596
drwxr-xr-x 2 root root 4096 2008-05-23 10:34 .
drwxr-xr-x 3 root root 4096 2008-05-23 10:34 ..
-rw-r--r-- 1 root root 21281 2008-05-23 10:34 apps.so
-rw-r--r-- 1 root root 18778 2008-05-23 10:34 assoc.so
-rw-r--r-- 1 root root 23131 2008-05-23 10:34 autofocus.so
-rw-r--r-- 1 root root 15151 2008-05-23 10:34 dhcp.so
-rw-r--r-- 1 root root 21425 2008-05-23 10:34 ethtypes.so
-rw-r--r-- 1 root root 25640 2008-05-23 10:34 ewma.so
-rw-r--r-- 1 root root 16434 2008-05-23 10:34 flowcount.so
-rw-r--r-- 1 root root 27356 2008-05-23 10:34 flow-reassembly.so
-rw-r--r-- 1 root root 16169 2008-05-23 10:34 frames.so
-rw-r--r-- 1 root root 20629 2008-05-23 10:34 hwtm.so
-rw-r--r-- 1 root root 14416 2008-05-23 10:34 ipssi.so
-rw-r--r-- 1 root root 14736 2008-05-23 10:34 macssi.so
-rw-r--r-- 1 root root 13290 2008-05-23 10:34 nfexlist.so
-rw-r--r-- 1 root root 16080 2008-05-23 10:34 pattern-search.so
-rw-r--r-- 1 root root 20117 2008-05-23 10:34 protocol.so
-rw-r--r-- 1 root root 27098 2008-05-23 10:34 scanner-detector.so
-rw-r--r-- 1 root root 17405 2008-05-23 10:34 ssid.so
-rw-r--r-- 1 root root 21645 2008-05-23 10:34 superaddr.so
-rw-r--r-- 1 root root 25659 2008-05-23 10:34 topaddr.so
-rw-r--r-- 1 root root 25433 2008-05-23 10:34 tophwaddr.so
-rw-r--r-- 1 root root 21516 2008-05-23 10:34 topports.so
-rw-r--r-- 1 root root 15122 2008-05-23 10:34 trace.so
-rw-r--r-- 1 root root 19080 2008-05-23 10:34 traffic.so
-rw-r--r-- 1 root root 27192 2008-05-23 10:34 tuple.so
-rw-r--r-- 1 root root 20970 2008-05-23 10:34 unknown-ports.so
-rw-r--r-- 1 root root 39547 2008-05-23 10:34 worm-signature.so

To enable any of module at run time, you can just execute -

shell>/usr/local/stow/como-1.5/bin/como topports

It also provides http access to the data but I keep getting the error below while connect to http://localhost:44444 -

Module "" not found in the current configuration

I believe I have the modules configured properly but I can't get rid of the error, anyway that's all for Como and I will write more about it after some testings.

Peace ;]

Argus 3 Release

This is nothing new, argus 3 is finally released after long testing period. Thanks to everyone who involves in the argus 3 development and testing cycle especially Carter. You can download it at -

- http://qosient.com/argus/downloads.htm

If you are using FreeBSD, good new is argus 3 port is available now and you can check out the information about it here -

- http://www.freshports.org/net-mgmt/argus3/
- http://www.freshports.org/net-mgmt/argus3-clients/

I will try out the argus 3 ports on FreeBSD and see how it goes, have fun!

Enjoy ;]

HeX: Hardware Compability List

We started this list a while ago, and if you have tried out HeX and for the basic part it works properly, please update the list or at least email me about it.

https://trac.security.org.my/hex/wiki/Hardwares

So what are the basic things that must work in order to be included in the list, here you go -

- boot properly
- display properly
- ethernet adapter is supported

Please help out to improve the list, thanks!!!!!

Cheers ;]

SecurityDistro: Interview

Thanks to Dakrone who has committed HeX to SecurityDistro which I don't know myself, and interestingly Josh from SecurityDistro sent me the interview questions via email and here's the interview result -

http://securitydistro.com/articles/407/Interview-with-CS-Lee-creator-of-HeX.php

Thanks to Josh for his kindness and free promotion from SecurityDistro.

I would like to thank to all the team members for progressive HeX development, and feel great to have you guys working together with me.

Cheers ;]

Network Flow Analysis: The Tools

I need to keep track of all the network flow analysis tools and study their offerings, this link contains many tools which may be useful for that purpose -

http://www.switch.ch/network/projects/completed/TF-NGN/floma/software.html

Enjoy ;]

Blog: Quick Update

I haven't been updating my blog lately, here's the quick one.

- I'm not in the mood of blogging but learning.

- I'm preparing myself for many things now which I can't tell yet.

- I'm learning network protocols that I'm not familiar with.

- I'm learning the advance usage of wireshark, and I'm glad the presentation slides of Sharkfest are available online here.

- I will spend 2 months of my free time on non-tech stuffs soon, which means I will still online but more for casual browsing, email checking and light reading. I need to be more focus!

- I will still blog even though the mood is not with me.

Cheers ;]

CERT: Vulnerability Analysis Blog

CERT has launched its Vulnerability Analysis Blog which you can find at -

http://www.cert.org/blogs/vuls/

Another useful resource for security professional.

Cheers ;]