Friday, December 29, 2006

Regex - Magic for NetSe[x|c]Anal(yst)?

There are certain knowledge that should be armed by network security analyst, however sometimes we don't know what should we learn and master when everything seems to be interesting, plus we are always lack of time(this is important factor indeed). Thus we need a clear scope and defined roadmap so that one can quickly pick up and becoming a decent network security analyst.

I will discuss the complete roadmap for network security analyst in my future blog post, while I think it won't apply to everyone, I hope it helps one to become better network security analyst. Hereby I would like to talk about one of important knowledge that considered crucial to most network security analyst - REGEX.

Regular expression is very powerful tool when one master it. I won't be explaining what is regex but you should already know it is mainly used for pattern matching most of the time. Not many network security analysts think that they should learn regex as it doesn't seem to be important.

As for me, I first learned about regex when using shell tool called egrep, egrep is very efficient tool when one need to process large log files and extract certain strings out of it. I realized that only by mastering regex then I can actually fully utilize it. However it doesn't seem so much related or applied to network security analyst job at that moment.

Later when I'm adapted to network security monitoring, there are many network security tools that I have to deal with. Surprisingly most of them have something to do with regex. Hereby I will show few examples of them, let's look at ngrep -

shell>ngrep -I ./bugger.pcap -i 'ping[0-9]\:'

U ->
U ->

Ngrep allows one to use extended regex when performing pattern matching on packet payloads(application layer). But it only performs matching on single packet instead of the complete network session, this maybe bypassed using fragmentation technique, no worry we have flowgrep for the job instead which will reassemble the network traffic and examine to avoid evasion.

Yet that's not the end, the most popular open source nids - snort, it needs to be compiled with pcre. Snort has very powerful signatures matching engines, and if we take a closer look at more complicated snort rules -

alert udp $EXTERNAL_NET any -> $HOME_NET 699 (msg:"EXPLOIT RADIUS registration vendor ATTR_TYPE_STR overflow at tempt"; content:"|01|"; depth:1; content:"|01 01 1A|"; depth:3; offset:32; content:"|00 00 15 9F|"; depth:4; of
fset:36; byte_test:1,>,30,1,relative; isdataat:29,relative; pcre:"/^\x01.{23}(\x25|\x26).{15}(\x0A|\x34)/smi";
reference:bugtraq,12759; reference:cve,2005-0699; reference:nessus,19120; classtype:attempted-admin; sid:3540; rev:3;)

alert udp $EXTERNAL_NET any -> $HOME_NET 5060 (msg:"EXPLOIT Ethereal SIP UDP CSeq overflow attempt"; content:"CSeq|3A|"; nocase; isdataat:16,relative; content:!"|0A|"; within:16; pcre:"/^CSeq\x3A\s*[^\nA-Za-z]*[A-Za-z][^\n]{16,}/smi"; reference:bugtraq,13504; reference:cve,2005-1461; reference:nessus,18986; reference:url,; classtype:attempted-dos; sid:3677; rev:5;)

Again it is regex thingy, what can I say more? Lets take a look at another nids alternative - bro-ids. Bro-ids has many policy scripts and it either contains signatures for worms or network protocols(application protocol can be very identical sometimes by looking at packet payloads). Lets navigate some of its signatures -

signature s2b-103-7 {
ip-proto == tcp
src-port == 27374
event "BACKDOOR subseven 22"
tcp-state established,originator
payload /.*\x0D\x0A\[RPL\]002\x0D\x0A/

signature s2b-106-8 {
ip-proto == tcp
src-port == 80
dst-port == 1054
header tcp[8:4] == 101058054
header tcp[13:1] & 255 == 16
header tcp[4:4] == 101058054
event "BACKDOOR ACKcmdC trojan scan"
tcp-state stateless

What you see in bold is regex to match the packet payload to identify subseven backdoor. In fact this is the one of snort signature that converted for Bro-ids compatibility. Last example that I would like to show is pads, the passive asset detection system to identify what network services that are running in your network which will be integrated to next version of sguil. The following are pads signatures -

www,v/Apache/$1//,Server: Apache\/([\S]+)[\r\n]
smtp,v/Postfix SMTP//$1/,^220 ([-.\w]+) ESMTP Postfix

Going through so many network security tools, one should already understand the important of regex which is widely utilized in the network security arsenal. I know most of the network security analyst wannabe that actually considering mastering network protocols(tcp/ip) or finishing their degree in university or college should do enough, but you should seriously think of taking yourself to next level, be competitive and serious network security analyst!

Trust me, regex can do more magic than you can imagine!!!!!

Enjoy (;])

P/S: I'm taking this as serious post, that's why it takes me so long to post it, it's my 300th blog post and the last one in year 2006, happy new year 2007!!!!!

Sunday, December 24, 2006

Pro OpenSSH

I have finished reading it while going to Border Book Store with the infamous Seymour at Queensbay. While I'm not a book reviewer, but I would like to recommend this book to anyone especially system administrators who use ssh in their daily administration task.

The book is written by Michael Stahnke. At first glance of this book, I thought it might be the book for people who want to understand about ssh protocols in depth(Pro OpenSSH), however I'm totally wrong, this book is not about the protocol itself but more of practical implementation and deployment of OpenSSH services. The book itself is not for OpenSSH power users only, but it introduces OpenSSH from basic to advance topics so that anyone can pick up OpenSSH quickly. The explanation of each OpenSSH options is not just man page kind only but rather showing various useful examples that can improve your ssh experience. It also introduces many usages of ssh management tools such as ssh-agent and so forth.

The automated task via ssh plus shell scripting also comes into handy, people can find it effective working in ssh environment since many administration jobs can be automated as long as you know the tricks with creativity.

The last chapter demonstrates ssh clients on different OS platforms(if I'm not forgotten), this may help people who want to learn how to use ssh on Windows Platform. It also guides you on how to install and run Windows SSH server. This is pretty neat when you need secure remote access to Windows Server without using other crappy services.

Overall the book is good, but actually I'm looking for kind of book which explains ssh protocols breakdown analysis to sharpen my skill. Again I seriously recommend this book to system administrators who work in *nixes environment. Book like this can be very useful for people who need to practice it in time.

Enjoy :]

Merry Christmas

Hiho ..... hohohohohohoohohohoh .....

To all people around the globe,

Merry Christmas & Joys to the world!!!!!


P/S: By the way I have a christmas wish that wanna tell - I wish to have my own "Network Security Analyst Lab & Library" when I able to buy all related books and accesories :P

Tuesday, December 19, 2006

Bro-IDS: Enable Full Content Data Logging

To enable Bro-ids full content data logging so that you can perform network forensic, just set it in bro.cfg under bro configuration directory(etc) will do -


You can just disable it by setting it to NO, the pcap file will be stored at the logs directory.

shell>file trace.hostname.06-12-19_00.36.41
trace.hostname.06-12-19_00.36.41: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 8192)

However in future, I think time machine will replace this for full content data logging management.

Enjoy :]

Sunday, December 17, 2006

3Com 3226: Enable Port Mirroring

If you have 3Com Network Switch 3226 Model, and you would like to monitor your network, it does provide port mirroring feature. In order to enable it, you can login to the console via telnet, then execute commands accordingly to the screenshot below -

feature -> rovingAnalysis -> add|remove|start|stop|summary

3Com uses Roving Analysis as the term for the port mirroring, you will have to specify monitor port and analyzer port. Monitor port is the port you will want to monitor and analyzer port is the port to mirror traffic on monitor port. From my example I will monitor port 1 and its network traffic will be mirrored to the analyzer port which is port 25. Once I start monitoring, you can view the summary where roving analysis is enabled.

By now you can just plug in your IDS sensor or traffic collector to port 25 of the switch and start your network security monitoring.

Enjoy :]

P/S: I by no mean promoting or selling 3Com product, this is just to help in case one has the same device or as my own reference.

Saturday, December 16, 2006

Bro Offline Packet Analysis(DPD Enabled)

Bro-ids is a beast, you will have to spend some times to actually understand how it works, or maybe just understand it partially. There's no good guide that you can find regarding Bro-ids, the only thing you can do is either subscribing to the mailing list, or either reading Bro-ids Wiki or maybe reading its codes(Ouch). The documentation that comes with Bro-ids is rather old. I think most of people out there are using Snort, it is lightweight, snort related documentation and books are everywhere too. I will have to admit that Bro-ids is not very easy to use but you can find it very powerful from time to time. Anyway you can download it at here, to install it -

shell>tar xvzf bro-1.2-devel.tar.gz

shell>cd bro-1.2-devel

shell>./configure --prefix=/usr/local/stow/bro-1.2-devel

shell>make && make install

You will have to run make install-brolite so that configuration can be done easily. Once you have finished the configuration, now let's navigate the installed bro. The reason why I choose to install bro in non-standard directory because it will be more easy for us when we need to either remove or upgrade it next time.

shell>ls -la /usr/local/stow/bro-1.2-devel/
total 38
drwxr-xr-x 16 bro wheel 512 Dec 15 19:33 .
drwxr-xr-x 16 root wheel 512 Dec 12 14:35 ..
drwxr-xr-x 2 bro wheel 512 Dec 6 09:17 archive
drwxr-xr-x 2 bro wheel 512 Dec 6 10:00 bin
drwxr-xr-x 5 bro wheel 512 Dec 6 10:00 doc
drwxr-xr-x 2 bro wheel 512 Dec 6 12:23 etc
drwxr-xr-x 2 bro wheel 512 Dec 6 09:57 include
drwxr-xr-x 2 bro wheel 512 Dec 6 09:57 lib
drwxr-xr-x 4 bro wheel 5632 Dec 15 16:04 logs
drwxr-xr-x 3 bro wheel 512 Dec 6 10:08 perl
drwxr-xr-x 4 bro wheel 4096 Dec 15 19:34 policy
drwxr-xr-x 2 bro wheel 512 Dec 6 09:17 reports
drwxr-xr-x 2 bro wheel 512 Dec 6 10:08 scripts
drwxr-xr-x 4 bro wheel 512 Dec 6 09:14 share
drwxr-xr-x 2 bro wheel 512 Dec 6 12:42 site
drwxr-xr-x 2 bro wheel 512 Dec 15 16:04 var

I won't be going through every directory and explaining them one by one, however you should look at site which is used to store your site configuration, you should check and tune the configuration of the files in site directory especially the one with your hostname as prefix. By default brolite is enabled to load all the default policies.

@load brolite

One of the thing you can do is enable the Snort signatures matching by uncomment the following line -

# To run signatures, uncomment the following line.
@load brolite-sigs

The next directory that are important is etc(configuration directory), you may find most of bro path variables are stored in bro.cfg in etc directory. Then comes to policy script, this directory needs a lot of tuning depends on your network environment. You can actually checkout brolite.bro script inside policy directory to see what is loaded by default. Other directories that are shown pretty self descriptive so I won't touch on that.

Since this post is about Bro-IDS Offline Packet Analysis, I will just talk about it. To actually do it, I will have to add or append the following part to .bash_profile since my environment variables are stored there(I'm using bash), you should change it if you use different shell.



Since I don't have clue for the first time how I can use bro with DPD(dynamic protocol detection), I refer to the brolite.bro to get to know which analyzers are needed if I want dynamic protocol detection. Here's the entry of brolite.bro

## Dynamic Protocol Detection configuration
# This is off by default, as it requires a more powerful Bro host.
# Uncomment next line to activate.
const use_dpd = T;

@ifdef ( use_dpd )
@load dpd
@load irc-bot
@load dyn-disable
@load detect-protocols
@load detect-protocols-http
@load proxy

# By default, DPD looks at all traffic except port 80.
# For lightly loaded networks, comment out the restrict_filters line.
# For heavily loaded networks, try adding addition ports (e.g., 25) to
# the restrict filters.
redef capture_filters += [ ["tcp"] = "tcp" ];
# redef restrict_filters += [ ["not-http"] = "not (port 80)" ];

The default offline packet analysis can be done by using mt.bro which is located at policy directory. Here's the content of my mt.bro -

# $Id: mt.bro 340 2004-09-09 06:38:27Z vern $

@load alarm
@load dns-lookup
@load hot
@load frag
@load tcp
@load scan
@load weird
@load finger
@load ident
@load ftp
@load login
@load portmapper
@load ntp
@load tftp
@load profiling
# Loading DPD
@load dpd
@load irc-bot
@load dyn-disable
@load detect-protocols
@load detect-protocols-http
@load proxy

You should see I have added the dynamic protocol detection(DPD) to mt.bro so that I can use the latest DPD when doing offline traffic analysis.

Running bro offline packet analysis
shell>bro -f tcp -r 060915-Event.pcap mt

After bro is successfully executed, you may find all the logs in current directory.

shell>ls -la *.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 alarm.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 conn.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 ftp.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 irc-bots.log
-rw-r--r-- 1 root wheel 104 Dec 11 14:01 irc-bots.summary.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 irc.detailed.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 irc.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 notice.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 prof.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 signatures.log
-rw-r--r-- 1 root wheel 0 Dec 11 14:27 weird.log

Dynamic Protocol Detection is still new or considered in early developemet, we will see more and more protocols will be supported in future. Interestingly, you can print all the bpf filter expressions of mt.bro

shell>bro -f tcp mt print-filter
(((((((((((ip[6:2] & 0x3fff != 0) and tcp) or (port 111)) or (tcp[13] & 7 != 0)) or (port 6667)) or (port finger)) or (port telnet or tcp port 513)) or (udp port 123)) or (port ftp)) or (tcp port 113)) or (udp port 69)) or (port 6666)

By the way I'm using the wrong term - Offline Packet Analysis, Bro doesn't work that way, it is more of Offline Traffic Analysis instead as it analyses the full network stream instead of packet wise.

Enjoy (;])

Qemu Network on Linux

If some of you have downloaded my OpenBSD Sguil qemu image, and you happen to use linux, I suggest you take a look at this post which explain how to get qemu networking works on Linux. By the way if you like the work of Snort-Inline, just drop by to the blog and say hi to Victor.

Thanks to Victor, this is the late post from me indeed.

Cheers :)

Monday, December 11, 2006

I know idiot is helpless

I'm pretty dissapointed about SecurePenang. First of all - the speaker from Itrain is just another CEH instructor that been trained to be a Certified Hacker who seriously doesn't know much about the ethic of being Security Professional.

The first event of the day is Wardriving, the funny thing about the speaker is that he told us the best card for Wardriving is Orinoco - that's definitely not true as it depends on the tools and drivers that written for the networking hardware. By the way the WarDriving is pretty boring with whatever stumbler.

Coming to the Presentations that done by Symantec and Microsoft, which don't do any better, everyone feel sleepy and we all know it is boring when comes to vendors talk. Then Niser gaves the Presentation regarding Malaysia Security Outlook, this is by far the best presentation in the day, I don't want to comment anything about it but just one, building threat statistic based on incident reporting of companies is totally inaccurate, I think most of companies prefer things under the umbrella instead of sun. Please deploy sensors at ISP layer instead. I don't think DDOS cases equal to none in a quarter of the year.

The last hacking demo - this is the big woo. The so called certified hacker setup his own web server and I think he added the host mapping to the hosts file - to his own web server with public IP. Then he uses nmap scanning to get all the opened port, then telling everyone that Maybank web server has so many ports opened including port 135 abd 139, therefore he used the script kiddies tool to launch dcom exploits and successfully getting into the server. He even shows how he able to retrieve file and so forth using tftp as well as defacing the website.

I definitely agree that there's no harm to perform live hacking demo, it should creates awareness among the audience to awake how easy your operating system can be compromised. However you should have told the audience that you are actually hacking into the demo server instead of Maybank web server, the certified hacker didn't explain and telling the audience he is hacking into demo server but Maybank Web Server. Almost 90%(can tell as most of audience are non-technical) of them believe that it is real Maybank web server. He is misguiding all the audience by showing how l33t he is, he even stated that there's no firewall or ids and they won't know what he has done(dumbass). Everyone says "WOW" and clapping hands like nobody.

Then when comes to the end of the demo, while people asking him question personally such as is he able to get into Maybank databases? Then he told that Maybank has Multilayer Network Structure and by hacking the Web Server you will have to dig out where the Database Server is located.

At the end, he still want people to believe that he(l33t) is actually hacking into Maybank Server. Remember your audience is public, you shouldn't deliver faulty information, what a breach!

Mike <- You are not qualified as ethical hacker. You just an idiot without brainer! I read from the site the Live Hacking Demo should be done by Scan Associate people and I'm wondering whether he is from Scan. The coolest statements of him - you have greater reputation if you have more bots. I bet he doesn't know the hacker communities enough. The second interesting statement is "I have never been infected by computer viruses" even I have used Windows for so long, and my antivirus program is disabled by default!" - Wow, he is definitely gosu alike.

My words to MIRC - The SecurePenang event sucks, you shouldn't have brought anyone like this to present or deliver.

P/S: I should kick Mike in the nutsack!!!!!!!! :P

Thursday, December 07, 2006

Penang ICT Week

I will be going to the Third Event of Penang ICT Week which is Secure Penang that will be held on 8th of December(Tomorrow), so it would be great if any of you are going to the event and we can most properly talked more about the Network Security Scene in Malaysia. There will be Scan Associate and Niser guys around so I hope to see some familiar faces there.

You can find the event detail here.

Anyway ping me if you are going.

Cheers :]

Wednesday, December 06, 2006


Working as sysadmin means that you will have to watch your puter screen most of the time, thus it is good to setup prefferable working environment. While you can't change your office layout, I'm sure you can change your screenie which don't kill. Most of my friends know that I prefer simple and slick style over anything, and I can't resist to show off my Fluxbox screenshot - it is too sexy to be true and serves as perfect workspace at least for me.

Thanks to tenner for his fluxbox style - litebox, you always rox!

Cheers ;)

P/S: Please learn to appreciate beautiful things :P

Tuesday, December 05, 2006


I think there are lots of people doing the same thing apparently, I just came across this new tool that released by honeynet community - honeysnap. It is very neat as it automated the process of pcap analysis and generating the analysis report which ease the job of analyst.

To avoid doing the same thing over and over again, I have my own shell script that doing similar stuffs which used to analyse pcap files but more of following NSM model. I make use of tools like capinfos, tcpdstat, argus clients, and some other tools to generate analysis report when I have to perform generic analysis on pcap files. Don't expect me to release that as I have to keep something on my own since I guess I have already shared so much of my stuffs indeed.

Anyway honeysnap should be good learning tool for network security analyst, you can check out the report output of honeysnap here. I have seen more and more applications looking at statistical and session data analysis now, is this the hype of NSM?

I would expect this in future - hire NSM analyst instead of IDS analyst

Cheers :]

Monday, December 04, 2006


I wonder why I put ragrep as a title, but I think this may help someone in case they are googling about the usabilities of ragrep, I hardly find any when I tried googling. So what's the use of ragrep, I think you should have idea by looking at the "name". Heard about flowgrep or ngrep, while ngrep is packet based and flowgrep is flow based which introduces more accuracies, argus offers you the tools that doing about the same thing.

While I have done lots of flow analysis using argus client, I don't do much on the payload wise, however you can actually do so with argus tool - ragrep. In order to use ragrep, you will have to capture user data bytes, this can generate expensive argus data if you are running on high speed network, but it is not bad idea to use ragrep if you have stored full content data, or at least it can tell you whether certain commands(http/ftp commands/codes) have been successfully executed in your network by just storing first 10-15 data bytes of connection. To capture user data bytes, you will have to configure argus with the following variables in argus.conf -


Or you can run argus with -U option, this is good when you want to perform offline analysis with generic pcap files. I usually do this -

shell>argus -U 15 -r test.pcap -w test.argus

Once you have data in argus format, you can check the ftp data connection close with transfer complete by running this -

shell>ragrep -i -e '226 Transfer' -r test.argus -nn

To increase the accuracy of it, you can examine the tcp state changes with -z option and checking source/destination bytes as well as duration of the flow. Racluster can be used to aggregate the record with the same pair of src/dst ip/port to avoid confusion with specific flow status interval timer. Remember smtp and ftp can be tricky to look at in the flow record when there's large bulk of data transfer thus it's good to aggregate this kind of flow record.

Enjoy (;])

Sunday, December 03, 2006

More ...

I have more things to read, but less things to blog, thus you hardly see me updating my blog these days. However I will still keep this blog going, currently I'm trying to figure out my own generic flow analysis research. I have to strengthen my knowledge of statistic and network protocols to form a better model when comes to analyse network flow.

By the way I'm setting up Sguil Demo Server of my own, the demo server will be available soon as well as the VMware image of Sguil.

Cheers :]