Regex - Magic for NetSe[x|c]Anal(yst)?

There are certain knowledge that should be armed by network security analyst, however sometimes we don't know what should we learn and master when everything seems to be interesting, plus we are always lack of time(this is important factor indeed). Thus we need a clear scope and defined roadmap so that one can quickly pick up and becoming a decent network security analyst.

I will discuss the complete roadmap for network security analyst in my future blog post, while I think it won't apply to everyone, I hope it helps one to become better network security analyst. Hereby I would like to talk about one of important knowledge that considered crucial to most network security analyst - REGEX.

Regular expression is very powerful tool when one master it. I won't be explaining what is regex but you should already know it is mainly used for pattern matching most of the time. Not many network security analysts think that they should learn regex as it doesn't seem to be important.

As for me, I first learned about regex when using shell tool called egrep, egrep is very efficient tool when one need to process large log files and extract certain strings out of it. I realized that only by mastering regex then I can actually fully utilize it. However it doesn't seem so much related or applied to network security analyst job at that moment.

Later when I'm adapted to network security monitoring, there are many network security tools that I have to deal with. Surprisingly most of them have something to do with regex. Hereby I will show few examples of them, let's look at ngrep -

shell>ngrep -I ./bugger.pcap -i 'ping[0-9]\:'

U 192.168.0.35:25555 -> 1.2.3.4:14730
d1:ad2:id20:.Q.ÐÏE¬8nÛ.?ÍYp¸£.ñ.e1:q4:ping1:t8:M..]`.&"1:y1:qe
####
U 192.168.0.35:25555 -> 1.2.3.4:27843
d1:ad2:id20:.Q.ÐÏE¬8nÛ.?ÍYp¸£.ñ.e1:q4:ping1:t8:só.ÔP£..1:y1:qe

Ngrep allows one to use extended regex when performing pattern matching on packet payloads(application layer). But it only performs matching on single packet instead of the complete network session, this maybe bypassed using fragmentation technique, no worry we have flowgrep for the job instead which will reassemble the network traffic and examine to avoid evasion.

Yet that's not the end, the most popular open source nids - snort, it needs to be compiled with pcre. Snort has very powerful signatures matching engines, and if we take a closer look at more complicated snort rules -

alert udp $EXTERNAL_NET any -> $HOME_NET 699 (msg:"EXPLOIT RADIUS registration vendor ATTR_TYPE_STR overflow at tempt"; content:"|01|"; depth:1; content:"|01 01 1A|"; depth:3; offset:32; content:"|00 00 15 9F|"; depth:4; of
fset:36; byte_test:1,>,30,1,relative; isdataat:29,relative; pcre:"/^\x01.{23}(\x25|\x26).{15}(\x0A|\x34)/smi";
reference:bugtraq,12759; reference:cve,2005-0699; reference:nessus,19120; classtype:attempted-admin; sid:3540; rev:3;)

alert udp $EXTERNAL_NET any -> $HOME_NET 5060 (msg:"EXPLOIT Ethereal SIP UDP CSeq overflow attempt"; content:"CSeq|3A|"; nocase; isdataat:16,relative; content:!"|0A|"; within:16; pcre:"/^CSeq\x3A\s*[^\nA-Za-z]*[A-Za-z][^\n]{16,}/smi"; reference:bugtraq,13504; reference:cve,2005-1461; reference:nessus,18986; reference:url,www.ethereal.com/news/item_20050504_01.html; classtype:attempted-dos; sid:3677; rev:5;)

Again it is regex thingy, what can I say more? Lets take a look at another nids alternative - bro-ids. Bro-ids has many policy scripts and it either contains signatures for worms or network protocols(application protocol can be very identical sometimes by looking at packet payloads). Lets navigate some of its signatures -

signature s2b-103-7 {
ip-proto == tcp
src-port == 27374
event "BACKDOOR subseven 22"
tcp-state established,originator
payload /.*\x0D\x0A\[RPL\]002\x0D\x0A/
}

signature s2b-106-8 {
ip-proto == tcp
src-port == 80
dst-port == 1054
header tcp[8:4] == 101058054
header tcp[13:1] & 255 == 16
header tcp[4:4] == 101058054
event "BACKDOOR ACKcmdC trojan scan"
tcp-state stateless
}

What you see in bold is regex to match the packet payload to identify subseven backdoor. In fact this is the one of snort signature that converted for Bro-ids compatibility. Last example that I would like to show is pads, the passive asset detection system to identify what network services that are running in your network which will be integrated to next version of sguil. The following are pads signatures -

www,v/Apache/$1//,Server: Apache\/([\S]+)[\r\n]
smtp,v/Postfix SMTP//$1/,^220 ([-.\w]+) ESMTP Postfix
bit,v/Bittorrent///,^\x13BitTorrent\x20protocol

Going through so many network security tools, one should already understand the important of regex which is widely utilized in the network security arsenal. I know most of the network security analyst wannabe that actually considering mastering network protocols(tcp/ip) or finishing their degree in university or college should do enough, but you should seriously think of taking yourself to next level, be competitive and serious network security analyst!

Trust me, regex can do more magic than you can imagine!!!!!

Enjoy (;])

P/S: I'm taking this as serious post, that's why it takes me so long to post it, it's my 300th blog post and the last one in year 2006, happy new year 2007!!!!!