Time Machine - Payload Centric

I don't think NSM is the only approach that appreciate the value of full content data, Bro-ids Community do believe and identify that full content data is important when one needs to perform network forensic and analysis, by travelling back to the incident scene with their time machine.

However, full content data collection is a problem when comes to implementation due to some limitations, again Bro-ids community tries to solve the problem with their innovative ideas. Since it is already well mentioned in the link below, I won't be mentioning it here -

http://www.net.t-labs.tu-berlin.de/research/tm/#doc

They have developed the tool called time machine where you can download from their site, I'm pretty interested to run it using my testing server as they state that time machine runs well on gigabits network. I remember I have packets drop when using other network based tools to log the packets.

I'm using FreeBSD as my platform for the setup, I untar the source after downloading it, the installation runs fine with ./configure and make, the developers of time machine encourage users to run time machine from the same directory you compile the source code.

It seems to be trivial at first but you will definitely get clearer picture after reading the how-to. I just configure the file - tm.conf before running tm. Here's my config -

# Example TM configuration file
# $Id: tm.conf 107 2006-11-11 03:11:40Z gregor $

main {
logfile "tm.log";
# these directories must exist when tm starts!
workdir "/nsm/tm/";
indexdir "indexes/";
queryfiledir "queries/";

log_interval 10;
device "fxp1";
# read_tracefile "trace.pcap";
# filter "not port 80";
# bro_connect_str "localhost:47757";
console 1;
conn_timeout 180;
rmtconsole 1;
rmtconsole_port 42042; # 42042 is default
rmtconsole_listen_addr 127.0.0.1; # 127.0.0.1 is default

}

class "all" {
filter "";
precedence 5;
cutoff 15k;
disk 50g;
filesize 1000m;
mem 300M;
pkts_to_disk 2;
}

I need to create few directories before running tm, the pcap log will archive by itself after reaching 1GB(1000m). The network interface that I'm monitoring is fxp1. You can tweak the cutoff value based on your need. I don't set any bpf filter as I want full content data.

shell>mkdir -p /nsm/tm/indexes
shell>mkdir /nsm/tm/queries

Now I just need to execute

shell>./tm -c ./tm.conf
timemachine version 20061111-0
capture started
CLI console thread started
tm@testing.org#

I navigate /nsm/tm and the pcap is logged properly, most importantly there's no packets loss, here's some of the entries in my log - /nsm/tm/tm.log

shell>cat /nsm/tm/tm.log
1163664440.846803 stats: 46061451/0 recvd/dropd P (0.00) 46061371 Pkts, 32689159206 B, 58.6 Mbit/s
1163664440.847054 class_all: 3978554364 11891524 28708322158 34132765 300526337 877840 1163663733.592655 3854246971 11013684 1163654237.963090
1163664440.847312 stats_conns: 36781 conns

shell>ls -la /nsm/tm
total 4126504
drwxr-xr-x 4 root sguil 512 Nov 16 16:20 .
drwxr-xr-x 18 sguil sguil 1024 Nov 16 13:17 ..
-rw-r--r-- 1 root sguil 1048575623 Nov 16 14:15 class_all_00000001
-rw-r--r-- 1 root sguil 1048575410 Nov 16 14:57 class_all_00000002
-rw-r--r-- 1 root sguil 1048574833 Nov 16 15:40 class_all_00000003
-rw-r--r-- 1 root sguil 1048575974 Nov 16 16:20 class_all_00000004
-rw-r--r-- 1 root sguil 28655616 Nov 16 16:21 class_all_00000005
drwxr-xr-x 2 root sguil 1536 Nov 16 16:04 indexes
drwxr-xr-x 2 root sguil 512 Nov 16 12:38 queries
-rw-r--r-- 1 root sguil 368285 Nov 16 16:21 tm.log

I run bpfstat as well and apparently there's no packets drop so far. I'm currently running the time machine for few days to test it. Maybe I will perform stress testing when I have time.

Time machine is aimed to solve the storage and performance issue while retaining the all possibilities to perform network forensic and analysis functions. At the moment the communication of time machine with bro-ids is done via brocolli and they will integrate time machine into bro-ids in future.

I'm sured time machine is one of efficient option available for you to collect full content data. We have already seen many protocol header centric solutions - netflow, argus and etc, time machine is packet payload centric instead.

Enjoy (:])