OpenBSD - fstat vs lsof

I'm all time lsof users when I need to list open files by the process that running on the system, however when I get lsof installed on OpenBSD and try to run it, I get an error which is real bad - Segmentation fault(core dumped).

shell>pkg_add ${PKG_PATH}lsof-4.75p0

shell> lsof
Segmentation fault (core dumped)

Then I turn to fstat which is the native tool that listing open file handles of running process. When I run it against process ID 25166, it shows me the INUM - inode number of the file and with it's mount point. But it seems that it only shows the parent path of the files and doesn't show the actual files but it's inode number only, here we see the advantage of using lsof over fstat alone.

shell>fstat -p 25166
USER CMD PID FD MOUNT INUM MODE R/W DV|SZ
www httpd 25166 root /var 37892 drwxr-xr-x r 512
www httpd 25166 wd /var 37892 drwxr-xr-x r 512
www httpd 25166 0 / 1432 crw-rw-rw- r null
www httpd 25166 1 / 1432 crw-rw-rw- w null
www httpd 25166 2 /var 38655 -rw-r--r-- w 14472
www httpd 25166 3 /var 38657 -rw-r--r-- w 14993
www httpd 25166 4 / 1430 crw-rw-rw- rw crypto
www httpd 25166 15 /var 38655 -rw-r--r-- w 14472
www httpd 25166 16* internet stream tcp 0xd3ae6000 *:80
www httpd 25166 17 /var 38656 -rw-r--r-- w 13091

How I gonna locate the inode, maybe I can use icat or ils in sleuthkit, but doesn't it sound overkill to locate open file by using Forensic tool, I quickly check the man page of OpenBSD and luckily I found this native tool in OpenBSD - ncheck_ffs. Ncheck_ffs is used to generate the file name from inode number,

If I want to just have quick checking on process 25166 and for example one of inode number is 37892, since my /var is mounting at /dev/wd0e, I can just run.

shell>ncheck -i 37892 /dev/wd0e
/dev/rwd0e:
37892 /www/.

However that's not effective since I have to check on it's inode manually and it's time consuming for single process with lots of inodes locating. Then I think of better idea where I generate all the file names based on it's inode numbers with ncheck_ffs so that I can reference later. But before that I need to identify my filesystem by checking my /etc/fstab or using df.

shell>df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/wd0a 245M 34.8M 198M 15% /
/dev/wd0g 145M 2.0K 138M 0% /home
/dev/wd0h 3.6G 1.2G 2.2G 35% /nsm
/dev/wd0d 245M 2.0K 233M 0% /tmp
/dev/wd0f 2.5G 1013M 1.3G 42% /usr
/dev/wd0e 1008M 8.2M 949M 1% /var

Creating directory to store all inodes reference file

shell>mkdir /nsm/ALLINODE

Generate file names with it's inode numbers for all filesystems.

shell>ncheck -m /dev/wd0a > /nsm/ALLINODE/root_inode
shell>ncheck -m /dev/wd0d > /nsm/ALLINODE/tmp_inode
shell>ncheck -m /dev/wd0e > /nsm/ALLINODE/var_inode
shell>ncheck -m /dev/wd0f > /nsm/ALLINODE/home_inode
shell>ncheck -m /dev/wd0g > /nsm/ALLINODE/usr_inode
shell>ncheck -m /dev/wd0h > /nsm/ALLINODE/nsm_inode

shell>cd /nsm/ALLINODE/

shell>cat root_inode tmp_inode var_inode \
home_inode usr_inode nsm_inode > all_inode

I'm now having all the inode numbers with it's file names in all_inode. I can refer back later, to ease my work, I decide to write a crap script to check on the process as I hate doing the same thing again and again. I named my script as iREFERENCE.sh, below it's my script content.

##################Script Starts#####################

#!/bin/ksh

# Fstat Enhanced Script to view the file with it's correlated inode
# Written by geek00L[20060302]

IRF=/nsm/ALLINODE/irf.sys
fstatmp=/nsm/fstat_tmp

# Creating Inode Reference File

if [[ -s /nsm/ALLINODE/all_inode ]];
then
print "\nInode reference file exists!!!!! Continue checking process\n"
else
mkdir /nsm/ALLINODE
ncheck -m /dev/wd0a > /nsm/ALLINODE/root_inode
ncheck -m /dev/wd0d > /nsm/ALLINODE/tmp_inode
ncheck -m /dev/wd0e > /nsm/ALLINODE/var_inode
ncheck -m /dev/wd0f > /nsm/ALLINODE/home_inode
ncheck -m /dev/wd0g > /nsm/ALLINODE/usr_inode
ncheck -m /dev/wd0h > /nsm/ALLINODE/nsm_inode
cd /nsm/ALLINODE/
cat root_inode tmp_inode var_inode home_inode usr_inode nsm_inode \
> all_inode
nawk '{ print $8, $9 }' all_inode > irf.sys
fi

ps auxww | nawk '{ print $1, $2, $11 }'
echo "Which process you want to check it's listed open file?"
read process

if [[ -x /usr/bin/fstat ]];
then
fstat -p $process | nawk '{ print $6 }' | egrep '^[0-9]' > $fstatmp
fi

for i in $(cat /nsm/fstat_tmp);
do
grep "^$i " $IRF >> /nsm/fstat_$process
done

echo "The open files by the specified process is stored in /nsm/fstat_$process."

##################Script Ends#####################

To use the script,

shell>./iREFERENCE.sh

Inode reference file exists!!!!! Continue checking process

USER PID COMMAND
root 1 /sbin/init
root 11508 syslogd:
_syslogd 32431 syslogd
root 18993 pflogd:
_pflogd 31619 pflogd:
www 4185 httpd:
root 13732 sendmail:
www 8854 httpd:
www 25166 httpd:
www 13886 httpd:
www 31657 httpd:
www 3031 httpd:
root 14993 /usr/sbin/sshd
root 14660 cron
root 32610 sshd:
root 7535 sshd:
root 3563 /bin/ksh
root 15570 -ksh
root 10236 -ksh
root 8500 /bin/ksh
root 6397 /bin/ksh
root 29120 ps
root 15303 -ksh
root 12908 script
root 485 script
root 31694 /usr/libexec/getty
root 8702 /usr/libexec/getty
root 5759 /usr/libexec/getty
root 6144 /usr/libexec/getty

Which process you want to check it's listed open file?
25166

The open files by the specified process is stored in /nsm/fstat_25166.

shell> cat /nsm/fstat_25166
37892 /www/.
37892 /lib/apache/modules/.
37892 /www/.
37892 /lib/apache/modules/.
1432 /dev/null
1432 /dev/null
38655 /www/logs/error_log
38657 /www/logs/access_log
1430 /dev/crypto
38655 /www/logs/error_log
38656 /www/logs/ssl_engine_log

Remember the /var is not shown because it won't show the parent path, especially since I ignore the / reference inode in ncheck_ffs without using -a option when generating all inodes. / is always with the inode of 2 so we can just ignore it. And if you are not too sured about where is parent directory of the file is, find command is your friend or you can just run fstat -p again to check it's mount point and you will get it revealed anyway. This script is only applicable on OpenBSD and I haven't tried out on other OS, and you are free to use and modify it. Againt I will upload the script to my dissectible.org site.