Sunday, November 19, 2006

File System Full

I'm running into very funny problem where my /var file system is full, I observe this while looking at my log -

Dec 11 18:59:17 trinity /bsd: uid 0 on /var: file system full
Dec 11 18:59:41 trinity last message repeated 21 times
Dec 11 19:00:14 trinity /bsd: uid 0 on /var: file system full
Dec 11 19:00:14 trinity pflogd[26078]: Logging suspended: fwrite: No space left
on device

I run df -h to check it,

Filesystem Size Used Avail Capacity Mounted on
/dev/wd0a 251M 43.4M 195M 18% /
/dev/wd0f 126M 22.0K 120M 0% /home
/dev/wd0h 11.5G 1.2G 9.7G 11% /nsm
/dev/wd0d 126M 6.0K 120M 0% /tmp
/dev/wd0g 5.9G 2.7G 2.9G 48% /usr
/dev/wd0e 502M 501M -24.2M 105% /var

I figure out that I have really big log file - pflog which is around 400MB under /var/log, thus I remove it. I try to check /var again with following command -

shell>du -sh /var
100M /var

That's cool, I think I have reclaim the space I need, but I can't log anything to /var due to file system full even after removing pflog. It seems odd to me, I try to recheck again -

shell>df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/wd0a 251M 43.4M 195M 18% /
/dev/wd0f 126M 22.0K 120M 0% /home
/dev/wd0h 11.5G 1.2G 9.7G 11% /nsm
/dev/wd0d 126M 6.0K 120M 0% /tmp
/dev/wd0g 5.9G 2.7G 2.9G 48% /usr
/dev/wd0e 502M 501M -24.2M 105% /var

It still shows the same thing, I can't think of why df is still showing the same result and it doesn't allow me to even create a file under /var. My only solution should be a "Reboot", since giving it a try do no harms, I rebooted my machine.

I run df -h again after system is rebooted -

Filesystem Size Used Avail Capacity Mounted on
/dev/wd0a 251M 43.4M 195M 18% /
/dev/wd0f 126M 22.0K 120M 0% /home
/dev/wd0h 11.5G 1.2G 9.7G 11% /nsm
/dev/wd0d 126M 6.0K 120M 0% /tmp
/dev/wd0g 5.9G 2.7G 2.9G 48% /usr
/dev/wd0e 502M 104M 373M 22% /var

Now it looks fine, but why does it need reboot to reclaim back the disk space? Pretty odd it seems on my OpenBSD 4.0 box.

Enjoy (:()

8 comments:

Anonymous said...

do u use softdep?

Anonymous said...

did you restart pf.
if not its because the file handle is still open so the OS cant reclaim the space yet.
du and df look at different things.

Anonymous said...

im a newbie and not very sure...
however, as far as I understand, UNIX only unlink your filename but doesn't physically remove and reclaim the storage until all active link to it is unlink and all active process that have access to it has release or close the access.
you may not really need to reboot if you use lsof to find out all the active open and link and kill all those processes.

Anonymous said...

I just wanted to give you the same answer as Bob. The file handle was propable still locked to the pf.log file. You needed to restart pflogd, then everything was ok.

C.S.Lee said...

Thanks guys for the pointer, that explains why. Appreciate it.

Anonymous said...

Thats why i use cat /dev/null > file, rather than rm and touch.

Anonymous said...

Hello,
I still have 74% usage on /var. How do i free the space on /var

Capacity of var is 989MB

http://visitformoney.blogspot.com

Anonymous said...

You can use the command
# du -sh Hhdx1 /

It will show the size of folders, then simply you can delete the BIG size ones

http://visitformoney.blogspot.com