I've a Solaris 8 box which is reporting that a file system is full:
db% tail -2 /var/adm/messages
Nov 22 08:32:27 db ufs: [ID 845546 kern.notice] NOTICE: alloc: /u03: file system full
Nov 22 08:34:51 db last message repeated 12 times
But df says it's not out of free blocks, and it's not out of free inodes either:
db% df -k /u03
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d6 282330903 254957403 24550191 92% /u03
db% df -o i /u03
Filesystem iused ifree %iused Mounted on
/dev/md/dsk/d6 29663278 4230866 88% /u03
So I thought perhaps some process is holding open file descriptor(s) to 20+GB of deleted files. But lsof, sorting by size, reports nothing over 2GB, and that's a legitimate file:
db% lsof /u03 | sort -n +6
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
[...]
oracle 1257 oracle 278u VREG 85,6 2097160192 9685782 /u03/oradata/(redacted)/data/foo_tab_14.dbf
db%
I'd appreciate any pointers to any other resource save free blocks and inodes whose exhaustion can fill up a file system, or some other way to "hide" used blocks/inodes, or any other idea anyone has. Rebooting this box or shutting down oracle is not a valid investigative option.
Edit: Khaled, I couldn't at the time. I can't post the output, because one of the DBAs has freed up about 4GB which means the machine can continue functioning, and if I "fill it up" again as a test, things will break. But this is the second time in 24 hours it's got to about 92% full and "filled up" (as in, new files can't be created, and /var/adm/messages
reports "file system full"), and yes, when it happens it definitely breaks the creation or extension of files on that FS.
Answer
Check the nbfree
value with
fstyp -v | head -18
If that says some low value, this blog post I found might help you. I quote the beginning of the post:
At work we have a Solaris 8 with a UFS which told the application
that it can not create new files. The df command showed plenty if
free inodes, and there was also enough space free in the FS. The
reason that the application got the error was that while there
where still plenty of fragments free, no free block was available
anymore. You can not create a new file only with fragments, you
need to have at least one free block for each new file.
To see the number of free blocks of a UFS you can call “fstyp -v |
head -18″ and look at the value behind “nbfree”.
No comments:
Post a Comment