Wednesday, March 4, 2015

filesystems - File system full - except it's not



I've a Solaris 8 box which is reporting that a file system is full:



db% tail -2 /var/adm/messages

Nov 22 08:32:27 db ufs: [ID 845546 kern.notice] NOTICE: alloc: /u03: file system full
Nov 22 08:34:51 db last message repeated 12 times


But df says it's not out of free blocks, and it's not out of free inodes either:



db% df -k /u03
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d6 282330903 254957403 24550191 92% /u03
db% df -o i /u03

Filesystem iused ifree %iused Mounted on
/dev/md/dsk/d6 29663278 4230866 88% /u03


So I thought perhaps some process is holding open file descriptor(s) to 20+GB of deleted files. But lsof, sorting by size, reports nothing over 2GB, and that's a legitimate file:



db% lsof /u03 | sort -n +6
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
[...]
oracle 1257 oracle 278u VREG 85,6 2097160192 9685782 /u03/oradata/(redacted)/data/foo_tab_14.dbf

db%


I'd appreciate any pointers to any other resource save free blocks and inodes whose exhaustion can fill up a file system, or some other way to "hide" used blocks/inodes, or any other idea anyone has. Rebooting this box or shutting down oracle is not a valid investigative option.



Edit: Khaled, I couldn't at the time. I can't post the output, because one of the DBAs has freed up about 4GB which means the machine can continue functioning, and if I "fill it up" again as a test, things will break. But this is the second time in 24 hours it's got to about 92% full and "filled up" (as in, new files can't be created, and /var/adm/messages reports "file system full"), and yes, when it happens it definitely breaks the creation or extension of files on that FS.


Answer



Check the nbfree value with



fstyp -v | head -18



If that says some low value, this blog post I found might help you. I quote the beginning of the post:




At work we have a Solaris 8 with a UFS which told the appli­ca­tion
that it can not cre­ate new files. The df com­mand showed plenty if
free inodes, and there was also enough space free in the FS. The
rea­son that the appli­ca­tion got the error was that while there
where still plenty of frag­ments free, no free block was avail­able

any­more. You can not cre­ate a new file only with frag­ments, you
need to have at least one free block for each new file.



To see the num­ber of free blocks of a UFS you can call “fstyp -v |
head -18″ and look at the value behind “nbfree”.



No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...