Friday, August 31, 2018

centos5 - Near 100% disk usage, df and du show very different results, lsof not the answer

The issue is that my CentOS 5.8 machine is telling me that I am nearly out of disk space when I am pretty confident this is not the case. I've done a fair amount of researching on this issue and have been unable to find a solution.




'df -h' shows 210G used, 8.6G avail



'ncdu' shows 28.6G used (same for apparent size)



As you can see, this is nearly a 10x difference in the reported 'used' space. Knowing what is stored on this drive, I think 28.6G is closer to reality.



Looking at the output of 'lsof' there are very few lines with (deleted) at the end. Moreover, the largest size of any of these lines is 6190. Finally, I've rebooted the machine a number of times which, if I understand the other threads correctly, would resolve the issue of phantom files anyhow.



Here is a summary of the output from ncdu:





22.7GiB [##########] /opt
2.8GiB [# ] /usr
1.5GiB [ ] /var
812.4MiB [ ] /root
310.6MiB [ ] /home
194.3MiB [ ] /lib
156.4MiB [ ] /etc
36.5MiB [ ] /sbin

7.3MiB [ ] /bin
128.0KiB [ ] /tmp
20.0KiB [ ] /mnt
e 16.0KiB [ ] /lost+found
e 8.0KiB [ ] /srv
e 8.0KiB [ ] /selinux
8.0KiB [ ] /media
e 4.0KiB [ ] /backup
> 0.0 B [ ] /sys
> 0.0 B [ ] /proc

> 0.0 B [ ] /net
> 0.0 B [ ] /misc
> 0.0 B [ ] /dev
> 0.0 B [ ] /boot
0.0 B [ ] .autorelabel
0.0 B [ ] .autofsck


Output of 'df -Th':





Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
ext3 221G 210G 8.6G 97% /
/dev/sda1 ext3 99M 25M 74M 25% /boot
tmpfs tmpfs 1.7G 0 1.7G 0% /dev/shm


This post mentions that outside of phantom files there are two other possible explanations:





  1. corrupt filesystem

  2. compromised machine



I'm looking for help on how to test the validity of these explanations. Obviously, explanation #2 is particularly concerning.



Thanks for your help!

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...