[CS-FSLUG] An odd Linux occurrence

[Tim Young]

Ok.  I am a little puzzled.

My home Linux server (CentOS release 6.3 (Final)) recently ran out of 
disk space.  I should have a lot of room on my 2TB RAID drive, but it 
was 100% full.  So I wandered around my system, cleaning off old 
backups and stuff.  I cleaned out 10GB, hoping that would get me 
through this next week.  I went off to do some other things and came 
back because I forgot if I had 11GB or 10GB free.  Nope.  100% full 
(well, 200MB free, but when I looked again, it was 160MB and counting 
the wrong way).  I did a number of diagnostics, freed up another 20GB 
of not-so-available backups, and then watched as that 20GB was slowly 
and methodically eaten.

The odd thing was that I could do a
du /*
and those numbers were not changing.
I dumped those numbers in files and then used diff to check to tell 
me what what dirs were changing in size to eat up GBs of space.

Turns out the only dir that was changing in size was the dir I was 
storing my temp files in.  And the change in size was measured in K.  
I would expect that, since I was making these tiny text files with 
space usage info in them.

I did an lsof and did not see anything interesting.

So, finally, I thought I would reboot.

After it came back up, it is reporting 1.2TB free, which is about 
what I would expect with the amount of data I have.

Anyone see anything like this before?

I do have some extra odd services running, and it could be one of 
those.  Also, I am running a yum update since there are a number of 
patches to install.  Just wondering if I am special, crazy, or just 
lucky.

     - Tim Young