filesystems - VXFS (veritas) File system filling up - except it's not

I have a Solaris 10 server which is reporting that a file system is filling up:

# df -h /opt/CSCObac/var
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL

# df -F vxfs /opt/CSCObac/var/
/opt/CSCObac/var   (/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL): 4469796 blocks   558723 files


# df -h | fgrep '/opt/'
/dev/vx/dsk/RDU-APP-DG/RDU-APP-VOL    68G   547M    63G     1%    /opt/CSCObac
/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL   134G   132G   2.0G    99%    /opt/CSCObac/var
/dev/vx/dsk/RDU-LOG-DG/RDU-LOG-VOL   134G   5.2G   121G     5%    /opt/CSCObac/var/rdu

Except there are no files on the filesystem which occupy 100G+, df reports it's not out of free blocks, and there's no lack of free inodes either:

# du -sh /opt/CSCObac/var
 5.1G   /opt/CSCObac/var

                   134G   132G   2.0G    99%    /opt/CSCObac/var

# df -g /opt/CSCObac/var/
/opt/CSCObac/var   (/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL):     8192 block size          1024 frag size  
280938496 total blocks    4469536 free blocks  4190192 available             558720 total files
558691 free files     74450896 filesys id  
vxfs fstype       0x00000004 flag             255 filename length

Also, no open file descriptors:

# lsof /opt/CSCObac/var/
#

This is a vxfs filesystem, see below for volume and size configurations:

# fstyp -v /dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL
vxfs
magic a501fcf5  version 7  ctime November 15, 2013  2:15:55 PM CET

logstart 0  logend 0
bsize  1024 size  140469248 dsize  0  ninode 140469248  nau 0
defiextsize 0  ilbsize 0  immedlen 96  ndaddr 10
aufirst 0  emap 0  imap 0  iextop 0  istart 0
bstart 0  femap 0  fimap 0  fiextop 0  fistart 0  fbstart 0
nindir 2048  aulen 32768  auimlen 0  auemlen 8
auilen 0  aupad 0  aublocks 32768  maxtier 15
inopb 4  inopau 0  ndiripau 0  iaddrlen 8 bshift 10
inoshift 2  bmask fffffc00  boffmask 3ff  checksum f790321b
oltext1 32  oltext2 1282  oltsize 1  checksum2 0

free 2234899  ifree 0
efree  5 9 7 2 3 5 4 4 2 0 3 1 2 3 2 0 1 2 1 1 1 0 0 0 0 0 0 0 0 0 0 0 

# vxlist disk | grep 'VAR'
disk disk_4     RDU-VAR-DG01   RDU-VAR-DG     133.96g   608.00k imported
disk disk_8     RDU-VAR-DG02   RDU-VAR-DG     133.96g   608.00k imported

# vxlist vol RDU-VAR-VOL
Volume:  RDU-VAR-VOL
GUID:               {0de33acc-1dd2-11b2-a8f9-00212800c1e0}

Disk Group:         name=RDU-VAR-DG id=1384520670.51.rdu2a
Host:               rdu2a
Size:               133.9g
Type:               fsgen
Status:             healthy
Layout:             concat-mirror
NMirrors:           2
Read Policy:        SELECT
Flags:              open writeback
Device Path:        /dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL

Permissions:        user=root group=root mode=0600

filesystem
TY   FSTYPE     %USED VSET   MOUNT-POINT
fs   vxfs         98% -      /opt/CSCObac/var

stats
TY   OPERATION       COUNT      BLOCKS   TIME(MSEC)
stat read          1160490     1731963      17538.5
stat write         3928639   294438233       7579.4


plexes
TY   NAME             TYPE     STATUS
plex RDU-VAR-VOL-03   simple   attached

# vxlist fs | fgrep 'VAR'
fs   var       vxfs       133.96g   2046.10m     99% /dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL   /opt/CSCObac/var

# vxassist -g RDU-VAR-DG -p maxgrow RDU-VAR-VOL
VxVM vxassist ERROR V-5-1-1178 Volume RDU-VAR-VOL cannot be extend within the given constraints


# fsadm -D /opt/CSCObac/var/ 

  Directory Fragmentation Report
             Dirs        Total      Immed    Immeds   Dirs to   Blocks to
             Searched    Blocks     Dirs     to Add   Reduce    Reduce
  total            11         3         8         0         0           0

What could be filling up my filesystem beyond inodes, blocks and file descriptors?

Answer

After some lenghty troubleshooting with our support vendor, we found the solution.
There was a process running that was started before the mounting of the vxfs filesystems.
This process was writing data to /opt/CSCObac/var/rdu which is a nested directory of the /opt/CSCObac/var/ mountpoint.
After killing this process, the disk usage was back to the expected 1%
The OS could not report back on contents of this directory as the rdu mountpoint obscures it.
The long running process started writing to a file in this directory mountpoint before the filesystem was mounted, which explains why it was increasing in size without seeing an increase in the visible files within the directory.