I just deleted a 33 GiB error log that had filled up my NFS shared disk. This didn't immediately free up any space on the device. This is because the file was being held open by another process and so the space can't be freed until those processes terminate and close their connections to the file.
To check for this condition you can use lsof
and then grep
for deleted files. I found what was keeping my file open and killed that process (it was a "grep -c" I'd forgotten I'd set off ages ago).
The process keeping the file open is listed in the first column.
For example, in the output below the grep process is holding my stackTraces.out file open:
"grep 1234 paul 3r REG 202,166 35110973440 3285868 /nfs_share/proj/stackTraces.out (deleted)"