Hi All,
I have an older CentOS 7.4 system that is used for computationally heavy work.
It has a 32G root filesystem, of which 33% is consumed.
Lately, one particular set of jobs (run through the SGE batch scheduler) seems
to cause a peculiar condition to occur in which the root filesystem space is
exhausted but I can't find any files that I can use to identify which process
is causing the problem. I'm guessing that the problem is being triggered by a
certain set of jobs since it only occurs when those jobs are running.
I would like to identify what the process is that is causing this condition to
occur and my standard approach is to identify which files are using the
otherwise empty space and then identify which process is using the files. It's
not working, I haven't been able to identify the files. They aren't there. All
attempts to measure disk usage of / by files shows that the disk usage is only
a percentage of available space and that there should be space available.
I know that space can be consumed by deleted files for which file handles
are still open, not visible by ls but detectable but by using techniques like:
lsof | grep deleted
and for sizing
find /proc/*/fd -ls 2>/dev/null | grep '(deleted)' | \
sed 's!.*\(/proc[^ ]*\).*!\1!' | xargs wc -c | sort -nr
Applying this technique during one of these episodes shows nothing of interest.
I'm definitely missing something. Are there any other techniques for
identifying other "invisible" files that I can look for?
Thanks in advance for any advice.
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
