The problem is that auditd "suspends" activities when /var falls below
20% free.
From /etc/audit/audit.conf
output {
mode = bin;
num-files = 4;
file-size = 20M;
file-name = "/var/log/audit.d/bin";
notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u
-C -T 20
%";
# AUDBIN THRESHOLDS:
# The above notify will cause auditd to enter 'suspend' mode when
# free space on the /var/ filesystem falls below 20%.
You can change the activity that auditd takes when the disk with /var
reaches 80%, or you can turn auditd off:
service audit stop
chkconfig audit off
Alfred Hovdestad, RHCE
University of Saskatchewan
Stephen Jascourt wrote:
I discovered that PAM authentication hangs in at least 4 different
kernels of Enterprise Linux, including one installed last week, when the
partition containing /var is more than 80% full (e.g. less than 20%
available).
The following appears in the /var/log/messages file:
audbin[2673]: saving binary audit log /var/log/audit.d/bin.2
audbin[2673]: threshold 20.00 exceeded for filesystem /var/log/audit.d/.
- free blocks down to 16.24%
auditd[2130]: Notify command /usr/sbin/audbin -S
/var/log/audit.d/save.%u -C -T 20% exited with status 1
auditd[2130]: output error
auditd[2130]: output error
auditd[2130]: output error; suspending execution
[numbers in brackets are probably process id numbers or something, they
change each time; all other parts of the message repeat on each boot
attempt]
I don't know what auditd does or what it's connection to authentication
is. What I do know is that when this happens at a time when you have
open xterm windows, you can still execute commands and use the system,
but you cannot ssh or sftp or otherwise enter from outside - you can
connect, but upon entering your password, the connection will hang.
Similarly, trying to change user from an open window or go to root via
"su" also hangs upon entering password. Other open windows and processes
will be unaffected. I don't have any cron scripts, but if they involve
authentication, they probably would also hang. However, once you end
your session and have no open windows into the machine, you are locked
out entirely. A regular boot proceeds normally aside from the above
messages in the message log, but it does not startx, the screen with the
color login box (control-alt-F7) does not appear, and logging in from
the text login screen (such as the screen on control-alt-F1) hangs upon
entering password, as do attempts to login from outside. Same result for
logging in as root.
The problem is resolved by reducing disk usage under 80% for the
partition containing /var. You need to find a bootable disk, mount your
hard drives, set permissions to write, and delete some files. After
getting under 80%, everything is fine, shut down and do a normal boot
and you're back in business.
I am not a system manager or IT person, nor am I a linux expert, just a
linux user who needed to access his work PC to do work when our local
linux experts are on their holiday vacation. I could not find any web
messages or other documentation on this problem, so apparently it has
not been documented before or it is documented in a place that does not
show up in web searches.
The problem had been happening intermittently for a while and would
clear up upon rebooting. I guess I may have been near 80% and booting
cleaned up all the temporary files bringing me under 80%. I didn't
troubleshoot the problem, didn't have any clues, so I didn't know about
the magic 80% threshold. Then I downloaded some large data files and got
locked out until someone suggested I use a bootable CD, mount my hard
drives, and someone else suggested looking in /var/log/messages, where I
found the above lines and tried deleting files. Once I got under 80%, it
was fine.
There may be some parameter in some configuration file that sets the 20%
threshold for available disk space required, but I don't know where that
would be. If so, I could lower it to say 1% or something small. Another
solution is when configuring a new system, you could partition /var by
itself with only the amount of space it will need and run a daily script
to monitor space in /var and send a warning message when space available
starts getting close to 20%. But it would be much better if folks who
work on linux source code could just fix the problem.
Stephen Jascourt
UCAR/COMET meteorologist at National Weather Service Headquarters
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list
