On 2016-05-29 10:42, John Cenile wrote:
> Also, the last message in /var/log/messages before the crash was:
<snip />
> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May
> 29 07:30:10 *hostname* kernel: imklog 5.8.10, log source = /proc/kmsg
> started
>
> Which seems very concerning.
Hi John,
TL;DR: prevention.
I can't say what happened, but I've a long-standing dread of
your situation. Here's some ways to prepare for (or prevent)
the next time this happens. Possibly you're already doing all
this but a recitation here might help someone else too.
- Set up remote logging. I favor rsyslog, but you can also
use syslog-ng. Have one central logging server. This way you
can look for signs of trouble that preceded the crash.
- Keep baselines from the guest VMs. You can run rpm --verify
and preserve the output off-host (last step in yum update).
Disable the nightly pre-link behavior (was this ever a good
idea?) to make comparing results more meaningful.
Post-crash, mount the victim read-only and re-run the verify
to pin-point what part of the filesystem was clobbered.
Knowing what was clobbered (and when) can help. Not long ago
an errant script in production cleared the wrong
directory but only when transaction volume crested some
threshold, wiping out a critical monitoring script.
- Treat your hosts like cattle, not pets. Automating creation
and maintenance of hosts gives you more and better options
for recovery when hosts go insane.
- Test and re-test your storage system. There are bugs lurking
in every storage code base and every HBA's firmware. The
physical connectors in your data path are built on a mass
of compromises and contradictory design goals and are just
waiting to fail. Flush bugs out before putting gear into
production.
- Restores, not backups, are your friends.[1] I ran into a
bug in Gnu tar (this year) that left me with silently
corrupted archives but only for thin-provisioned virtual
filesystems >16GB that compressed to <8GB. Only a full
restore unearthed the ugly truth.
- Consider ECC RAM. Once you have a few tens of GB's you've
essentially got your own cosmic ray detector. If you
figure your time at $50/hour and it takes ten hours to deal
with with one ephemeral mysterious incident then springing
for $500 worth of ECC RAM is a good bet. Figure in the cost
of downtime and it's a no brainer.
Best regards,
--
Charles Polisher
[1] http://web.archive.org/web/20070920215346/http://people.qualcomm.com/ggr/GoB.txt
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
