Re: Unable to boot CentOS 6 - Segmentation Erorr

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



On 2016-05-29 10:42, John Cenile wrote:
> Also, the last message in /var/log/messages before the crash was:
<snip />
> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May
> 29 07:30:10 *hostname* kernel: imklog 5.8.10, log source = /proc/kmsg
> started
> 
> Which seems very concerning.

Hi John,

TL;DR: prevention.

I can't say what happened, but I've a long-standing dread of 
your situation. Here's some ways to prepare for (or prevent)
the next time this happens. Possibly you're already doing all 
this but a recitation here might help someone else too.

- Set up remote logging. I favor rsyslog, but you can also
  use syslog-ng. Have one central logging server. This way you 
  can look for signs of trouble that preceded the crash.
 
- Keep baselines from the guest VMs. You can run rpm --verify 
  and preserve the output off-host (last step in yum update).
  Disable the nightly pre-link behavior (was this ever a good 
  idea?) to make comparing results more meaningful. 
  Post-crash, mount the victim read-only and re-run the verify
  to pin-point what part of the filesystem was clobbered.
  Knowing what was clobbered (and when) can help. Not long ago
  an errant script in production cleared the wrong
  directory but only when transaction volume crested some
  threshold, wiping out a critical monitoring script.

- Treat your hosts like cattle, not pets. Automating creation
  and maintenance of hosts gives you more and better options 
  for recovery when hosts go insane.

- Test and re-test your storage system. There are bugs lurking
  in every storage code base and every HBA's firmware. The
  physical connectors in your data path are built on a mass
  of compromises and contradictory design goals and are just 
  waiting to fail. Flush bugs out before putting gear into
  production.

- Restores, not backups, are your friends.[1] I ran into a
  bug in Gnu tar (this year) that left me with silently
  corrupted archives but only for thin-provisioned virtual 
  filesystems >16GB that compressed to <8GB. Only a full 
  restore unearthed the ugly truth.

- Consider ECC RAM. Once you have a few tens of GB's  you've 
  essentially got your own cosmic ray detector. If you 
  figure your time at $50/hour and it takes ten hours to deal 
  with with one ephemeral mysterious incident then springing 
  for $500 worth of ECC RAM is a good bet. Figure in the cost 
  of downtime and it's a no brainer.

Best regards,
-- 
Charles Polisher

[1] http://web.archive.org/web/20070920215346/http://people.qualcomm.com/ggr/GoB.txt


_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos



[Index of Archives]     [CentOS]     [CentOS Announce]     [CentOS Development]     [CentOS ARM Devel]     [CentOS Docs]     [CentOS Virtualization]     [Carrier Grade Linux]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [Xorg]     [Linux USB]
  Powered by Linux