>>>>> "Olaf" == Olaf Seibert <o.seibert@xxxxxxxxxxxx> writes: Olaf> Replying to myself: Olaf> On 30.05.22 10:16, Olaf Seibert wrote: >> First, John, thanks for your reply. Olaf> I contacted the customer and it turned out their VM's disk (this Olaf> LV) was broken anyway. So there is no need any more to try to Olaf> repair it... So I'm not really surprised, because when that disk dies, it probably took out their data, or at least a chunk of it, so even though it looks like it might have kept running, it probably also got corrupted in a big way too. So I think you guys need to re-architect your storage design. If you have paying customers on there, you should really be using MD with RAID10, and a hot spare disk on there as well, so when a disk dies, it can be automatically replaced, even if it fails at 2am in the morning. It's not cheap, but neither is a customer losing data. The other critical thing to do here is to make sure you're using disks with proper SCTERC timeouts, so that when they have problems, the disks just fail quickly, without blocking the system and causing outages. Look back in the linux-raid mailing list archives for discussions on this. And of course I'd also try to setup a remote backup server with even bigger disks, so that you can replicate customer data onto other storage just in case. Olaf> Thanks for your thoughts anyway. Glad I could try to help, been flat out busy with $WORK and just now following up here. Sorry! Olaf> -- Olaf> SysEleven GmbH Olaf> Boxhagener Straße 80 Olaf> 10245 Berlin Olaf> T +49 30 233 2012 0 Olaf> F +49 30 616 7555 0 Olaf> http://www.syseleven.de Olaf> http://www.facebook.com/SysEleven Olaf> https://www.instagram.com/syseleven/ Olaf> Aktueller System-Status immer unter: Olaf> http://www.twitter.com/syseleven Olaf> Firmensitz: Berlin Olaf> Registergericht: AG Berlin Charlottenburg, HRB 108571 B Olaf> Geschäftsführer: Marc Korthaus, Jens Ihlenfeld, Andreas Hermann Olaf> _______________________________________________ Olaf> linux-lvm mailing list Olaf> linux-lvm@xxxxxxxxxx Olaf> https://listman.redhat.com/mailman/listinfo/linux-lvm Olaf> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ _______________________________________________ linux-lvm mailing list linux-lvm@xxxxxxxxxx https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
