Re: Network Flapping Causing Slow Ops and Freezing VMs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello Mahnoosh,

Just to double check, can you confirm that you are NOT using a
physically separate cluster network and private network? A
configuration with such physically separate networks is inherently
vulnerable and therefore cannot be recommended. VLANs on the same
physical interface are probably acceptable, but I have never seen a
cluster configured like this.

https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd/#flapping-osds

On Sat, Jan 6, 2024 at 9:28 PM mahnoosh shahidi <mahnooosh.shd@xxxxxxxxx> wrote:
>
> Hi all,
>
> I hope this message finds you well. We recently encountered an issue on one
> of our OSD servers, leading to network flapping and subsequently causing
> significant performance degradation across our entire cluster. Although the
> OSDs were correctly marked as down in the monitor, slow ops persisted until
> we resolved the network issue. This incident resulted in a major
> disruption, especially affecting VMs with mapped RBD images, leading to
> their freezing.
>
> In light of this, I have two key questions for the community:
>
> 1. Why did slow ops persist even after marking the affected server as down
> in the monitor?
>
> 2.Are there any recommended configurations for OSD suicide or OSD down
> reports that could help us better handle similar network-related issues in
> the future?
>
> Best Regards,
> Mahnoosh
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx



-- 
Alexander E. Patrakov
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux