Re: Cluster experiencing complete operational failure, various cephx authentication errors

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2020-08-24 20:35, Mathijs Smit wrote:
> Hi everyone,
> 
> I have a serious problem which currently exists of my entire Ceph no longer being able to provide service. As if yesterday I added 10 OSD's total 2 per node, the rebalance started and took some IO but seemed to be doing its work. This morning the cluster was still processing the rebalance and taking so much IO that nearly all OSD's where marked as "slow ops" and from there everything went wrong. As attempt to clear as much IO for de rebalance I stoped all the clients and waited for the rebalance to finish. After it finished the cluster remained extremely slow and unusable. Whilst trying to debug I restarted several services and nodes trying to find the problem. Now the cluster has entered a state where multiple OSD's remain slow, various OSD's show a "BADAUTHORIZER" message and the mgr on all nodes also has issues "verify_authorizer".
> 
> I verified all the clocks on all servers and they are sinked to the same NTP service and seem good.
> 
> Please please please advise as straight 13 hours of debugging got me nowhere.

If you can pause all client IO, I don't think it can harm to upgrade
everything to 14.2.11, just to be sure you don't hit a bug that is
already fixed. But it depends on the current health of your cluster I
guess.

Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux