Hi Igor, I want to give you a short update, since I tried now for quite some time to reproduce the problem as you suggested. I've tried to simulate every imaginable load that the cluster might have done before the three OSD crashed. I rebooted the servers many times while the Custer was under load. If more than a single node was rebooted at the same time, the client hangs until enough servers are up again. Which is perfectly fine! I really tried hard to crash it, but I failed. Wich is excellent in general, but unfortunately not helpful for finding the root cause of the problem with the corrupted Rocks DBs. However, I found another problem, that occurred now at least two times. I do not think that it is related to the problem with the corrupted Rocks DBs, but I want to let you know. If I enable some Snapshot Schedules for a ceph FS directory, they stopes creating new snapshots unexpectedly. It my have something to do with a reboots of the ceph FS MDS, but I’m not sure. Best regards, Sebastian > On 24.12.2021, at 13:05, Igor Fedotov <igor.fedotov@xxxxxxxx> wrote: > > Hey Sebastian, > > On 12/22/2021 1:53 AM, Sebastian Mazza wrote: >> >>> 9) Would you be able to run some long lasting (and potentially data corrupting) experiments at this cluster in an attempt to pin point the issue. I'm thinking about periodic OSD shutdown under the load to catch the corrupting event. With a raised debug level for that specific OSD. The major problem with this bug debugging is that we can see its consequences - but we have no clue about what was happening when actual corruption happened. Hence we need to reproduce that somehow. So please let me know if we can use your cluster/help for that... >> I want to help. Destroying the data on the cluster is not a problem. The question is, if I can find enough time, but I will do what I can. So, you are welcome to give me detailed instructions what I should test. >> One thing that could be important: I don’t think there was a significant load on the OSDs when this problem happened. > > So we want to repoduce the same issue with a more verbose logging. Hence my suggestion for the first attempt would be to try to restart the cluster in the same manner you did it before. With some preceeding steps: > > 1) Bring cluster back to the healthy state be redeplying broken OSDs. > > 2) inject verbose bluefs/bdev logging shortly before the restart (do not leave the cluster with these debug levels for a long time as it might consume tons of disk space for logging) > > ceph tell osd.* injectargs "--debug-bluefs 20 --debug-bdev 20" > > 3) bring some load to the cluster to force disk writing. > > 4) restart the cluster and check OSD status on completion. If someone is/are broken - save the relevant logs > > > May be try the above multiple times if every OSD is fine on reboot. > > > Thanks in advance, > > -- > Igor Fedotov > Ceph Lead Developer > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |