Hello Van Diep, I read this after you got out of trouble. According to your ceph osd tree, it looks like your problems started when the ceph orchestrator created osd.29 on node 'cephgw03' because it looks very unlikely that you created a 100MB OSD on a node that's named after "GW". You may have added the 'osds' label to the 'cephgw03' node and/or played with the service_type:osd and/or added the cephgw03 node to the crushmap, which triggered the creation of osd.29 by the orchestrator. cephgw03 node being part of the 'default' root bucket, other OSDs legitimately started to send objects to osd.29, way to small to accommodate them, PGs then becoming 'backfill_toofull'. To get out of this situation, you could have: $ ceph osd crush add-bucket closet root $ ceph osd crush move cephgw03 root=closet This would have moved 'cephgw03' node out of the 'default' root and probably fixed your problem instantly. Regards, Frédéric. -----Message original----- De: Anthony <anthony.datri@xxxxxxxxx> à: nguyenvandiep <nguyenvandiep@xxxxxxxxxxxxxx> Cc: ceph-users <ceph-users@xxxxxxx> Envoyé: samedi 24 février 2024 16:24 CET Sujet : Re: [Urgent] Ceph system Down, Ceph FS volume in recovering There ya go. You have 4 hosts, one of which appears to be down and have a single OSD that is so small as to not be useful. Whatever cephgw03 is, it looks like a mistake. OSDs much smaller than, say, 1TB often aren’t very useful. Your pools appear to be replicated, size=3. So each of your cephosd* hosts stores one replica of each RADOS object. You added the 10TB spinners to only two of your hosts, which means that they’re only being used as though they were 4TB OSDs. That’s part of what’s going on. You want to add a 10TB spinner to cephosd02. That will help your situation significantly. After that, consider adding a cephosd04 host. Having at least one more failure domain than replicas lets you better use uneven host capacities. > On Feb 24, 2024, at 10:06 AM, nguyenvandiep@xxxxxxxxxxxxxx wrote: > > Hi Mr Anthony, > > pls check the output > > https://anotepad.com/notes/s7nykdmc > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |