Thanks , we have backport some PR to 12.2.12,but the problem remain.Are there any other fixes? eg: > os/bluestore: apply garbage collection against excessive blob count growth https://github.com/ceph/ceph/pull/28229 AND the ceph-bluestore-tool fsck/repair https://github.com/ceph/ceph/pull/38050 > <https://github.com/ceph/ceph/pull/38050> Konstantin Shalygin <k0ste@xxxxxxxx> 于2022年6月14日周二 15:37写道： > Hi, > > Many of fixes for "zombie blobs" landed in last Nautilus release > I suggest to upgrade to last Nautilus version > > > k > > > On 14 Jun 2022, at 10:23, tao song <alansong1023@xxxxxxxxx> wrote: > > > > I have a old Cluster 12.2.12 running bluestore ,use iscsi + RBD in EC > > pools（k:m=2:1） with ec_overwrites flags. Multiple OSD crashes occurred > due > > to assert (0 == "no available blob id"). > > The problems occur periodically when the RBD volume is cyclically > > overwritten. > > > > 2022-05-24 22:08:19.950550 7fcb41894700 1 osd.171 pg_epoch: 47676 > >> pg[4.1467s2( v 44365'8455207 (44045'8453660,44365'8455207] > >> local-lis/les=44415/44416 n=21866 ec=16123/349 lis/c 47665/44415 les/c/f > >> 47666/44416/4714 47676/47676/39511) > >> [115,16,171]/[115,2147483647,2147483647]p115(0) r=-1 lpr=47676 > >> pi=[44415,47676)/2 crt=44260'8455206 lcod 0'0 remapped NOTIFY mbc={}] > >> state<Start>: transitioning to Stray > >> 2022-05-24 22:08:20.834007 7fcb41894700 1 osd.171 pg_epoch: 47677 > >> pg[4.1467s2( v 44365'8455207 (44045'8453660,44365'8455207] > >> local-lis/les=44415/44416 n=21866 ec=16123/349 lis/c 47665/44415 les/c/f > >> 47666/44416/4714 47676/47677/39511) [115,16,171]p115(0) r=2 lpr=47677 > >> pi=[44415,47677)/2 crt=44260'8455206 lcod 0'0 unknown NOTIFY mbc={}] > >> start_peering_interval up [115,16,171] -> [115,16,171], acting > >> [115,2147483647,2147483647] -> [115,16,171], acting_primary 115(0) -> > 115, > >> up_primary 115(0) -> 115, role -1 -> 2, features acting > 4611087853746454523 > >> upacting 4611087853746454523 > >> 2022-05-24 22:08:20.834073 7fcb41894700 1 osd.171 pg_epoch: 47677 > >> pg[4.1467s2( v 44365'8455207 (44045'8453660,44365'8455207] > >> local-lis/les=44415/44416 n=21866 ec=16123/349 lis/c 47665/44415 les/c/f > >> 47666/44416/4714 47676/47677/39511) [115,16,171]p115(0) r=2 lpr=47677 > >> pi=[44415,47677)/2 crt=44260'8455206 lcod 0'0 unknown NOTIFY mbc={}] > >> state<Start>: transitioning to Stray > >> 2022-05-24 22:08:22.097055 7fcb3a085700 -1 > >> /ceph-12.2.12/src/os/bluestore/BlueStore.cc: In function 'bid_t > >> BlueStore::ExtentMap::allocate_spanning_blob_id()' thread 7fcb3a085700 > time > >> 2022-05-24 22:08:22.091806 > >> /ceph-12.2.12/src/os/bluestore/BlueStore.cc: 2083: FAILED assert(0 == > "no > >> available blob id") > >> > >> ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous > >> (stable) > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > >> const*)+0x110) [0x560e41dbd520] > >> 2: (()+0x8fce4e) [0x560e41c15e4e] > >> 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, > >> std::shared_ptr<KeyValueDB::TransactionImpl>)+0x13da) [0x560e41c6fc6a] > >> 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, > >> std::shared_ptr<KeyValueDB::TransactionImpl>)+0x1ab) [0x560e41c7131b] > >> 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, > >> std::vector<ObjectStore::Transaction, > >> std::allocator<ObjectStore::Transaction> >&, > >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3fd) > >> [0x560e41c8cc4d] > >> 6: > >> (PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, > >> std::allocator<ObjectStore::Transaction> >&, > >> boost::intrusive_ptr<OpRequest>)+0x65) [0x560e419efac5] > >> 7: (ECBackend::handle_sub_write(pg_shard_t, > >> boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&, > >> Context*)+0x631) [0x560e41b18331] > >> 8: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x349) > >> [0x560e41b29ba9] > >> 9: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) > >> [0x560e41a255f0] > >> 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, > >> ThreadPool::TPHandle&)+0x59c) [0x560e4198f97c] > >> 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, > >> boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9) > >> [0x560e4180af59] > >> 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> > >> const&)+0x57) [0x560e41a9ac27] > >> 13: (OSD::ShardedOpWQ::_process(unsigned int, > >> ceph::heartbeat_handle_d*)+0xfce) [0x560e4183a20e] > >> 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x83f) > >> [0x560e41dc304f] > >> 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) > [0x560e41dc4fe0] > >> 16: (()+0x7dd5) [0x7fcb5913fdd5] > >> 17: (clone()+0x6d) [0x7fcb5822fead] > >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed > >> to interpret this. > >> > > > > Some would not restart with "no available blob id" assertion.We adjust > the > > following parameters to to ensure that the OSD can be started > > bluestore_extent_map_shard_target_size=2000 default 500， > > bluestore_extent_map_shard_target_size_slop=0.300000 default 0.200000， > > > > > >> We found several related bugs : > >> https://tracker.ceph.com/issues/48216 > >> https://tracker.ceph.com/issues/38272 > > > > The PR : > > > > os/bluestore: apply garbage collection against excessive blob count > growth > > > > https://github.com/ceph/ceph/pull/28229 > >> we have backport the PR to 12.2.12,but it didn't solve the problem. > > > > The workaround that works is to fsck / repair the stopped OSD : > >> ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<osd_id> --command > >> repair > >> > > But it's not a long term solution. > >> I have seen a PR merged in 2019 here : > >> https://github.com/ceph/ceph/pull/28229 > > > > The fsck log: > > > >> 2022-06-11 14:33:00.524108 7ff94ce7eec0 -1 > >> bluestore(/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbdd648a:::rbd_data.3.3404c86b8b4567.0000000001977896:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x5567bd482690 spanning 7 > >> blob([!~40000] csum crc32c/0x1000) use_tracker(0x4*0x10000 0x[0,0,0,0]) > >> SharedBlob(0x5567bd482150 sbid 0x0)) > >> 2022-06-11 14:33:00.620716 7ff94ce7eec0 -1 > >> bluestore(/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbdda2c7:::rbd_data.3.3404d16b8b4567.0000000000dd0d7c:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x5567b1790230 spanning 0 > >> blob([!~50000] csum crc32c/0x1000) use_tracker(0x5*0x10000 > 0x[0,0,0,0,0]) > >> SharedBlob(0x5567b17902a0 sbid 0x0)) > >> 2022-06-11 14:33:00.659399 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbddcdc9:::rbd_data.3.3404d76b8b4567.00000000005c3915:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x5568059181c0 spanning 0 > >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0]) > >> SharedBlob(0x556805919ce0 sbid 0x0)) > >> 2022-06-11 14:33:00.666271 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbddd3fa:::rbd_data.3.3404d16b8b4567.00000000008d9fcf:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x55675655ea80 spanning 1 > >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0]) > >> SharedBlob(0x55675655eaf0 sbid 0x0)) > >> 2022-06-11 14:33:00.830832 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbde5cdd:::rbd_data.3.3404d76b8b4567.0000000000aff911:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x556810246690 spanning 0 > >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0]) > >> SharedBlob(0x5567aff68f50 sbid 0x0)) > >> 2022-06-11 14:33:00.875893 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbde8f23:::rbd_data.3.3404c86b8b4567.000000000151a7dc:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x5567f2a7b650 spanning 8 > >> blob([!~50000] csum crc32c/0x1000) use_tracker(0x5*0x10000 > 0x[0,0,0,0,0]) > >> SharedBlob(0x5567f2a7b810 sbid 0x0)) > >> 2022-06-11 14:33:00.894370 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbdea5c3:::rbd_data.3.3404d16b8b4567.0000000000dcecaf:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x5567b5faa5b0 spanning 0 > >> blob([!~50000] csum crc32c/0x1000) use_tracker(0x5*0x10000 > 0x[0,0,0,0,0]) > >> SharedBlob(0x5567b5fabf10 sbid 0x0)) > >> 2022-06-11 14:33:00.930450 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbdebbe9:::rbd_data.3.3404c86b8b4567.0000000000be844c:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x556737a395e0 spanning 8 > >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0]) > >> SharedBlob(0x556737b7dc70 sbid 0x0)) > >> 2022-06-11 14:33:01.132212 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbdf864f:::rbd_data.3.3404c86b8b4567.0000000001d29b47:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x55673c858620 spanning 7 > >> blob([!~40000] csum crc32c/0x1000) use_tracker(0x4*0x10000 0x[0,0,0,0]) > >> SharedBlob(0x55673c354a80 sbid 0x0)) > >> 2022-06-11 14:33:01.157017 7ff94ce7eec0 -1 bluestore > >> (/var/lib/ceph/osd/ceph-162/) fsck error: > >> 2#4:fbdfa612:::rbd_data.3.3404d16b8b4567.0000000000a73d63:head# - 1 > zombie > >> spanning blob(s) found, the first one: Blob(0x55673f6fcf5 > > > > > > What is the root cause of zombie blobs? How can we avoid this problem? > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |