On Sat, 13 Apr 2019, zengran zhang wrote: > Hi developers: > > Always, we may see some slow requests of list omap, some even will > cause the op thread suicide。 > recently, we introduced the rocksdb delete range api, but found the > cluster be more easier complaining > list omap timed out. I checked the rocksdb of one osd, and I think I > found the reason. > > the omap iterator when listing omap use the tail of '~', when the > iterator moved to the last key of the omaps > we wanted, we will try to call extra next(), usually this will be > another object's omap header(with '-'). *IF* > there are some deleted key or tombstones, rocksdb will fall in the > loop of `FindNextUserEntryInternal` until > find a valid key, so it will travels all dead key in mid and read the > sst file heavily. If I'm understanding the problem correctly, it's that object foo has a bunch of omap items, but the object(s) that came after foo are all deleted, so that when we enumerate foo's omap, we have to skip over all the delete objects' sst files to reach the head for the next non-deleted object, right? > I think there may be 3 approachs: > 1) change the omap header from '-' to '~', let it play role of end when iterate. > 2) force to add the omap end key(use '~') on metadata pool. This seems like the way to fix it. We just need to make sure we create the end record in a backard-compatible way. Two options: 1) We create the tail when we first set FLAG_OMAP. This will leave the non-optimal behavior in place for existing objects. Or, 2) We add a new FLAG_OMAP_TAIL and create the tail if we don't have one yet. Either way, we have to make sure that on deletion, we clean up the tail keys... > 3) when iterate, found the rbegin of omap key, then we have the end > key, and could avoid extra next(). sage
