Hey Mike, Maybe it's this? https://tracker.ceph.com/issues/48632 Cheers, Dan On Wed, Feb 17, 2021, 6:53 PM Mike Cave <mcave@xxxxxxx> wrote: > I am bumping this email to hopefully get some more eyes on it. > > We are continuing to have this problem. Unfortunately the cluster is very > lightly used currently until we go full production so we do not have the > level of traffic that would generate a lot of statistics. > > We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to > correlate with when the errors started popping up. > > Our current plan is to roll back the version to 14.2.10 again and rerun > the test that causes the issue. > > I noted there was another email thread regarding latencies for a user who > also updated to 14.2.16 recently and I'm not sure if this could be related > or not to my issue. > > Any suggestions you may have are very welcomed. > > Cheers, > -- > Mike Cave > > > > On 2021-02-11, 8:37 AM, "Mike Cave" <mcave@xxxxxxx> wrote: > > > So, as the subject states I have an issue with buckets returning a 404 > error when they are listed immediately after being created; as well the > bucket fails to be deleted if you try to delete it immediately after > creation. > > The behaviour is intermittent. > > If I leave the bucket in place for a few minutes, the bucket behaves > normally. I’m thinking this is a metadata issue or something along those > lines but I’m out of my depth now. > > To the best of our knowledge the cluster has not changed in any way > since the same tests were run in December with no errors. > > We are running Ceph 14.2.16 on all parts of the cluster. > > I am using the python-swift client for the connection on a CentOS7 > machine. > > Can replicate the results from the mons or an external client as well. > > I’m willing to share my test script as well if you would like to see > how I’m generating the error. > > Here is a piece of the logs in case I missed something in the > interpretation (log level at 20): > > 14:23:17.069 7faba00df700 1 ====== starting new request > req=0x55fb7a138700 ===== > 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id > = tx000000000000000000094-0060245cd5-2b8949-default > 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 > 14:23:17.069 7faba00df700 10 host=<NameRemoved> > 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 > in_hosted_domain_s3website=0 > 14:23:17.069 7faba00df700 -1 res_query() failed > 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= > in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= > s->info.request_uri=/swift/v1/404test > 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= > 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT > 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 > 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket > scheduling with dmclock client=3 cost=1 > 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > verifying requester > 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket > rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine > 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket > rgw::auth::swift::TempURLEngine denied with reason=-13 > 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket > rgw::auth::swift::DefaultStrategy: trying > rgw::auth::swift::SignedTokenEngine > 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket > swift_user=xmcc:swift > 14:23:17.069 7faba00df700 20 build_token > token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 > 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket > rgw::auth::swift::SignedTokenEngine granted access > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > normalizing buckets and tenants > 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init > permissions > 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 > obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 > 14:23:17.069 7faba00df700 10 cache get: > name=default.rgw.meta+root+404test : hit (negative entry) > 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 > obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 > 14:23:17.069 7faba00df700 10 cache get: > name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) > 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set > empty > 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag > 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 > obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 > 14:23:17.069 7faba00df700 10 cache get: > name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) > 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set > empty > 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > recalculating target > 14:23:17.069 7faba00df700 10 Starting retarget > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > reading permissions > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > verifying op mask > 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket > required_mask= 4 user.op_mask=7 > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > verifying op permissions > 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- > Getting permissions begin with perm_mask=50 > 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket > Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> > rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, > acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 > 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc > 14:23:17.069 7faba00df700 5 Found permission: 15 > 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 > 14:23:17.069 7faba00df700 5 Permissions for group not found > 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 > 14:23:17.069 7faba00df700 5 Permissions for group not found > 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- > Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() > -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, > acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 > 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket > identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier > -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, > perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, > user_perm_mask=2, acl perm=2 > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > verifying op params > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > pre-executing > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > executing > 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: > bucket 404test not found > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket > completing > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op > status=-2002 > 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http > status=404 > 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op > status=-2002 http_status=404 latency=0s ====== > > -- > Mike Cave > > I acknowledge and respect the Lekwungen-speaking Peoples on whose > traditional territories the university stands and the Songhees, Esquimalt > and WSANEC peoples whose historical relationships with the land continue to > this day. > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |