I've update the dc3 site from octopus to pacific and the problem is still there. I find it very weird that in only happens from one single zonegroup to the master and not from the other two. Am Mi., 21. Juni 2023 um 01:59 Uhr schrieb Boris Behrens <bb@xxxxxxxxx>: > I recreated the site and the problem still persists. > > I've upped the logging and saw this for a lot of buckets (i've stopped the > debug log after some seconds). > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: > rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:s3bucket-fra2 > state=0x7fcba05ac0a0 s->prefetch_data=0 > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: > name=dc3.rgw.meta+root+s3bucket-fra2 : miss > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: > name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x6 > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding > dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: > name=dc3.rgw.meta+root+s3bucket-fra2 : type miss (requested=0x1, cached=0x6) > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: > name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x1 > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving > dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: > rctx=0x7fcaab7f9320 > obj=dc3.rgw.meta:root:.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > state=0x7fcba43ce0a0 s->prefetch_data=0 > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: > name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > : miss > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: > name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > info.flags=0x16 > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding > dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > to cache LRU end > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: > name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > : type miss (requested=0x13, cached=0x16) > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: > name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > info.flags=0x13 > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving > dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > to cache LRU end > 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 chain_cache_entry: > cache_locator=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 > > Am Di., 20. Juni 2023 um 19:29 Uhr schrieb Boris <bb@xxxxxxxxx>: > >> Hi Casey, >> already did restart all RGW instances. Only helped for 2 minutes. We now >> stopped the new site. >> >> I will remove and recreate it later. >> As twi other sites don't have the problem I currently think I made a >> mistake in the process. >> >> Mit freundlichen Grüßen >> - Boris Behrens >> >> > Am 20.06.2023 um 18:30 schrieb Casey Bodley <cbodley@xxxxxxxxxx>: >> > >> > hi Boris, >> > >> > we've been investigating reports of excessive polling from metadata >> > sync. i just opened https://tracker.ceph.com/issues/61743 to track >> > this. restarting the secondary zone radosgws should help as a >> > temporary workaround >> > >> >> On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens <bb@xxxxxxxxx> wrote: >> >> >> >> Hi, >> >> yesterday I added a new zonegroup and it looks like it seems to cycle >> over >> >> the same requests over and over again. >> >> >> >> In the log of the main zone I see these requests: >> >> 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: >> >> fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET >> >> >> /admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26 >> >> HTTP/1.1" 200 44 - - - >> >> >> >> Only thing that changes is the &id. >> >> >> >> We have two other zonegroups that are configured identical (ceph.conf >> and >> >> period) and these don;t seem to spam the main rgw. >> >> >> >> root@host:~# radosgw-admin sync status >> >> realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) >> >> zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) >> >> zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) >> >> metadata sync syncing >> >> full sync: 0/64 shards >> >> incremental sync: 64/64 shards >> >> metadata is caught up with master >> >> >> >> root@host:~# radosgw-admin period get >> >> { >> >> "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", >> >> "epoch": 92, >> >> "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", >> >> "sync_status": [ >> >> "", >> >> ... >> >> "" >> >> ], >> >> "period_map": { >> >> "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", >> >> "zonegroups": [ >> >> { >> >> "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", >> >> "name": "dc3", >> >> "api_name": "dc3", >> >> "is_master": "false", >> >> "endpoints": [ >> >> ], >> >> "hostnames": [ >> >> ], >> >> "hostnames_s3website": [ >> >> ], >> >> "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", >> >> "zones": [ >> >> { >> >> "id": "96f5eca9-425b-4194-a152-86e310e91ddb", >> >> "name": "dc3", >> >> "endpoints": [ >> >> ], >> >> "log_meta": "false", >> >> "log_data": "false", >> >> "bucket_index_max_shards": 11, >> >> "read_only": "false", >> >> "tier_type": "", >> >> "sync_from_all": "true", >> >> "sync_from": [], >> >> "redirect_zone": "" >> >> } >> >> ], >> >> "placement_targets": [ >> >> { >> >> "name": "default-placement", >> >> "tags": [], >> >> "storage_classes": [ >> >> "STANDARD" >> >> ] >> >> } >> >> ], >> >> "default_placement": "default-placement", >> >> "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", >> >> "sync_policy": { >> >> "groups": [] >> >> } >> >> }, >> >> ... >> >> >> >> -- >> >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend >> im >> >> groüen Saal. >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users@xxxxxxx >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > >> > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |