Hi,
I have problem with RGW in multisite configuration with Nautilus 14.2.11. Both zones with SSD and 10Gbps network. Master zone consist from 5x DELL R740XD servers (every 256GB RAM, 8x800GB SSD for CEPH, 24xCPU). Secondary zone (temporary for testing) consist from 3x HPE DL360 Gen10 servers (every 256GB RAM, 6x800GB SSD, 48CPU).
We have 17 test buckets with manual sharding (101 shards). Every bucket with 10M of small objects (10kB - 15kB). Zonegroup configuration is attached bellow. Replication of 150M objects from master to secondary zone takes almost 28 hours and the replication completed with success.
After deleting objects from one bucket in master zone is not possible to sync zones properly. I tried to restart both secondary RGWs, but without success. Sync status on secondary zone is behind master. The number of objects in buckets on master zone is different than on secondary zone.
Ceph HEALTH status is WARNING on both zones. On master zone I have 146 large objects found in pool 'prg2a-1.rgw.buckets.index' 16 large objects found in pool 'prg2a-1.rgw.log'. On secondary zone 88 large objects found in pool 'prg2a-2.rgw.log' 1584 large objects found in pool 'prg2a-2.rgw.buckets.index'.
AVG OSD latencies on secondary zone during sync was "read 0,158ms, write 1,897ms, overwrite 1,634ms". After unsuccesfull sync (after 12h of sync fall down RGW requests, IOPS and throughput) jumps up AVG OSD latencies to "read 125ms, write 30ms, overwrite 272ms". After stopping of both RGWs on secondary zone are AVG OSD latencies almost 0ms, but when I start RGWs on secondary zone again, OSD latencies will rise again to "read 125ms, write 30ms, overwrite 272ms" with spikes up to 3 seconds.
We have seen the same behaviour of ceph multisite with large number of object in one bucket (150M+ objects), so we tried different strategy with smaller buckets, but results are same.
I will appreciate any help or advice, how tune or diagnose multisite problems.
Does anyone else have any ideas? Is there anyone else with a similar use-case? I do not know what is wrong.
Thank you and best regards,
Miroslav
radosgw-admin zonegroup get
{
"id": "ac0005da-2e9f-4f38-835f-72b289c240d0",
"name": "prg2a",
"api_name": "prg2a",
"is_master": "true",
"endpoints": [
"http://s3.prg1a.sys.cz:80";,
"http://s3.prg2a.sys.cz:80";
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "d9ebbd1f-3312-4083-b4c2-843e1fb899ad",
"zones": [
{
"id": "d9ebbd1f-3312-4083-b4c2-843e1fb899ad",
"name": "prg2a-1",
"endpoints": [
"http://10.104.200.101:7480";,
"http://10.104.200.102:7480";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
},
{
"id": "fdd76c02-c679-4ec7-8e7d-c14d2ac74fb4",
"name": "prg2a-2",
"endpoints": [
"http://10.104.200.221:7480";,
"http://10.104.200.222:7480";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "cb831094-e219-44b8-89f3-fe25fc288c00"
ii radosgw 14.2.11-pve1 amd64 REST gateway for RADOS distributed object store
ii ceph 14.2.11-pve1 amd64 distributed storage and file system
ii ceph-base 14.2.11-pve1 amd64 common ceph daemon libraries and management tools
ii ceph-common 14.2.11-pve1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fuse 14.2.11-pve1 amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 14.2.11-pve1 amd64 metadata server for the ceph distributed file system
ii ceph-mgr 14.2.11-pve1 amd64 manager for the ceph distributed storage system
ii ceph-mon 14.2.11-pve1 amd64 monitor server for the ceph storage system
ii ceph-osd 14.2.11-pve1 amd64 OSD server for the ceph storage system
ii libcephfs2 14.2.11-pve1 amd64 Ceph distributed file system client library
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
