Hum, it seems I hit that bug : https://tracker.ceph.com/issues/50785
Merged in Pacific, not backported to Octopus, at least not linked to
this issue.
One more reason to migrate to Pacific...
Le 2022-03-15 10:43, Gilles Mocellin a écrit :
Hello Cephers,
We've just configured multi-site between an existing Octopus cluster
and a new one in another Datacenter.
On our biggest bucket (replic_cfn_prod/cfb : 1.4 M objects, 670 GB) we
have many errors like that on the new site :
2022-03-15T10:21:00.800+0100 7f9834750700 1 ====== starting new
request req=0x7f9988470620 =====
2022-03-15T10:21:00.800+0100 7f9834750700 1 ====== req done
req=0x7f9988470620 op status=0 http_status=200 latency=0s ======
2022-03-15T10:21:00.800+0100 7f9834750700 1 beast: 0x7f9988470620:
100.69.103.105 - - [2022-03-15T10:21:00.800105+0100] "HEAD / HTTP/1.0"
200 5 - - -
2022-03-15T10:21:00.912+0100 7f998b7fe700 0 ERROR: curl error:
Transferred a partial file, maybe network unstable
2022-03-15T10:21:00.912+0100 7f996b7fe700 0 store->fetch_remote_obj()
returned r=-11
2022-03-15T10:21:01.068+0100 7f98bf7fe700 0
RGW-SYNC:data:sync:shard[75]:entry[replic_cfn_prod/cfb:aefd4003-1866-4b16-b1b3-2f308075cd1c.9774642.1:49]:bucket_sync_sources[target=:[]):source_bucket=:[]):source_zone=aefd4003-1866-4b16-b1b3-2f308075cd1c]:bucket[replic_cfn_prod/cfb:aefd4003-1866-4b16-b1b3-2f308075cd1c.9774642.1:49<-replic_cfn_prod/cfb:aefd4003-1866-4b16-b1b3-2f308075cd1c.9774642.1:49]:inc_sync[replic_cfn_prod/cfb:aefd4003-1866-4b16-b1b3-2f308075cd1c.9774642.1:49]:entry[10264336-2213-11ec-a9c2-fa163e1fd8b9/615ac4886acbe-cloture-littoral-949-86.pdf]:
ERROR: failed to sync object:
replic_cfn_prod/cfb:aefd4003-1866-4b16-b1b3-2f308075cd1c.9774642.1:49/10264336-2213-11ec-a9c2-fa163e1fd8b9/615ac4886acbe-xxx-949-86.pdf
2022-03-15T10:21:01.140+0100 7f998b7fe700 0 ERROR: curl error:
Transferred a partial file, maybe network unstable
2022-03-15T10:21:01.140+0100 7f99727fc700 0 store->fetch_remote_obj()
returned r=-11
2022-03-15T10:21:01.428+0100 7f998b7fe700 0 ERROR: curl error:
Transferred a partial file, maybe network unstable
2022-03-15T10:21:01.432+0100 7f99677f6700 0 store->fetch_remote_obj()
returned r=-11
The network seems good, I have some retries if I benchmark with iperf3
and 10 simultaneous transfers. But manual curl work.
One thing I noticed : radosgw / beast, even if bounded to a specific
interface/network IP, the front storage, it uses another source IP for
communication, if the default gateway is not on the storage network.
In our conf, we have 3 networks :
- management (for ssh... it is the default gateway)
- storage (Ceph "public" network)
- replication (Ceph cluster network)
Ping from one node to another on the storage network has the
management ip as source.
We added source routing on nodes so that responses on the storage
network leaves via the storage network.
As radosgw is bound to our storage network IP (we can see it with
netstat / lsof), I thought that outgoing packets will have that IP.
But no. Requests for replication in multi-site are made with a source
IP of the management network.
It doesn't sound good, so I added a static route to my storage subnet
of the other zone via the storage network.
At first, I thought it was better, but, with some time I finally had
the same errors.
Does someone have a clue on the curl error: Transferred a partial
file, maybe network unstable ?
PS:
On the first cluster, we had problems with large omaps, and resharding
that would not pass because of bi_list() I/O errors
(https://tracker.ceph.com/issues/51429)
We had re-created the bucket, and manually shard it before activating
the multi-site sync.
We still have 6 Large OMAP objects. I Don't know if there are old ones
that will be corrected with deep-scrub (actually running on the
concerned PG).
Perhaps it can cause issues, but `radosgw-admin bi list` is now
running fine on the bucket.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
