Hi Jean: Thanks for your info. Unfortunately I check the secondary cluster and non-objects had been synced. The only way I have is to force rewrite objects for whole buckets. I have tried to set up multiple site between Nautilus and octopus. It works pretty well. But after I upgrade primary cluster to octopus, we have this issue. :( Here is the issue: https://tracker.ceph.com/issues/49542#change-193975 Thanks Jean-Sebastien Landry <Jean-Sebastien.Landry.6@xxxxxxxxx> 于2021年4月27日周二 下午7:52写道： > Hi, I hit the same errors when doing multisite sync between luminous and > octopus, but what I founded is that my sync errors was mainly on old > multipart and shadow objects, at the "rados level" if I might say. > (leftovers from luminous bugs) > > So check at the "user level", using s3cmd/awscli and the objects md5, > you will probably find that your pretty much in sync. Hopefully. > > Cheers! > > On 4/25/21 11:29 PM, 特木勒 wrote: > > [Externe UL*] > > > > Another problem I notice for a new bucket, the first object in the bucket > > will not be sync. the sync will start with the second object. I tried to > > fix the index on the bucket and manually rerun bucket sync, but the first > > object still does not sync with secondary cluster. > > > > Do you have any ideas for this issue? > > > > Thanks > > > > 特木勒 <twl007@xxxxxxxxx> 于2021年4月26日周一 上午11:16写道： > > > >> Hi Istvan: > >> > >> Thanks Amit's suggestion. > >> > >> I followed his suggestion to fix bucket index and re-do sync on buckets, > >> but it still did not work for me. > >> > >> Then I tried to use bucket rewrite command to rewrite all the objects in > >> buckets and it works for me. I think the reason is there's something > wrong > >> with bucket index and rewrite has rebuilt the index. > >> > >> Here's the command I use: > >> `sudo radosgw-admin bucket rewrite -b BUCKET-NAME --min-rewrite-size 0` > >> > >> Maybe you can try this to fix the sync issues. > >> > >> @Amit Ghadge <amitg.b14@xxxxxxxxx> Thanks for your suggestions. Without > >> your suggestions, I will not notice something wrong with index part. > >> > >> Thanks :) > >> > >> Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx> 于2021年4月26日周一 上午9:57写道： > >> > >>> Hi, > >>> > >>> > >>> > >>> No, doesn’t work, now we will write our own sync app for ceph, I gave > up. > >>> > >>> > >>> > >>> Istvan Szabo > >>> Senior Infrastructure Engineer > >>> --------------------------------------------------- > >>> Agoda Services Co., Ltd. > >>> e: istvan.szabo@xxxxxxxxx > >>> --------------------------------------------------- > >>> > >>> > >>> > >>> *From:* 特木勒 <twl007@xxxxxxxxx> > >>> *Sent:* Friday, April 23, 2021 7:50 PM > >>> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx> > >>> *Cc:* ceph-users@xxxxxxx > >>> *Subject:* Re: [Suspicious newsletter] RGW: Multiple Site > >>> does not sync olds data > >>> > >>> > >>> > >>> Hi Istvan: > >>> > >>> > >>> > >>> We just upgraded whole cluster to 15.2.10 and the multiple site still > >>> cannot sync whole objects to secondary cluster. 🙁 > >>> > >>> > >>> > >>> Do you have any suggestions on this? And I open another issues in ceph > >>> tracker site: > >>> > >>> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.ceph.com%2Fissues%2F50474&data=04%7C01%7Cjean-sebastien.landry%40dti.ulaval.ca%7C7758dcb00aa0481e098408d90863da13%7C56778bd56a3f4bd3a26593163e4d5bfe%7C1%7C0%7C637550047218082332%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=flvOwiaSDmDRQsbhzZsw5Q16hTmzUj9RXhcBrL6wsjo%3D&reserved=0 > >>> > >>> > >>> > >>> Hope someone could go to check this issue. > >>> > >>> > >>> > >>> Thanks > >>> > >>> > >>> > >>> 特木勒 <twl007@xxxxxxxxx>于2021年3月22日 周一下午9:08写道： > >>> > >>> Thank you~ > >>> > >>> > >>> > >>> I will try to upgrade cluster too. Seem like this is the only way for > >>> now. 😭 > >>> > >>> > >>> > >>> I will let you know once I complete testing. :) > >>> > >>> > >>> > >>> Have a good day > >>> > >>> > >>> > >>> Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx>于2021年3月22日 周一下午3:38写道： > >>> > >>> Yeah, doesn't work. Last week they fixed my problem ticket which caused > >>> the crashes, and due to the crashes stopped the replication I'll give > a try > >>> this week again after the update if the daemon doesn't crash, maybe it > will > >>> work, because if crash hasn't happened, the data was synced. Fingers > >>> crossed ;) Don't give up 😄 > >>> ------------------------------ > >>> > >>> *From:* 特木勒 <twl007@xxxxxxxxx> > >>> *Sent:* Monday, March 22, 2021 1:38 PM > >>> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx> > >>> *Cc:* ceph-users@xxxxxxx <ceph-users@xxxxxxx> > >>> > >>> > >>> *Subject:* Re: [Suspicious newsletter] RGW: Multiple Site > >>> does not sync olds data > >>> > >>> > >>> > >>> Hi Istvan: > >>> > >>> > >>> > >>> Do you have any update on directional sync? > >>> > >>> > >>> > >>> I am trying to upgrade cluster to 15.2.10 to see if the problem is > >>> solved. :( > >>> > >>> > >>> > >>> Thanks > >>> > >>> > >>> > >>> Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx> 于2021年3月1日周一 上午10:01写道： > >>> > >>> So-so. I had some interruption so it failed on one site, but the other > is > >>> kind of working. This is the first time when I saw data caught up in > the > >>> radosgw-admin data sync status on 1 side. > >>> > >>> Today will finish the other problematic site, I’ll let you know the > >>> result is it working or not. > >>> > >>> > >>> > >>> Istvan Szabo > >>> Senior Infrastructure Engineer > >>> --------------------------------------------------- > >>> Agoda Services Co., Ltd. > >>> e: istvan.szabo@xxxxxxxxx > >>> --------------------------------------------------- > >>> > >>> > >>> > >>> *From:* 特木勒 <twl007@xxxxxxxxx> > >>> *Sent:* Sunday, February 28, 2021 1:34 PM > >>> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx> > >>> *Cc:* ceph-users@xxxxxxx > >>> *Subject:* Re: [Suspicious newsletter] RGW: Multiple Site > >>> does not sync olds data > >>> > >>> > >>> > >>> Email received from outside the company. If in doubt don't click links > >>> nor open attachments! > >>> ------------------------------ > >>> > >>> Hi Istvan: > >>> > >>> > >>> > >>> Thanks for your reply. > >>> > >>> > >>> > >>> Does directional sync solve the problem? I tried to run `radosgw-admin > >>> sync init`, bit it still did not work. :( > >>> > >>> > >>> > >>> Thanks > >>> > >>> > >>> > >>> Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx> 于2021年2月26日周五 上午7:47写道： > >>> > >>> Same for me, 15.2.8 also. > >>> I’m trying directional sync now, looks like symmetrical has issue. > >>> > >>> Istvan Szabo > >>> Senior Infrastructure Engineer > >>> --------------------------------------------------- > >>> Agoda Services Co., Ltd. > >>> e: istvan.szabo@xxxxxxxxx<mailto:istvan.szabo@xxxxxxxxx> > >>> --------------------------------------------------- > >>> > >>> On 2021. Feb 26., at 1:03, 特木勒 <twl007@xxxxxxxxx> wrote: > >>> > >>> Email received from outside the company. If in doubt don't click links > >>> nor open attachments! > >>> ________________________________ > >>> > >>> Hi all: > >>> > >>> ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) > >>> > >>> I have a strange question, I just create a multiple site for Ceph > cluster. > >>> But I notice the old data of source cluster is not synced. Only new > data > >>> will be synced into second zone cluster. > >>> > >>> Is there anything I need to do to enable full sync for bucket or this > is a > >>> bug? > >>> > >>> Thanks > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users@xxxxxxx > >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>> > >>> ________________________________ > >>> This message is confidential and is for the sole use of the intended > >>> recipient(s). It may also be privileged or otherwise protected by > copyright > >>> or other legal rules. If you have received it by mistake please let us > know > >>> by reply email and delete it from your system. It is prohibited to copy > >>> this message or disclose its content to anyone. Any confidentiality or > >>> privilege is not waived or lost by any mistaken delivery or > unauthorized > >>> disclosure of the message. All messages sent to and from Agoda may be > >>> monitored to ensure compliance with company policies, to protect the > >>> company's interests and to remove potential malware. Electronic > messages > >>> may be intercepted, amended, lost or deleted, or contain viruses. > >>> > >>> > >>> > >>> Internal > >>> > >>> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > *ATTENTION : L’émetteur de ce courriel est externe à l’Université Laval. > > Évitez de cliquer sur un hyperlien, d’ouvrir une pièce jointe ou de > transmettre des informations si vous ne connaissez pas l’expéditeur du > courriel. En cas de doute, contactez l’équipe de soutien informatique de > votre unité ou hameconnage@xxxxxxxxx. > > ________________________________ > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |