On Sat, Feb 17, 2018 at 3:16 AM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote: > Now that the sync-to-the-cloud work is almost complete, I was thinking > a bit and did some research about bi-directional sync. The big > difficulty I had with the syncing from the cloud process is the need > to rework the whole data sync paths where we identify changes. These > are quite complicated, and these kind of changes are quite a big > project. I'm not quite sure now that this is needed. > What I think we could do in a relatively easy (and less risky) way is > that instead of embedding a new mechanism within the sync logic, we > can create a module that turns upstream cloud changes into the > existing rgw logs: that is, data log, and bucket index logs (no > metadata log needed). In this way we break the problem into two > separate issues, where one of the issues is already solved. The > ingesting rgw could then do the same work it is doing with regular > zones (fetching these logs, and pulling the data from a remote > endpoint) -- albeit with various slight changes that are required > since we can't have some of the special apis that we created to assist > us. Sounds like a good plan, it may increase the time we detect changes. If we can give the user an estimation I think it will be acceptable. > We'll need to see how these could be replaced and what will be the > trade offs, but we'll need to do that anyway with any solution. > The changes discovery module that will turn remote cloud changes into > local logs could do it by either polling the remote endpoints, or (for > S3 for example) could use buckets notifications mechanism. It will > build its local changes logs by setting new entries on them according > to the changes it identifies. The radosgw zone that will sync from the > cloud will have two endpoints one that will be used to fetch the the > logs, and another one that will be used to sync in the data. > I'm a bit simplifying it, there are a few more issues there, but > that's the gist of it. > > Any thoughts? can this used for syncing indexless buckets? Regards, Orit > > Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
