On Mon, Sep 10, 2018 at 1:35 PM <Patrick.Mclean@xxxxxxxx> wrote: > > Hi, > We utilize Ceph RBDs for our users' storage and need to keep data synchronized across data centres. For this we rely on 'rbd export-diff / import-diff'. Lately we have been noticing cases in which the file system on the 'destination RBD' is corrupt. We have been trying to isolate the issue, which may or may not be due to Ceph. We suspect the problem could be in 'rbd export-diff / import-diff' and are wondering if people have been seeing issues with these tools. Let me explain our use case and issue in more detail. > We have a number of data centres each with a Ceph cluster storing tens of thousands of RBDs. We maintain extra copies of each RBD in other data centres. After we are 'done' using a RBD, we create a snapshot and use 'rbd export-diff' to create a diff between the most recent 'common' snapshot at the other data center. We send the data over the network, and use 'rbd import-diff' on the destination. When we apply a diff to a destination RBD we can guarantee its 'HEAD' is clean. Of course we guarantee that an RBD is only used in one data centre at a time. > We noticed corruption at the destination RBD based on fsck failures, further investigation showed that checksums on the RBD mismatch as well. Somehow the data is sometimes getting corrupted either by our software or 'rbd export-diff / import-diff'. Our investigation suggests that the the problem is in 'rbd export-diff/import-diff'. The main evidence of this is that occasionally we sync an RBD between multiple data centres. Each sync is a separate job with its own 'rbd export-diff'. We noticed that both destination locations have the same corruption (and the same checksum) and the source is healthy. Any chance you are using OSD tiering on your RBD pool? The export-diffs from a cache tier pool are almost guaranteed to be corrupt if that's the case since the cache tier provides incorrect object diff stats [1]. > In addition to this, we are seeing a similar type of corruption in another use case when we migrate RBDs and snapshots across pools. In this case we clone a version of an RBD (e.g. HEAD-3) to a new pool and rely on 'rbd export-diff/import-diff' to restore the last 3 snapshots on top. Here too we see cases of fsck and RBD checksum failures. > We maintain various metrics and logs. Looking back at our data we have seen the issue at a small scale for a while on Jewel, but the frequency increased recently. The timing may have coincided with a move to Luminous, but this may be coincidence. We are currently on Ceph 12.2.5. > We are wondering if people are experiencing similar issues with 'rbd export-diff / import-diff'. I'm sure many people use it to keep backups in sync. Since it is backups, many people may not inspect the data often. In our use case, we use this mechanism to keep data in sync and actually need the data in the other location often. We are wondering if anyone else has encountered any issues, it's quite possible that many people may have this issue, buts simply don't realize. We are likely hitting it much more frequently due to the scale of our operation (tens of thousands of syncs a day). If you are able to recreate this reliably without tiering, it would assist in debugging if you could capture RBD debug logs during the export along w/ the LBA of the filesystem corruption to compare against. > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [1] http://tracker.ceph.com/issues/20896 -- Jason _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |