> On 09 Sep 2014, at 16:39, Michal Kozanecki <mkozanecki at evertz.com> wrote: > On 9 September 2014 08:47, Blair Bethwaite <blair.bethwaite at gmail.com> wrote: >> On 9 September 2014 20:12, Dan Van Der Ster <daniel.vanderster at cern.ch> wrote: >>> One thing I?m not comfortable with is the idea of ZFS checking the data in addition to Ceph. Sure, ZFS will tell us if there is a checksum error, but without any redundancy at the ZFS layer there will be no way to correct that error. Of course, the hope is that RADOS will ensure 100% data consistency, but what happens if not?... >> >> The ZFS checksumming would tell us if there has been any corruption, which as you've pointed out shouldn't happen anyway on top of Ceph. > > Just want to quickly address this, someone correct me if I'm wrong, but IIRC even with replica value of 3 or more, ceph does not(currently) have any intelligence when it detects a corrupted/"incorrect" PG, it will always replace/repair the PG with whatever data is in the primary, meaning that if the primary PG is the one that?s corrupted/bit-rotted/"incorrect", it will replace the good replicas with the bad. According to the the "scrub error on firefly? thread, repair "tends to choose the copy with the lowest osd number which is not obviously corrupted. Even with three replicas, it does not do any kind of voting at this time.? Cheers, Dan
 |