ECC will not be able to recover the data, but it will always be able to detect that data is corrupted. AFAIK under Linux this results in immediate halt of system, so it would not be able to report bad checksum data during deep-scrub. -- Tomasz Kuzemko tomasz.kuzemko@xxxxxxxxxxxx W dniu 26.11.2015 o 15:46, Major Csaba pisze: > Hi, > > We don't use ECC modules but the ECC doesn't mean you're safe. > > See the presentation I linked earlier: > <https://www.nsc.liu.se/lcsc2007/presentations/LCSC_2007-kelemen.pdf%3Ehttps://www.nsc.liu.se/lcsc2007/presentations/LCSC_2007-kelemen.pdf>https://www.nsc.liu.se/lcsc2007/presentations/LCSC_2007-kelemen.pdf>https://www.nsc.liu.se/lcsc2007/presentations/LCSC_2007-kelemen.pdf > > You can see it also here, for example: > <https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/>https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/ > "/However, it's important to note that ECC RAM can only correct 1 bit > flip per byte (8 bits). If you have 2 bit flips per byte, ECC RAM will > not be able to recover the data./" > > Regards, > Csaba > > On 11/26/2015 03:31 PM, Tomasz Kuzemko wrote: >> Hi, >> I have also seen inconsistent PGs despite md5 being the same on all >> objects, however all my hardware uses ECC RAM, which as I understand >> should prevent this type of error. To be clear - in your case you were >> using ECC or non-ECC module? >> >> -- >> Tomasz Kuzemko >> tomasz.kuzemko@xxxxxxx >> >> W dniu 26.11.2015 o 15:23, Major Csaba pisze: >>> Hi, >>> >>> On 11/25/2015 06:41 PM, Robert LeBlanc wrote: >>>> Since the one that is different is not your primary for the pg, then >>>> pg repair is safe. >>> Ok, that's clear thanks. >>> I think we managed to identify the root cause of the scrubbing errors >>> even if the files are identical. >>> It seems to be a hardware issue (faulty RAM module), which is really >>> hard to detect, even if you have an ECC capable module. >>> >>> The glitch happens here: >>> *node2:~# while true; do sha1sum >>> /var/lib/ceph/osd/ceph-12/current/1.14e_head/DIR_E/DIR_4/DIR_D/rb.0.532aa.238e1f29.000000016ce4__head_DE208D4E__1; >>> sleep 0.1; done** >>> **acd62deb72530e22b7ebdce3e2e47e0480af533b >>> /var/lib/ceph/osd/ceph-12/current/1.14e_head/DIR_E/DIR_4/DIR_D/rb.0.532aa.238e1f29.000000016ce4__head_DE208D4E__1** >>> **... >>> **acd62deb72530e22b7ebdce3e2e47e0480af533b >>> /var/lib/ceph/osd/ceph-12/current/1.14e_head/DIR_E/DIR_4/DIR_D/rb.0.532aa.238e1f29.000000016ce4__head_DE208D4E__1** >>> **acd62deb72530e22b7ebdce3e2e47e0480af533b >>> /var/lib/ceph/osd/ceph-12/current/1.14e_head/DIR_E/DIR_4/DIR_D/rb.0.532aa.238e1f29.000000016ce4__head_DE208D4E__1** >>> **acd62deb72530e22b7ebdce3e2e47e0480af533b >>> /var/lib/ceph/osd/ceph-12/current/1.14e_head/DIR_E/DIR_4/DIR_D/rb.0.532aa.238e1f29.000000016ce4__head_DE208D4E__1** >>> **acd62deb72530e22b7ebdce3e2e47e0480af533b >>> /var/lib/ceph/osd/ceph-12/current/1.14e_head/DIR_E/DIR_4/DIR_D/rb.0.532aa.238e1f29.000000016ce4__head_DE208D4E__1** >>> **4163ca9a76ed7b0b9f0e69ab5a1793cd1cf7d1c4 >>> /var/lib/ceph/osd/ceph-12/current/1.14e_head/DIR_E/DIR_4/DIR_D/rb.0.532aa.238e1f29.000000016ce4__head_DE208D4E__1** >>> **....* >>> >>> So, sometimes it calculates different values. We managed to copy this >>> file several times to find the difference: >>> *# diff 48.bin 49.bin ** >>> **40095c40095** >>> **< >>> hMxs8+iPzA5BRi/Nq4iuovTkR/Q9RXV15qgHTiGO6jtPvjT5bdQFZQH8BuCP65E4JDmn8yC7/laC** >>> **---** >>> **> >>> hMxs8+iPzA5BRi/Nq4iuovTkR/Q9RXV15qgHTiGO6jtTvjT5bdQFZQH8BuCP65E4JDmn8yC7/laC* >>> So, it has a single bit difference (0x50 vs 0x54) >>> >>> I think this presentation could be very useful about the silent >>> corruption of data: >>> <https://www.nsc.liu.se/lcsc2007/presentations/LCSC_2007-kelemen.pdf>https://www.nsc.liu.se/lcsc2007/presentations/LCSC_2007-kelemen.pdf >>> >>> We will test all of our RAM modules now (it should have happened before, >>> of course...), but it seems you have to be very careful with the cheap >>> commodity hardware. >>> >>> Regards, >>> Csaba >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
