On Mon, Jun 29, 2015 at 8:31 AM, Dałek, Piotr <Piotr.Dalek@xxxxxxxxxxxxxx> wrote: >> -----Original Message----- >> From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel- >> owner@xxxxxxxxxxxxxxx] On Behalf Of Erik G. Burrows >> Sent: Friday, June 26, 2015 6:49 PM >> >> All, >> Can someone explain to me the rationale for performing in-software CRC32 >> hashes of all messages through the Pipe and AsyncMessage classes? >> >> On my servers, operf shows that 20% of the total CPU time in my benchmark >> tests are being spent in the librados ceph_crc32c_sctp function. I can see that >> the library is trying to use CPU accelerations if available, but what I'd like to >> understand is: why checksum the messages at all? > > As Somnath already wrote, you can disable CRC checking for messages. But they're also used for journals, among other things, so you'll always see some CPU usage spent on CRC32 calculations. > >> If the messages are local, there should not be any corruption at all, and if >> they are coming in over IP, then the kernel and NIC should do Layer-2/3 CRCs >> and reject any corrupted packets. So why re-CRC the messages at the Ceph >> layer? > > I can imagine data corruption coming from Ceph itself and not caught by IP layers, for example due to bug in Ceph code or mainboard/RAM failure. And it's a nice debug feature you can use when dealing with low-level code. > That's not to mention that the TCP checksum is remarkably weak. We've just had an incident where a broken router was quite efficiently corrupting something like 1/66 packets in a way which was invisible to the TCP checksum. Some example corruptions are here our report -- note that it's still a work in progress: https://cds.cern.ch/record/2026187/files/Adler32_Data_Corruption.pdf Thankfully CRC32-C /probably/ prevented this broken router from corrupting our Ceph volumes. Cheers, Dan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
