Whatever we choose, we'll need to provide specific guidelines so that people can understand what level of data integrity they actually have. They'll be able to get the HW UBER from their vendor, but they will need specific formulas, etc. to understand how the various block-sizes and algorithmic choices yield a system-level of UBER. Allen Samuels Software Architect, Fellow, Systems and Software Solutions 2880 Junction Avenue, San Jose, CA 95134 T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > -----Original Message----- > From: Gregory Farnum [mailto:gfarnum@xxxxxxxxxx] > Sent: Wednesday, March 30, 2016 4:04 PM > To: Sage Weil <sage@xxxxxxxxxxxx> > Cc: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Igor Fedotov > <ifedotov@xxxxxxxxxxxx>; ceph-devel <ceph-devel@xxxxxxxxxxxxxxx> > Subject: Re: Adding compression/checksum support for bluestore. > > On Wed, Mar 30, 2016 at 3:57 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote: > > On Wed, 30 Mar 2016, Allen Samuels wrote: > >> One thing to also factor in is that if you increase the span of a > >> checksum, you degrade the quality of the checksum. So if you go with > >> 128K chunks of data you'll likely want to increase the checksum > >> itself from something beyond a CRC-32. Maybe somebody out there has a > >> good way of describing this quanitatively. > > > > Good point. FWIW, I think we should default to xxhash over crc32c: > > > > https://github.com/Cyan4973/xxHash > > > > Note that there is a 64-bit version that's faster on 64-bit procs. > > Random googling (...and StackOverflow) lead me to > https://users.ece.cmu.edu/~koopman/roses/dsn04/koopman04_crc_poly_ > embedded.pdf, > which only extends up to 2KB (with a crc16, which makes me think crc32 can > go a long way) but for anybody who actually reads it can probably be > extended to larger block sizes without much difficulty. > -Greg ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
