Michael,
The problem looks very strange. We haven't come across such an issue (in
glusterfs) so far. However I do recall seeing such bit flips at a customer
site in the past, and in the end it was diagnosed to be a hardware issue.
Can you retry a few runs of same rsync directly to the backends through nfs
or rsyncd (without involving gluster or ssh), from the same client and same
data set, to both the servers and see if you can reproduce such a
md5sum/sha1sum mismatch?
Avati
On Mon, Aug 26, 2013 at 7:03 AM, Michael Peek <peek at nimbios.org> wrote:
> Hi gurus,
>
> This is a follow-up to a previous report about data integrity problems
> with Gluster 3.4.0. I will be as thorough as I can, but this is already a
> pretty long post. So feel free to see my previous post for more
> information specific to my previous run of tests.
>
>
> 1. I am running a fully up-to-date version of Ubuntu 12.04, with
> Gluster 3.4.0final-ubuntu1~precise1.
>
> 2. My cluster consists of four nodes. Each node consists of:
> 1. Hostnames: bkupc1-a -to- bkupc1-d
> 2. Bricks: Each host has /export/a/glusterfs/ and
> /export/b/glusterfs/, which are 4TB ext4 drives
> 3. Clients: I have a client that mounts the volume as /data/bkupc1/
> using the fuse driver.
>
> 3. My volume was created with:
> /usr/sbin/gluster peer probe bkupc1-a
> /usr/sbin/gluster peer probe bkupc1-b
> /usr/sbin/gluster peer probe bkupc1-c
> /usr/sbin/gluster peer probe bkupc1-d
> /usr/sbin/gluster volume create bkupc1 replica 2 transport tcp \
> bkupc1-a:/export/a/glusterfs bkupc1-b:/export/a/glusterfs \
> bkupc1-c:/export/a/glusterfs bkupc1-d:/export/a/glusterfs \
> bkupc1-a:/export/b/glusterfs bkupc1-b:/export/b/glusterfs \
> bkupc1-c:/export/b/glusterfs bkupc1-d:/export/b/glusterfs
> /usr/sbin/gluster volume set bkupc1 auth.allow {list of IP addresses}
>
> 4. On the client I have a 1TB drive filled with 900+GB of data in
> 156,554 test files. These files are encrypted backups that are dispersed
> throughout many subdirectories. They are ugly to look at. Here's an
> example:
>
> data/
> 884b9a38-0443-11e3-b8fb-f46d04e15793/
> 884a7040-0443-11e3-b8fb-f46d04e15793/
> 8825c6c8-0443-11e3-b8fb-f46d04e15793/
> 880f8f0c-0443-11e3-b8fb-f46d04e15793/
> iMmV,UqdiqZRie5QUu341iRS7s,-OK7PzXSuPgr0o30yNDXNG6uvqA0Wyr7RRR3MBE4
>
> <Line breaks for readability>
>
> I have pre-calculated MD5 and SHA1 checksums for all of these files,
> and I have verified that the checksums are correct on the client drive.
>
> 5. My first set of runs involved using rsync. Nothing fancy here:
> 1. The volume is empty when I begin
> 2. I create /data/bkupc1/BACKUPS-rsync.${timestamp}/
> 3. Use rsync to copy files from the client to the volume
> 4. Here's my script:
> #!/bin/bash -x
>
> timestamp="${1}"
>
> /bin/date
>
> mkdir /data/bkupc1/BACKUPS-rsync.${timestamp}
>
> rsync \
> -a \
> -v \
> --delete \
> --delete-excluded \
> --force \
> --ignore-errors \
> --one-file-system \
> --stats \
> --inplace \
> ./ \
> /data/bkupc1/BACKUPS-rsync.${timestamp}/ \
> #
>
> /bin/date
>
> (\
> cd /data/bkupc1/BACKUPS-rsync.${timestamp}/ \
> && md5sum -c --quiet md5sums \
> )
>
> /bin/date
>
> (\
> cd /data/bkupc1/BACKUPS-rsync.${timestamp}/ \
> && sha1sum -c --quiet sha1sums \
> )
>
> /bin/date
>
> /usr/bin/diff -r -q ./ /data/bkupc1/BACKUPS-rsync.${timestamp}/
>
> /bin/date
>
> 5. As you can see from the script, after rsyncing, I check the
> files on the volume
> 1. Against their MD5 checksums
> 2. Then against their SHA1 checksums
> 3. Then, just to beat a dead horse, I use diff to do a
> byte-for-byte check between the files on the client and the files on the
> volume. (Note to self: I should replace diff with cmp, as I have run into
> "out of memory" errors with diff on files that cmp can handle just fine.)
>
> 6. What I have found is that about 50% of the time, there will
> be one or two files out of those 156,554 that differ. I documented my
> findings in more detail in my previous email.
>
> 6. One though that occurred to me is that this could be the fault
> of rsync. So I have repeated the tests using plain old /bin/cp. Here's my
> (very similar) script:
> #!/bin/bash -x
>
> timestamp="${1}"
>
> /bin/date
>
> mkdir /data/bkupc1/BACKUPS-cp.${timestamp}
>
> /bin/cp -ar ./ /data/bkupc1/BACKUPS-cp.${timestamp}/
>
> /bin/date
>
> (\
> cd /data/bkupc1/BACKUPS-cp.${timestamp}/ \
> && md5sum -c --quiet md5sums \
> )
>
> /bin/date
>
> (\
> cd /data/bkupc1/BACKUPS-cp.${timestamp}/ \
> && sha1sum -c --quiet sha1sums \
> )
>
> /bin/date
>
> /usr/bin/diff -r -q ./ /data/bkupc1/BACKUPS-cp.${timestamp}/
>
> /bin/date
>
> 7. Results:
> 1. Output from the script:
> + timestamp=20130821-081918
> + /bin/date
> Wed Aug 21 08:19:18 EDT 2013
> + mkdir /data/bkupc1/BACKUPS-cp.20130821-081918
> + /bin/cp -ar ./ /data/bkupc1/BACKUPS-cp.20130821-081918/
> + /bin/date
> Wed Aug 21 13:51:53 EDT 2013
> + cd /data/bkupc1/BACKUPS-cp.20130821-081918/
> + md5sum -c --quiet md5sums
> data/884b9a38-0443-11e3-b8fb-f46d04e15793/884a7040-0443-11e3-b8fb-f46d04e15793/87fdc790-0443-11e3-b8fb-f46d04e15793/87f54d22-0443-11e3-b8fb-f46d04e15793/KAfe4MUAlmO-Lt4N0KqVQTtf0im3mcoTuAyJvSP,t0o2Lc,FGce49pe9wEPDiIIt201oEks-taGDbc5-Nph6AacR:
> FAILED
> data/a34bc588-0443-11e3-b8fb-f46d04e15793/a34a8bf0-0443-11e3-b8fb-f46d04e15793/a3494b3c-0443-11e3-b8fb-f46d04e15793/a34808b2-0443-11e3-b8fb-f46d04e15793/a346cd08-0443-11e3-b8fb-f46d04e15793/a3456e2c-0443-11e3-b8fb-f46d04e15793/a344366a-0443-11e3-b8fb-f46d04e15793/8c9e94a0-0443-11e3-b8fb-f46d04e15793/NLrXi5u80FoUV6Gi2ouEybAebOgnF7p1PtEYmPbd0huh,1:
> FAILED
> md5sum: WARNING: 2 computed checksums did NOT match
> + /bin/date
> Wed Aug 21 16:54:13 EDT 2013
> + cd /data/bkupc1/BACKUPS-cp.20130821-081918/
> + sha1sum -c --quiet sha1sums
> data/884b9a38-0443-11e3-b8fb-f46d04e15793/884a7040-0443-11e3-b8fb-f46d04e15793/8825c6c8-0443-11e3-b8fb-f46d04e15793/8810e21c-0443-11e3-b8fb-f46d04e15793/7LOu,NZ5eMXxxrqjZHv5a9-4aHd641hN2tGaneMa1D2Kl9wLXf1f71nX6g-8ps2BpABovO7w68Wy63pH0gU3yLnyLEfFfT25Zk5jNvpDU6eQ,1:
> FAILED
> sha1sum: WARNING: 1 computed checksum did NOT match
> + /bin/date
> Wed Aug 21 19:54:29 EDT 2013
> + /usr/bin/diff -r -q ./ /data/bkupc1/BACKUPS-cp.20130821-081918/
> + /bin/date
> Thu Aug 22 00:16:53 EDT 2013
>
> 2. A listing of files that were reported as different (line breaks
> for readability):
> 1. MD5 failure:
> data/
> 884b9a38-0443-11e3-b8fb-f46d04e15793/
> 884a7040-0443-11e3-b8fb-f46d04e15793/
> 87fdc790-0443-11e3-b8fb-f46d04e15793/
> 87f54d22-0443-11e3-b8fb-f46d04e15793/
>
> KAfe4MUAlmO-Lt4N0KqVQTtf0im3mcoTuAyJvSP,t0o2Lc,FGce49pe9wEPDiIIt201oEks-taGDbc5-Nph6AacR
>
> 2. MD5 failure:
> data/
> a34bc588-0443-11e3-b8fb-f46d04e15793/
> a34a8bf0-0443-11e3-b8fb-f46d04e15793/
> a3494b3c-0443-11e3-b8fb-f46d04e15793/
> a34808b2-0443-11e3-b8fb-f46d04e15793/
> a346cd08-0443-11e3-b8fb-f46d04e15793/
> a3456e2c-0443-11e3-b8fb-f46d04e15793/
> a344366a-0443-11e3-b8fb-f46d04e15793/
> 8c9e94a0-0443-11e3-b8fb-f46d04e15793/
> NLrXi5u80FoUV6Gi2ouEybAebOgnF7p1PtEYmPbd0huh,1
>
> 3. SHA1 failure:
> data/
> 884b9a38-0443-11e3-b8fb-f46d04e15793/
> 884a7040-0443-11e3-b8fb-f46d04e15793/
> 8825c6c8-0443-11e3-b8fb-f46d04e15793/
> 8810e21c-0443-11e3-b8fb-f46d04e15793/
>
> 7LOu,NZ5eMXxxrqjZHv5a9-4aHd641hN2tGaneMa1D2Kl9wLXf1f71nX6g-8ps2BpABovO7w68Wy63pH0gU3yLnyLEfFfT25Zk5jNvpDU6eQ,1
>
> 3. A byte-for-byte comparison:
> 1. File from 7.2.1 above:
>
> (KAfe4MUAlmO-Lt4N0KqVQTtf0im3mcoTuAyJvSP,t0o2Lc,FGce49pe9wEPDiIIt201oEks-taGDbc5-Nph6AacR)
> 1. After the test, this file exist on three locations:
> client:/export/d/eraseme/ <-- the original
> bkupc1-a:/export/a/glusterfs/ <-- replicated volume copy 1
> of 2
> bkupc1-b:/export/a/glusterfs/ <-- replicated volume copy 2
> of 2
>
> 2. MD5sums:
> 68ce7073e462fda42d4b551a843bd71f <-- bkupc1-a (directly from
> the brick)
> 68ce7073e462fda42d4b551a843bd71f <-- bkupc1-b (directly from
> the brick)
> 68ce7073e462fda42d4b551a843bd71f <-- client
> 68ce7073e462fda42d4b551a843bd71f <-- volume (from the mount
> via the fuse driver)
>
> NOTE: There is no difference between the MD5 checksums
>
> 3. SHA1sums:
> c5c59c18f5cc0c1b6e4dd80b2d41fc3bc7148509 <-- bkupc1-a
> c5c59c18f5cc0c1b6e4dd80b2d41fc3bc7148509 <-- bkupc1-b
> c5c59c18f5cc0c1b6e4dd80b2d41fc3bc7148509 <-- client
> c5c59c18f5cc0c1b6e4dd80b2d41fc3bc7148509 <-- volume
>
> NOTE: There is no difference between the SHA1 checksums
>
> 4. Both /usr/bin/diff and /usr/bin/cmp report no difference
> between these files.
>
> 2. File 7.2.2 from above:
> (NLrXi5u80FoUV6Gi2ouEybAebOgnF7p1PtEYmPbd0huh,1)
> 1. After the test, this file exists on three locations:
> client:/export/d/eraseme/ <-- the original
> bkupc1-a:/export/a/glusterfs/ <-- replicated volume copy 1
> of 2
> bkupc1-b:/export/a/glusterfs/ <-- replicated volume copy 2
> of 2
>
> 2. MD5sums:
> 78696407263ef75ae2795ed7cb4eb24a <-- bkupc1-a
> 77fdce4ebe9e94f611848d174de01357 <-- bkupc1-b
> 78696407263ef75ae2795ed7cb4eb24a <-- client
> 78696407263ef75ae2795ed7cb4eb24a <-- volume
>
> 3. SHA1sums:
> de93bcc7b64458926505dfc5ac4c597f3fefe6db <-- bkupc1-a
> 0254c117b92ca95987aa7389980fb0bcc850e9c5 <-- bkupc1-b
> de93bcc7b64458926505dfc5ac4c597f3fefe6db <-- client
> de93bcc7b64458926505dfc5ac4c597f3fefe6db <-- volume
>
> 4. Byte differences: The output of /usr/bin/cmp -l, when
> comparing the version of the file on the client with the version of the
> file on bkupc1-b:
>
> 3262724555 274 234
>
> If I'm reading this right, then this means that the files
> differ by only one byte (274 vs. 234).
>
> 3. File 7.2.3 from above:
>
> (7LOu,NZ5eMXxxrqjZHv5a9-4aHd641hN2tGaneMa1D2Kl9wLXf1f71nX6g-8ps2BpABovO7w68Wy63pH0gU3yLnyLEfFfT25Zk5jNvpDU6eQ,1)
> 1. After the test, this file exists on three locations:
> client:/export/d/eraseme/ <-- the original
> bkupc1-c:/export/a/glusterfs/ <-- replicated volume copy 1
> of 2
> bkupc1-d:/export/a/glusterfs/ <-- replicated volume copy 2
> of 2
>
> 2. MD5sums:
> 05ea9e04df984cc7ed514f93dc79067e <-- bkupc1-c
> 868c0eafa2bde7386d808b722166a283 <-- bkupc1-d
> 05ea9e04df984cc7ed514f93dc79067e <-- client
> 868c0eafa2bde7386d808b722166a283 <-- volume
>
> 3. SHA1sums:
> a6cf53c106b1856826db8de8b947273b05eb6391 <-- bkupc1-c
> f56e01f982028eed9c115f0696346861bf3b7169 <-- bkupc1-d
> a6cf53c106b1856826db8de8b947273b05eb6391 <-- client
> f56e01f982028eed9c115f0696346861bf3b7169 <-- volume
>
> 4. Byte differences: The output of /usr/bin/cmp -l, when
> comparing the version of the file on the client with the version of the
> file on bkupc1-d:
>
> 181361479 226 206
>
>
> For 7.3.1, md5sum reported a difference between the client and the volume,
> even though re-comparing it with md5sum, sha1sum, cmp, and diff on both the
> mounted volume and the individual bricks showed no difference at all. This
> would imply that there may be a data read error somewhere in gluster.
>
> For 7.3.2 and 7.3.3, each file differed between the client, the volume,
> and between replica bricks, by exactly one byte (if I've read the output
> from cmp correctly). This would imply that there may be a data write error
> in gluster.
> (And please, for the love of Pete, tell me if I have done anything
> stupid, b/c I really wanted gluster to be the silver bullet solution to my
> data storage problems.)
>
> Michael Peek
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130826/4668776c/attachment-0001.html>
