On Thu, 2 Oct 2014 08:28:16 Sage Weil wrote:
> My guess is a btrfs issue. The weird thing about your report is the byte
> totals are off by an uneven number of bytes (3 bytes, 9 bytes, etc.).
> We haven't ever seen this. We do test RBD over cache tiers on btrfs,
> but not with EC on the base. I'll add that combo to the matrix. My first
> guess is a btrfs issue, honestly.
I think I found where it is happening: for a while I was using Btrfs-based
OSDs with journals on ext4 partition on SSD. As an experiment I've decided to
try moving all journal files back to their OSDs and it eliminated
inconsistencies. I've updated the ticket with this information.
This behaviour is reproducible on 0.80.6.
It looks like Btrfs snapshotting do not affect this issue.
> Does it continue to come up after the kernels are upgraded (and after a
> full cycle of scrub and repairs have been done to clear out
> inconsistencies introduced while running the older kernel)?
Yes, I tried many times after every kernel update or any change in cluster
whatsoever. Repair is usually ineffective and doesn't change anything: it
would log "repair 1 errors, 1 fixed" but "ceph pg scrub" will find an error
right away. Moreover repair is not even necessary -- inconsistencies stay on
some PGs for a while then "move" to different PGs. For example "ceph pg scrub
19.NN" sometimes would be clearing affected pg from "inconsistent" state or
discover a new inconsistency seemingly at random.
Thank you.
--
Cheers,
Dmitry Smirnov.
---
Odious ideas are not entitled to hide from criticism behind the human
shield of their believers' feelings.
-- Richard Stallman
Attachment:
signature.asc
Description: This is a digitally signed message part.
