Posted 2 files ceph-1-rbd_data.3ef8442ae8944a.0000000000000aff ceph-post-file: 10cddc98-7177-47d8-9a97-4868856f974b ceph-7-rbd_data.3ef8442ae8944a.0000000000000aff ceph-post-file: f2861c0a-fc9a-4078-b95c-d5ba3cf6057e By the way, file names were found like this /var/lib/ceph/osd/ceph-1/current/3.d8_head/DIR_8/DIR_D/DIR_2/rbd\udata.3ef8442ae8944a.0000000000000aff__head_CC49F2D8__3 /var/lib/ceph/osd/ceph-7/current/3.d8_head/DIR_8/DIR_D/DIR_2/DIR_F/rbd\udata.3ef8442ae8944a.0000000000000aff__head_CC49F2D8__3 There is "rbd\udata." in the prefix, not "rbd_data.". Don't know if it relevant to this case or not. Just FYI. Сахинов Константин тел.: +7 (909) 945-89-42 2015-08-09 17:23 GMT+03:00 Sage Weil <sage@xxxxxxxxxxxx>: > On Sat, 8 Aug 2015, ?????????? ??????? wrote: >> Hi! >> >> I have a large number of inconsistent pgs 229 of 656, and it's >> increasing every hour. >> I'm using ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3). >> >> For example, pg 3.d8: >> # ceph health detail | grep 3.d8 >> pg 3.d8 is active+clean+scrubbing+deep+inconsistent, acting [1,7] >> >> # grep 3.d8 /var/log/ceph/ceph-osd.1.log | less -S >> 2015-08-07 13:10:48.311810 7f5903f7a700 0 log_channel(cluster) log >> [INF] : 3.d8 repair starts 2015-08-07 13:12:05.703084 7f5903f7a700 -1 >> log_channel(cluster) log [ERR] : repair 3.d8 >> cbd2d0d8/rbd_data.6a5cf474b0dc51.0000000000000b1f/head//3 on disk data >> digest 0x6e4d80bf != 0x6fb5b103 2015-08-07 13:13:26.837524 >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : repair 3.d8 >> b5892d8/rbd_data.dbe674b0dc51.00000000000001b9/head//3 on disk data >> digest 0x79082779 != 0x9f102f3d 2015-08-07 13:13:44.874725 >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : repair 3.d8 >> ee6dc2d8/rbd_data.e7592ae8944a.0000000000000833/head//3 on disk data >> digest 0x63ab49d0 != 0x68778496 2015-08-07 13:14:19.378582 >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : repair 3.d8 >> d93e14d8/rbd_data.3ef8442ae8944a.0000000000000729/head//3 on disk data >> digest 0x3cdb1f5c != 0x4e0400c2 2015-08-07 13:23:38.668080 >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : 3.d8 repair 4 errors, >> 0 fixed 2015-08-07 13:23:38.714668 7f5903f7a700 0 log_channel(cluster) >> log [INF] : 3.d8 deep-scrub starts 2015-08-07 13:25:00.656306 >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : deep-scrub 3.d8 >> cbd2d0d8/rbd_data.6a5cf474b0dc51.0000000000000b1f/head//3 on disk data >> digest 0x6e4d80bf != 0x6fb5b103 2015-08-07 13:26:18.775362 >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : deep-scrub 3.d8 >> b5892d8/rbd_data.dbe674b0dc51.00000000000001b9/head//3 on disk data >> digest 0x79082779 != 0x9f102f3d 2015-08-07 13:26:42.084218 >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : deep-scrub 3.d8 >> ee6dc2d8/rbd_data.e7592ae8944a.0000000000000833/head//3 on disk data >> digest 0x59a6e7e0 != 0x68778496 2015-08-07 13:26:56.495207 > > This indicates the stored crc doesn't match the observed crc, > and > >> 7f5903f7a700 -1 log_channel(cluster) log [ERR] : be_compare_scrubmaps: >> 3.d8 shard 1: soid >> cc49f2d8/rbd_data.3ef8442ae8944a.0000000000000aff/head//3 data_digest >> 0x4e20a792 != known data_digest 0xc0e9b2d2 from auth shard 7 > > this indicates two replicas do not match. > >> 2015-08-07 13:27:12.134765 7f5903f7a700 -1 log_channel(cluster) log >> [ERR] : deep-scrub 3.d8 >> d93e14d8/rbd_data.3ef8442ae8944a.0000000000000729/head//3 on disk data >> digest 0x3cdb1f5c != 0x4e0400c2 >> >> osd.7.log is clean for that period of time. >> /var/log/dmesg is also clean. > > This really shouldn't happen, but there has been one recently fixed bug > that could have corrupted a replica. Can you locate two mismatched copies > of some object on two different OSDs, and use ceph-post-file so that we > can take a look at the actual corruption? For this PG, for example, the > mismatched copies are on osd.1 and osd.7. On those hosts, you can find > the backing file with > > find /var/lib/ceph/osd/ceph-1/current/3.d8_head | grep rbd_data.3ef8442ae8944a.0000000000000aff > > Alternatively, if the data is sensitive, can you diff the hexdump -C > output of both files, see what the differing bytes look like, and describe > that to us? > > Thanks! > sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
