-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Since the one that is different is not your primary for the pg, then pg repair is safe. - ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Nov 25, 2015 at 2:42 AM, Major Csaba wrote: > Hi, > > I have some magic deep scrubbing error, and can't find the reason. > The system contains 5 nodes with 20 OSDs in total and everything works fine, > except these scrubbing errors. > > Sometimes the deep-scrub finds inconsistencies, but not exactly clear why. > The content of the objects are exactly the same, based on the md5sum. The > extended attributes are also matching, the only difference I can see is the > date when these files were accessed or modified by the system. The clocks > are in sync on all node. > > So, the question is how this checksum is calculated? What else should be > checked when the checksum differs? > As the data matches on all 3 OSDs, I can safely use the "pg repair" in this > case. Is it correct? > > The version of the ceph on all node are the same: > ~# ceph-osd -v > ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) > > For example, I can see this status: > pg 1.da is active+clean+inconsistent, acting [18,12,1] > > This is in the log of the OSD18: > 2015-11-25 08:00:11.486633 7f3f2f0b5700 0 log_channel(cluster) log [INF] : > 1.da deep-scrub starts > 2015-11-25 08:01:13.820282 7f3f2f0b5700 -1 log_channel(cluster) log [ERR] : > 1.da shard 12: soid 2d268cda/rb.0.532aa.238e1f29.00000002df78/head//1 > data_digest 0x9c482f6c != known data_digest 0x2a8b75d1 from auth shard 1 > 2015-11-25 08:01:26.718426 7f3f2f0b5700 -1 log_channel(cluster) log [ERR] : > 1.da deep-scrub 0 missing, 1 inconsistent objects > 2015-11-25 08:01:26.718435 7f3f2f0b5700 -1 log_channel(cluster) log [ERR] : > 1.da deep-scrub 1 errors > > The md5sum of the object looks identical on all OSDs: > > ~# md5sum > /var/lib/ceph/osd/ceph-18/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > 4b051db39517ff4c38049a5c2d50ce81 > /var/lib/ceph/osd/ceph-18/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > ~# md5sum > /var/lib/ceph/osd/ceph-12/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > 4b051db39517ff4c38049a5c2d50ce81 > /var/lib/ceph/osd/ceph-12/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > ~# md5sum > /var/lib/ceph/osd/ceph-1/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > 4b051db39517ff4c38049a5c2d50ce81 > /var/lib/ceph/osd/ceph-1/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > > The stat is a little bit different: > ~# stat > /var/lib/ceph/osd/ceph-18/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > File: > ‘/var/lib/ceph/osd/ceph-18/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1’ > Size: 4194304 Blocks: 8200 IO Block: 4096 regular file > Device: 831h/2097d Inode: 280718364 Links: 1 > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-11-22 19:35:39.508017680 +0100 > Modify: 2015-11-22 19:35:39.512017725 +0100 > Change: 2015-11-22 19:36:01.168257254 +0100 > Birth: - > > ~# stat > /var/lib/ceph/osd/ceph-12/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > File: > '/var/lib/ceph/osd/ceph-12/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1' > Size: 4194304 Blocks: 8200 IO Block: 4096 regular file > Device: 831h/2097d Inode: 405959653 Links: 1 > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-11-22 19:35:39.333851735 +0100 > Modify: 2015-11-22 19:35:39.333851735 +0100 > Change: 2015-11-22 19:36:01.157710026 +0100 > Birth: - > > ~# stat > /var/lib/ceph/osd/ceph-1/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1 > File: > '/var/lib/ceph/osd/ceph-1/current/1.da_head/DIR_A/DIR_D/DIR_C/rb.0.532aa.238e1f29.00000002df78__head_2D268CDA__1' > Size: 4194304 Blocks: 8200 IO Block: 4096 regular file > Device: 841h/2113d Inode: 408838486 Links: 1 > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-30 12:25:50.539795416 +0100 > Modify: 2015-11-20 13:30:06.026592488 +0100 > Change: 2015-11-22 00:39:15.778009055 +0100 > Birth: - > > Thanks in advance, > Csaba > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -----BEGIN PGP SIGNATURE----- Version: Mailvelope v1.2.3 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWVfKrCRDmVDuy+mK58QAA0EsQAIj4Ol01zFHXCkwiQUNJ 2ykTMU1/bulQM4qwNcp0K8gWYJdE41qzI69INJ+DlR2H6lIaHGZx2meKo5BR pV3DoBvIcakeQb89RNn4dVygYWKLvp0eyoRSoooLXAW3pRehLqB+d6dm9dyG rbK9BYh8+xW5l27VOhK7+fhA74NaLsUGCcV3x3bGos1QTSFLxnkT4TDc6gzs BzzCGG1/gQm9m3lKIvksNzpU5s7/XRGpSlh/urELjLqy/hYidBuKOOjFgNNB Evld/LixcQApLOL9q2UU+ky4W1hxTPyMDY0zJs0PaPdifHEK0VKTxB1ASfnz bMiBDdTcnooVSHynxlC7vjlQY9btegasP0WDzE3ib2tqPsmLhQ+kuGX7ne3x j3hM0AMwVSiacOEhx6pIerx4iLaKEU1Ldh/LtpnbumAV+Y4pUIuVrM011Yao K7nAJRiuARgAerlNnnW8Mxn/fJb9IzfdwdDWsY7ePpYlvq8KL7tYa0sw3SV7 OiJlA87xYiMr4tXale5U+GQdPLhJvUxzusofVc5Jdhmj0MxPT1gSazd0PoTV pTfrfdCNRa+yzd9wVxEWx0CrYnZqUmxkbdsqGUbN7OgMIsbBPV6QLOfKfI3j W0cwWw8wNU6OM8vzF6n2XBVxh6cNwehhxuzX2AFY+TSks6EoNHGtqCdv3+xi F/ST =X071 -----END PGP SIGNATURE----- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |