|
I didn't see any guidance on how to resolve the check some error online. Any hints?
Rhian Resnick Assistant Director Middleware and HPC Office of Information Technology
Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222
From: Gregory Farnum <gfarnum@xxxxxxxxxx>
Sent: Monday, July 3, 2017 11:49 AM To: Rhian Resnick Cc: ceph-users Subject: Re: Inconsistent pgs with size_mismatch_oi On Mon, Jul 3, 2017 at 6:02 AM, Rhian Resnick <rresnick@xxxxxxx> wrote:
> > Sorry to bring up an old post but on Kraken I am unable to repair a PG that is inconsistent in a cache tier . We remove the bad object but am still seeing the following error in the OSD's logs. It's possible, but the digest error means they checksum differently, rather than having different sizes (and the size check precedes the digest one). The part where all three of them are exactly the same is interesting and actually makes me suspect that something just went wrong in calculating the checksum... > > > > Prior to removing invalid object: > > /var/log/ceph/ceph-osd.126.log:928:2017-07-03 08:07:55.331479 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f shard 63: soid 1:fa86fe35:::10006cdc2c5.00000000:head data_digest 0x931041e9 != data_digest 0xcd130b55 from auth oi 1:fa86fe35:::10006cdc2c5.00000000:head(25726'1664129 client.8168902.0:607753 dirty|data_digest s 1713351 uv 1664129 dd cd130b55 alloc_hint [0 0 0]) > /var/log/ceph/ceph-osd.126.log:929:2017-07-03 08:07:55.331483 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f shard 126: soid 1:fa86fe35:::10006cdc2c5.00000000:head data_digest 0x931041e9 != data_digest 0xcd130b55 from auth oi 1:fa86fe35:::10006cdc2c5.00000000:head(25726'1664129 client.8168902.0:607753 dirty|data_digest s 1713351 uv 1664129 dd cd130b55 alloc_hint [0 0 0]) > /var/log/ceph/ceph-osd.126.log:930:2017-07-03 08:07:55.331487 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f shard 143: soid 1:fa86fe35:::10006cdc2c5.00000000:head data_digest 0x931041e9 != data_digest 0xcd130b55 from auth oi 1:fa86fe35:::10006cdc2c5.00000000:head(25726'1664129 client.8168902.0:607753 dirty|data_digest s 1713351 uv 1664129 dd cd130b55 alloc_hint [0 0 0]) > /var/log/ceph/ceph-osd.126.log:931:2017-07-03 08:07:55.331491 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f soid 1:fa86fe35:::10006cdc2c5.00000000:head: failed to pick suitable auth object > /var/log/ceph/ceph-osd.126.log:932:2017-07-03 08:08:27.605139 7f95a4be6700 -1 log_channel(cluster) log [ERR] : 1.15f repair 3 errors, 0 fixed > > > > Post Removing invalid object: > /var/log/ceph/ceph-osd.126.log:3433:2017-07-03 08:37:03.780584 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f shard 63: soid 1:fa86fe35:::10006cdc2c5.00000000:head data_digest 0x931041e9 != data_digest 0xcd130b55 from auth oi 1:fa86fe35:::10006cdc2c5.00000000:head(25726'1664129 client.8168902.0:607753 dirty|data_digest s 1713351 uv 1664129 dd cd130b55 alloc_hint [0 0 0]) > /var/log/ceph/ceph-osd.126.log:3434:2017-07-03 08:37:03.780591 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f shard 126: soid 1:fa86fe35:::10006cdc2c5.00000000:head data_digest 0x931041e9 != data_digest 0xcd130b55 from auth oi 1:fa86fe35:::10006cdc2c5.00000000:head(25726'1664129 client.8168902.0:607753 dirty|data_digest s 1713351 uv 1664129 dd cd130b55 alloc_hint [0 0 0]) > /var/log/ceph/ceph-osd.126.log:3435:2017-07-03 08:37:03.780593 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f shard 143 missing 1:fa86fe35:::10006cdc2c5.00000000:head > /var/log/ceph/ceph-osd.126.log:3436:2017-07-03 08:37:03.780594 7f95a73eb700 -1 log_channel(cluster) log [ERR] : 1.15f soid 1:fa86fe35:::10006cdc2c5.00000000:head: failed to pick suitable auth object > /var/log/ceph/ceph-osd.126.log:3437:2017-07-03 08:37:39.278991 7f95a4be6700 -1 log_channel(cluster) log [ERR] : 1.15f repair 3 errors, 0 fixed > > > > Is it possible this thread is related to the error we are seeing? > > > Rhian Resnick > > Assistant Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > > > > > ________________________________ > From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Gregory Farnum <gfarnum@xxxxxxxxxx> > Sent: Monday, May 15, 2017 6:28 PM > To: Lincoln Bryant; Weil, Sage > Cc: ceph-users > Subject: Re: Inconsistent pgs with size_mismatch_oi > > > > On Mon, May 15, 2017 at 3:19 PM Lincoln Bryant <lincolnb@xxxxxxxxxxxx> wrote: >> >> Hi Greg, >> >> Curiously, some of these scrub errors went away on their own. The example pg in the original post is now active+clean, and nothing interesting in the logs: >> >> # zgrep "36.277b" ceph-osd.244*gz >> ceph-osd.244.log-20170510.gz:2017-05-09 06:56:40.739855 7f0184623700 0 log_channel(cluster) log [INF] : 36.277b scrub starts >> ceph-osd.244.log-20170510.gz:2017-05-09 06:58:01.872484 7f0186e28700 0 log_channel(cluster) log [INF] : 36.277b scrub ok >> ceph-osd.244.log-20170511.gz:2017-05-10 20:40:47.536974 7f0186e28700 0 log_channel(cluster) log [INF] : 36.277b scrub starts >> ceph-osd.244.log-20170511.gz:2017-05-10 20:41:38.399614 7f0184623700 0 log_channel(cluster) log [INF] : 36.277b scrub ok >> ceph-osd.244.log-20170514.gz:2017-05-13 20:49:47.063789 7f0186e28700 0 log_channel(cluster) log [INF] : 36.277b scrub starts >> ceph-osd.244.log-20170514.gz:2017-05-13 20:50:42.085718 7f0186e28700 0 log_channel(cluster) log [INF] : 36.277b scrub ok >> ceph-osd.244.log-20170515.gz:2017-05-15 00:10:39.417578 7f0184623700 0 log_channel(cluster) log [INF] : 36.277b scrub starts >> ceph-osd.244.log-20170515.gz:2017-05-15 00:11:26.189777 7f0186e28700 0 log_channel(cluster) log [INF] : 36.277b scrub ok >> >> (No matches in the logs for osd 175 and osd 297 — perhaps already rotated away?) >> >> Other PGs still exhibit this behavior though: >> >> # rados list-inconsistent-obj 36.2953 | jq . >> { >> "epoch": 737940, >> "inconsistents": [ >> { >> "object": { >> "name": "1002378da6c.00000001", >> "nspace": "", >> "locator": "", >> "snap": "head", >> "version": 2213621 >> }, >> "errors": [], >> "union_shard_errors": [ >> "size_mismatch_oi" >> ], >> "selected_object_info": "36:ca95a23b:::1002378da6c.00000001:head(737930'2177823 client.36346283.1:5635626 dirty s 4067328 uv 2213621)", >> "shards": [ >> { >> "osd": 113, >> "errors": [ >> "size_mismatch_oi" >> ], >> "size": 0 >> }, >> { >> "osd": 123, >> "errors": [ >> "size_mismatch_oi" >> ], >> "size": 0 >> }, >> { >> "osd": 173, >> "errors": [ >> "size_mismatch_oi" >> ], >> "size": 0 >> } >> ] >> } >> ] >> } >> >> Perhaps new data being written to this pg cleared things up? > > > Hmm, somebody else did report the same thing (and the symptoms disappearing) recently as well. I wonder if we broke the synchronization around eviction and scrubbing within cache pools. Sage, you've done work on cache pools recently; any thoughts? > -Greg > >> >> >> The only other data point that I can add is that, due to some tweaking of the cache tier size before this happened, the cache tier was reporting near full / full in `ceph -s` for a brief amount of time (maybe <1hr ?). >> >> Thanks for looking into this. >> >> --Lincoln >> >> > On May 15, 2017, at 4:50 PM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: >> > >> > On Mon, May 1, 2017 at 9:28 AM, Lincoln Bryant <lincolnb@xxxxxxxxxxxx> wrote: >> >> Hi all, >> >> >> >> I’ve run across a peculiar issue on 10.2.7. On my 3x replicated cache tiering cache pool, routine scrubbing suddenly found a bunch of PGs with size_mismatch_oi errors. From the “rados list-inconsistent-pg tool”[1], I see that all OSDs are reporting size 0 for a particular pg. I’ve checked this pg on disk, and it is indeed 0 bytes: >> >> -rw-r--r-- 1 root root 0 Apr 29 06:12 100235614fe.00000005__head_6E9A677B__24 >> >> >> >> I’ve tried re-issuing a scrub, which informs me that the object info size (2994176) doesn’t match the on-disk size (0) (see [2]). I’ve tried a repair operation as well to no avail. >> >> >> >> For what it’s worth, this particular cluster is currently migrating several disks from one CRUSH root to another, and there is a nightly cache flush/eviction script that is lowering the cache_target_*_ratios before raising them again in the morning. >> >> >> >> This issue is currently affecting ~10 PGs in my cache pool. Any ideas how to proceed here? >> > >> > Did anything come from this? It's tickling my brain (especially with >> > the cache pool) but I'm not seeing anything relevant when I search my >> > email. >> > >> >> >> >> Thanks, >> >> Lincoln >> >> >> >> [1]: >> >> { >> >> "epoch": 721312, >> >> "inconsistents": [ >> >> { >> >> "object": { >> >> "name": "100235614fe.00000005", >> >> "nspace": "", >> >> "locator": "", >> >> "snap": "head", >> >> "version": 2233551 >> >> }, >> >> "errors": [], >> >> "union_shard_errors": [ >> >> "size_mismatch_oi" >> >> ], >> >> "selected_object_info": "36:dee65976:::100235614fe.00000005:head(737928'2182216 client.36346283.1:5754260 dirty s 2994176 uv 2233551)", >> >> "shards": [ >> >> { >> >> "osd": 175, >> >> "errors": [ >> >> "size_mismatch_oi" >> >> ], >> >> "size": 0 >> >> }, >> >> { >> >> "osd": 244, >> >> "errors": [ >> >> "size_mismatch_oi" >> >> ], >> >> "size": 0 >> >> }, >> >> { >> >> "osd": 297, >> >> "errors": [ >> >> "size_mismatch_oi" >> >> ], >> >> "size": 0 >> >> } >> >> ] >> >> } >> >> ] >> >> } >> >> >> >> [2]: >> >> 2017-05-01 10:50:13.812992 7f0184623700 0 log_channel(cluster) log [INF] : 36.277b scrub starts >> >> 2017-05-01 10:51:02.495229 7f0186e28700 -1 log_channel(cluster) log [ERR] : 36.277b shard 175: soid 36:dee65976:::100235614fe.00000005:head size 0 != size 2994176 from auth oi 36:dee65976:::100235614fe.00000005:head(737928'2182216 client.36346283.1:5754260 dirty s 2994176 uv 2233551) >> >> 2017-05-01 10:51:02.495234 7f0186e28700 -1 log_channel(cluster) log [ERR] : 36.277b shard 244: soid 36:dee65976:::100235614fe.00000005:head size 0 != size 2994176 from auth oi 36:dee65976:::100235614fe.00000005:head(737928'2182216 client.36346283.1:5754260 dirty s 2994176 uv 2233551) >> >> 2017-05-01 10:51:02.495326 7f0186e28700 -1 log_channel(cluster) log [ERR] : 36.277b shard 297: soid 36:dee65976:::100235614fe.00000005:head size 0 != size 2994176 from auth oi 36:dee65976:::100235614fe.00000005:head(737928'2182216 client.36346283.1:5754260 dirty s 2994176 uv 2233551) >> >> 2017-05-01 10:51:02.495328 7f0186e28700 -1 log_channel(cluster) log [ERR] : 36.277b soid 36:dee65976:::100235614fe.00000005:head: failed to pick suitable auth object >> >> 2017-05-01 10:51:02.495450 7f0186e28700 -1 log_channel(cluster) log [ERR] : scrub 36.277b 36:dee65976:::100235614fe.00000005:head on disk size (0) does not match object info size (2994176) adjusted for ondisk to (2994176) >> >> 2017-05-01 10:51:20.223733 7f0184623700 -1 log_channel(cluster) log [ERR] : 36.277b scrub 4 errors >> >> >> >> _______________________________________________ >> >> ceph-users mailing list >> >> ceph-users@xxxxxxxxxxxxxx >> >> https://na01.safelinks.protection.outlook.com/?url=""> >> > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > https://na01.safelinks.protection.outlook.com/?url=""> > |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
