Thanks Dan and Anthony your suggestions have pointed me in the right
direction. Looking back through the logs at when the first error was
detected I found this:
ceph-osd: 2021-01-24 01:04:55.905 7f0c17821700 -1 log_channel(cluster)
log [ERR] : 17.7ffs0 scrub : stat mismatch, got 112867/112868 objects,
0/0 clones, 112867/112868 dirty, 0/0 omap, 0/0 pinned, 0/0
hit_set_archive, 0/0 whiteouts, 473372381184/473376575488 bytes, 0/0
manifest objects, 0/0 hit_set_archive bytes.
As Anthony suggested the error is not in the rados objects but
actually the stats.
I assume that a repair will fix this up?
Thanks again everyone.
Rich
On Thu, 28 Jan 2021 at 03:59, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote:
>
> Usually the ceph.log prints the reason for the inconsistency when it
> is first detected by scrubbing.
>
> -- dan
>
> On Wed, Jan 27, 2021 at 12:41 AM Richard Bade <hitrich@xxxxxxxxx> wrote:
> >
> > Hi Everyone,
> > I also have seen this inconsistent with empty when you do list-inconsistent-obj
> >
> > $ sudo ceph health detail
> > HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1
> > pgs not deep-scrubbed in time
> > OSD_SCRUB_ERRORS 1 scrub errors
> > PG_DAMAGED Possible data damage: 1 pg inconsistent
> > pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21]
> > PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time
> > pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811
> >
> > $ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty
> > {
> > "epoch": 183807,
> > "inconsistents": []
> > }
> >
> > Usually these are caused by read errors on the disks, but I've checked
> > all osd hosts that are part of this osd and there's no smart or dmesg
> > errors.
> >
> > Rich
> >
> > ------------------------------
> > >
> > > Date: Sun, 17 Jan 2021 14:00:01 +0330
> > > From: Seena Fallah <seenafallah@xxxxxxxxx>
> > > Subject: Re: PG inconsistent with empty inconsistent
> > > objects
> > > To: "Alexander E. Patrakov" <patrakov@xxxxxxxxx>
> > > Cc: ceph-users <ceph-users@xxxxxxx>
> > > Message-ID:
> > > <CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw@xxxxxxxxxxxxxx>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > It's for a long time ago and I don't have the `ceph health detail` output!
> > >
> > > On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov <patrakov@xxxxxxxxx>
> > > wrote:
> > >
> > > > For a start, please post the "ceph health detail" output.
> > > >
> > > > сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah@xxxxxxxxx>:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm facing something strange! One of the PGs in my pool got inconsistent
> > > > > and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty`
> > > > > the `inconsistents` key was empty! What is this? Is it a bug in Ceph
> > > > or..?
> > > > >
> > > > > Thanks.
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > > >
> > > >
> > > >
> > > > --
> > > > Alexander E. Patrakov
> > > > CV: http://u.pc.cd/wT8otalK
> > > >
> > >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
