Hello Eugen,
We have tried to export and import but the object is still unfound. After we identified the impacted RBD we had to delete the unfound object with "ceph pg 16.1e mark_unfound_lost delete" . The PG is now "active+clean" and the cluster HEALTH_OK. Now we are definitely consider to upgrade to pool size 3.
Thanks for the suggestion,Martin
On Wednesday, July 27, 2022 at 05:32:37 PM GMT+3, Eugen Block <eblock@xxxxxx> wrote:
Hi,
this seems to be another example why a pool size = 2 is a bad idea.
This has been discussed so many times...
> - If we stop osd.131 the PG becomes inactive and down (like it is
> the only osd containing the objects): Reduced data availability: 1
> pg inactive, 1 pg down
Because it is, ceph can't find the other replica.
> - Used ceph-objectstore-tool to search for the unfound object
> (rbd_data.ad5ab66b8b4567.0000000000011055) on all osd's involved,
> the object is present only on osd.41 and osd.131 even the PG is
> mapped to other OSD's.
Can you export it and import on a different OSD [1] [2]? I haven't
tried that yet but maybe this will work.
[1] https://docs.ceph.com/en/pacific/man/8/ceph-objectstore-tool/
[2]
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7AWMDL5CWKW2WBHM7TVIRLXYJSNS5EIX/
Zitat von Martin Culcea <martin_culcea@xxxxxxxxx>:
> Hello,
>
> After a host reboot the cluster could not find an object. The
> cluster was in stable state with all osd active+clean, no OSD was
> out, no other OSD was restarted during host reboot. It was 1 month
> ago, we hoped that the cluster will find the object eventually, but
> it did not.
> Cluster version: ceph version 16.2.9, ceph-deploy cluster, pool size 2.
>
> Attached are ceph.log, osds logs, pg query and other logs
>
> Cluster status:
> cluster:
> id: 2517da9e-af62-405e-b71f-1f2e145822f7
> health: HEALTH_ERR
> client is using insecure global_id reclaim
> mons are allowing insecure global_id reclaim
> 1/606943089 objects unfound (0.000%)
> Possible data damage: 1 pg recovery_unfound
> Degraded data redundancy: 7252/1219946300 objects degraded
> (0.001%), 1 pg degraded, 1 pg undersized
> 1 pgs not deep-scrubbed in time
> 1 pgs not scrubbed in time
>
> data:
> volumes: 1/1 healthy
> pools: 12 pools, 6560 pgs
> objects: 606.94M objects, 85 TiB
> usage: 169 TiB used, 268 TiB / 438 TiB avail
> pgs: 7252/1219946300 objects degraded (0.001%)
> 7250/1219946300 objects misplaced (0.001%)
> 1/606943089 objects unfound (0.000%)
> 6554 active+clean
> 4 active+clean+scrubbing+deep
> 1 active+recovery_unfound+undersized+degraded+remapped
> 1 active+clean+scrubbing
>
> io:
> client: 1.2 GiB/s rd, 1.4 GiB/s wr, 40.87k op/s rd, 72.80k op/s wr
>
> progress:
> Global Recovery Event (2w)
> [===========================.] (remaining: 4m)
>
>
>
> Ceph health detail
>
> HEALTH_ERR clients are using insecure global_id reclaim; mons are
> allowing insecure global_id reclaim; 1/606997573 objects unfound
> (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data
> redundancy: 7294/1220048932 objects degraded (0.001%), 1 pg
> degraded, 1 pg undersized; 1 pgs not deep-scrubbed in time; 1 pgs
> not scrubbed in time
> ...
> [WRN] OBJECT_UNFOUND: 1/606997573 objects unfound (0.000%)
> pg 16.1e has 1 unfound objects
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
> pg 16.1e is active+recovery_unfound+undersized+degraded+remapped,
> acting [131], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 7294/1220048932 objects
> degraded (0.001%), 1 pg degraded, 1 pg undersized
> pg 16.1e is stuck undersized for 3h, current state
> active+recovery_unfound+undersized+degraded+remapped, last acting
> [131]
> [WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
> pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
> [WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
> pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
>
> The PG is acting only on osd.131, even we move the PG to other OSD:
> ceph pg map 16.1e
> osdmap e723093 pg 16.1e (16.1e) -> up [41,141] acting [131]
>
> On ceph osd dump the pg is mapped as a pg_temp:
>
> ceph osd dump | grep -w 16.1e
> pg_temp 16.1e [131]
>
> What we did:
> - restarted all osd and hosts involved
> - force a deep-scrub on PG (the pg cannot be scrubed anymore)
> - If we stop osd.131 the PG becomes inactive and down (like it is
> the only osd containing the objects): Reduced data availability: 1
> pg inactive, 1 pg down
> - If we take out the osd.131, the pg is not moving to the new osd,
> it remains the only object on osd.131
> - ceph force recovery
> - ceph force repeer
> - ceph pg repair 16.1e
> - Used ceph-objectstore-tool to search for the unfound object
> (rbd_data.ad5ab66b8b4567.0000000000011055) on all osd's involved,
> the object is present only on osd.41 and osd.131 even the PG is
> mapped to other OSD's.
> - ceph-objectstore-tool
> ceph pg remap: we tryed to remap the pg to others OSD's (ceph osd
> pg-upmap-items 16.1e 131 141) but the PG does not move to new OSD's,
> remain on osd.41 and osd.131 (ceph pg map 16.1e: osdmap e723093 pg
> 16.1e (16.1e) -> up [41,141] acting [131])
>
> Why is this happening ?
> How can we help the cluster to find the lost object?
> Can we remove pg_temp 16.1e [131] from upmap (ceph osd dump) ?
>
> Thank you,
> Martin Culcea
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
