Re: Ceph objects unfound

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

this seems to be another example why a pool size = 2 is a bad idea. This has been discussed so many times...

- If we stop osd.131 the PG becomes inactive and down (like it is the only osd containing the objects): Reduced data availability: 1 pg inactive, 1 pg down

Because it is, ceph can't find the other replica.

- Used ceph-objectstore-tool to search for the unfound object (rbd_data.ad5ab66b8b4567.0000000000011055) on all osd's involved, the object is present only on osd.41 and osd.131 even the PG is mapped to other OSD's.

Can you export it and import on a different OSD [1] [2]? I haven't tried that yet but maybe this will work.

[1] https://docs.ceph.com/en/pacific/man/8/ceph-objectstore-tool/
[2] https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7AWMDL5CWKW2WBHM7TVIRLXYJSNS5EIX/

Zitat von Martin Culcea <martin_culcea@xxxxxxxxx>:

Hello,

After a host reboot the cluster could not find an object. The cluster was in stable state with all osd active+clean, no OSD was out, no other OSD was restarted during host reboot. It was 1 month ago, we hoped that the cluster will find the object eventually, but it did not.
Cluster version: ceph version 16.2.9, ceph-deploy cluster, pool size 2.

Attached are ceph.log, osds logs, pg query and other logs

Cluster status:
 cluster:
 id: 2517da9e-af62-405e-b71f-1f2e145822f7
 health: HEALTH_ERR
 client is using insecure global_id reclaim
 mons are allowing insecure global_id reclaim
 1/606943089 objects unfound (0.000%)
 Possible data damage: 1 pg recovery_unfound
Degraded data redundancy: 7252/1219946300 objects degraded (0.001%), 1 pg degraded, 1 pg undersized
 1 pgs not deep-scrubbed in time
 1 pgs not scrubbed in time

data:
 volumes: 1/1 healthy
 pools: 12 pools, 6560 pgs
 objects: 606.94M objects, 85 TiB
 usage: 169 TiB used, 268 TiB / 438 TiB avail
 pgs: 7252/1219946300 objects degraded (0.001%)
 7250/1219946300 objects misplaced (0.001%)
 1/606943089 objects unfound (0.000%)
 6554 active+clean
 4 active+clean+scrubbing+deep
 1 active+recovery_unfound+undersized+degraded+remapped
 1 active+clean+scrubbing

io:
 client: 1.2 GiB/s rd, 1.4 GiB/s wr, 40.87k op/s rd, 72.80k op/s wr

progress:
 Global Recovery Event (2w)
 [===========================.] (remaining: 4m)



Ceph health detail

HEALTH_ERR clients are using insecure global_id reclaim; mons are allowing insecure global_id reclaim; 1/606997573 objects unfound (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 7294/1220048932 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 1 pgs not deep-scrubbed in time; 1 pgs not scrubbed in time
...
[WRN] OBJECT_UNFOUND: 1/606997573 objects unfound (0.000%)
 pg 16.1e has 1 unfound objects
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound [WRN] PG_DEGRADED: Degraded data redundancy: 7294/1220048932 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 16.1e is stuck undersized for 3h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
 pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
[WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
 pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300

The PG is acting only on osd.131, even we move the PG to other OSD:
ceph pg map 16.1e
osdmap e723093 pg 16.1e (16.1e) -> up [41,141] acting [131]

On ceph osd dump the pg is mapped as a pg_temp:

ceph osd dump | grep -w 16.1e
pg_temp 16.1e [131]

What we did:
- restarted all osd and hosts involved
- force a deep-scrub on PG (the pg cannot be scrubed anymore)
- If we stop osd.131 the PG becomes inactive and down (like it is the only osd containing the objects): Reduced data availability: 1 pg inactive, 1 pg down - If we take out the osd.131, the pg is not moving to the new osd, it remains the only object on osd.131
- ceph force recovery
- ceph force repeer
- ceph pg repair 16.1e
- Used ceph-objectstore-tool to search for the unfound object (rbd_data.ad5ab66b8b4567.0000000000011055) on all osd's involved, the object is present only on osd.41 and osd.131 even the PG is mapped to other OSD's.
- ceph-objectstore-tool
ceph pg remap: we tryed to remap the pg to others OSD's (ceph osd pg-upmap-items 16.1e 131 141) but the PG does not move to new OSD's, remain on osd.41 and osd.131 (ceph pg map 16.1e: osdmap e723093 pg 16.1e (16.1e) -> up [41,141] acting [131])

Why is this happening ?
How can we help the cluster to find the lost object?
Can we remove pg_temp 16.1e [131] from upmap (ceph osd dump) ?

Thank you,
Martin Culcea



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux