rocksdb corruption, stale pg, rebuild bucket index

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We ended in a bad situation with our RadosGW (Cluster is Nautilus 14.2.1, 350 OSDs with BlueStore):

1. There is a bucket with about 60 million objects, without shards.

2. radosgw-admin bucket reshard --bucket $BIG_BUCKET --num-shards 1024

3. Resharding looked fine first, it counted up to the number of objects, but then it hang.

4. 3 OSDs crashed with a segfault: "rocksdb: Corruption: file is too short"

5. Trying to start the OSDs manually led to the same segfaults.

6. ceph-bluestore-tool repair ...

7. The repairs all aborted, with the same rocksdb error as above.

8. Now 1 PG is stale. It belongs to the radosgw bucket index pool, and it contained the index of this big bucket.

Is there any hope in getting these rocksdbs up again?

Otherwise: how would we fix the bucket index pool? Our ideas:

1. ceph pg $BAD_PG mark_unfound_lost delete
2. rados -p .rgw.buckets ls, search $BAD_BUCKET_ID and remove these objects. The hope of this step would be to make the following step faster, and avoid another similar problem.
3. radosgw-admin bucket check --check-objects

Will this really rebuild the bucket index? Is it ok to leave the existing bucket indexes in place? Is it ok to run for all buckets at once, or has it to be run bucket by bucket? Is there a risk that the indexes that are not affected by the BAD_PG will be broken afterwards?

Some more details that may be of interest.

ceph-bluestore-repair says:

2019-06-12 11:15:38.345 7f56269670c0 -1 rocksdb: Corruption: file is too short (6139497190 bytes) to be an sstabledb/079728.sst 2019-06-12 11:15:38.345 7f56269670c0 -1 bluestore(/var/lib/ceph/osd/ceph-49) _open_db erroring opening db:
error from fsck: (5) Input/output error

The repairs also showed several warnings like:

tcmalloc: large alloc 17162051584 bytes == 0x56167918a000 @ 0x7f5626521887 0x56126a287229 0x56126a2873a3 0x56126a5dc1ec 0x56126a584ce2 0x56126a586a05 0x56126a587dd0 0x56126a589344 0x56126a38c3cf 0x56126a2eae94 0x56126a30654e 0x56126a337ae1 0x56126a1a73a1 0x7f561b228b97 0x56126a28077a

The processes showed up with like 45 GB of RAM used. Fortunately, there was no Out-Of-Memory.

 Harry
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux