【nautilus cluster one osd start up failed when osd superblock crc fail】

[nank <knan2012@xxxxxxx>]

【version】

ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)

【trigger operation 】

Under normal operation of the cluster, power down the equipment manually, and then power up the equipment again.

【appearance】

One OSD failed to start up in the cluster.

The logging shows that the OSD Superblock CRC checksum is inconsistent with the actual data on disk.

Try to turn validation off. Dump OSD superblock decodes normally and the fields are correct. 

Analyzing the refresh OSD Superblock process confirms that the refresh operation is a defere write transaction.

【dump osd super block no crc】

[root@node145 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-23 --type bluestore --op dump-super
{
    "cluster_fsid": "a28d823e-eb42-4eba-9603-e1722e8bc884",
    "osd_fsid": "53b6d708-2ac2-4323-bedc-7082e08c5791",
    "whoami": 23,
    "current_epoch": 6112,
    "oldest_map": 5249,
    "newest_map": 6112,
    "weight": 0,
    "compat": {
        "compat": {},
        "ro_compat": {},
        "incompat": {
            "feature_1": "initial feature set(~v.18)",
            "feature_2": "pginfo object",
            "feature_3": "object locator",
            "feature_4": "last_epoch_clean",
            "feature_5": "categories",
            "feature_6": "hobjectpool",
            "feature_7": "biginfo",
            "feature_8": "leveldbinfo",
            "feature_9": "leveldblog",
            "feature_10": "snapmapper",
            "feature_11": "sharded objects",
            "feature_12": "transaction hints",
            "feature_13": "pg meta object",
            "feature_14": "explicit missing set",
            "feature_15": "fastinfo pg attr",
            "feature_16": "deletes in missing set"
        }
    },
    "clean_thru": 6112,
    "last_epoch_mounted": 5884
}

【log】   

2020-10-09 15:00:22.296 7f637345fdc0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xce54de81, expected 0xebc2f895, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# [BlueStore::_verify_csum:9368]
2020-10-09 15:00:22.297 7f637345fdc0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xce54de81, expected 0xebc2f895, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# [BlueStore::_verify_csum:9368]
2020-10-09 15:00:22.297 7f637345fdc0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xce54de81, expected 0xebc2f895, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# [BlueStore::_verify_csum:9368]
2020-10-09 15:00:22.297 7f637345fdc0 -1 osd.15 0 OSD::init() : unable to read osd superblock [OSD::init:3115]

_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx