High read IO on RocksDB/WAL since upgrade to Octopus

[Glen Baars <glen@xxxxxxxxxxxxxxxxxxxxxx>]

Hello Ceph Users,

Since upgrading from Nautilus to Octopus ( cluster started in luminous ) I have been trying to debug why the RocksDB/WAL is maxing out the SSD drives. ( QD > 32, 12000 read IOPS, 200 write IOPS ).

The omap upgrade on migration was disabled initially but I reenabled it and restarted all OSD's. This completed without issue.

I have increased the memory target from 4 to 6GB per OSD but it doesn't look like it is using it all anyway ( based on top ).

I have offline compacted all OSDs. This seems to help for about 4-6 hours ( backfilling is occuring - maybe this triggers it? ).

RGW garbage collection is upto date.

Pg_log on some PG's are high due to them not being in a clean state ( 8% PGs > 3000 ) remainder of PG's I have reduced to 500 logs - no change.

I've been working on this issue for days not without much luck. Nothing in the logs indicates a major issue.

The client impact is a major reduction in speed.

{
    "mon": {
        "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 5
    },
    "mgr": {
        "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 1
    },
    "osd": {
        "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 18,
        "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 280
    },
    "mds": {},
    "rgw": {
        "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 2
    },
    "tcmu-runner": {
        "ceph version 14.2.13-450-g65ea1b614d (65ea1b614db8b6d10f334a8ff67c4de97f73bcbf) nautilus (stable)": 2
    },
    "overall": {
        "ceph version 14.2.13-450-g65ea1b614d (65ea1b614db8b6d10f334a8ff67c4de97f73bcbf) nautilus (stable)": 2,
        "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 18,
        "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 288
    }
}

Any assistance in debugging would be greatly helpful.

Glen
This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx