Hello everyone! I've noticed something strange since updating our cluster from Nautilus to Pacific 16.2.7. Out of 40 OSDs, one was created with Pacific 16.2.7 and all others in the cluster were created with Nautilus or Mimic (the daemons are all running Pacific). Every few days, the OSD created with Pacific will suddenly start writing to its device constantly. As I type this, it is writing 250-350 MiB/s to the drive (according to iotop). All other OSDs are writing about 15-30 MiB/s to their devices. Read activity is normal - all OSDs are reading 50-100 MiB/s. There isn't nearly enough client activity to justify this activity, the cluster is healthy, nothing is rebalancing or scrubbing. Using "ceph osd status" shows the OSD has about the same number of reads and writes as all the others. I tried using "ceph tell osd.X config set" to increase every debug_* option to its maximum setting, but nothing seemed to stand out. There was some additional output (not much) and it was mostly "bluestore.MempoolThread(x) _resize_shards", "prioritycache tune_memory", "heartbeat osd_stat" and "ms_handle_reset con". What else can I do to troubleshoot this? Is this a bug? Restarting the OSD daemon "fixes" it for a few days, then it always seems to start happening again. I'm planning to recreate all of the OSDs in this cluster this weekend (to split each NVMe drive into multiple OSDs), so I'm concerned about every OSD showing this behavior next week. Should I postpone this weekend's work? I haven't restarted the OSD daemon yet this morning, so I can still try some additional debugging while the writing is going on. -- Sam Clippinger ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient(s) and contain information that may be Garmin confidential and/or Garmin legally privileged. If you have received this email in error, please notify the sender by reply email and delete the message. Any disclosure, copying, distribution or use of this communication (including attachments) by someone other than the intended recipient is prohibited. Thank you. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |