Re: avg apply latency went up after update from octopus to pacific

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

we have a very similar situation. We updated from nautilus -> pacific (16.2.11) and saw a rapid increase in the commit_latency and op_w_latency (>10s on some OSDs) after a few hours. We also have nearly exclusive rbd workload.

After deleting old snapshots we saw an improvenent, and after recreating snapshots the numbers went up again. Without snapshots the numbers are slowly getting higher but not as fast as before with existing snapshots. We also use SAS connected NVMe-SSDs. 
bluefs_buffered_io made no difference. We compacted the rocksdb on a single OSD yesterday, and funnily enough this is now the OSD with the highest op_w_latency. I generated a perf graph for this single OSD and can generate more, but I'm not sure how to share this data with you...?

I saw in the thread that Boris redeployed all OSDs. Could that be a more permanent solution or is this also just temporarily (like deleting the snapshots)?

Greetings,
Jan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux