Hello, We recently upgraded Ceph from version 12.2.2 to version 12.2.5. Since the upgrade we've been having performance issues which seem to relate to when deep-scrub actions are performed. Most of the time deep-scrub actions only takes a couple of seconds at most, however ocassionaly it takes 10 minutes. It's either a few seconds or 10 minutes, never a couple of minutes or anything else in between. This has been happening since the upgrade. For example see the following: 2018-06-14 10:11:46.337086 7fbd3528b700 0 log_channel(cluster) log [DBG] : 15.2dc deep-scrub starts 2018-06-14 10:11:50.947843 7fbd3528b700 0 log_channel(cluster) log [DBG] : 15.2dc deep-scrub ok 2018-06-14 10:45:49.575042 7fbd32a86700 0 log_channel(cluster) log [DBG] : 14.1 deep-scrub starts 2018-06-14 10:55:53.326309 7fbd32a86700 0 log_channel(cluster) log [DBG] : 14.1 deep-scrub ok 2018-06-14 10:58:28.652360 7fbd33a88700 0 log_channel(cluster) log [DBG] : 15.5f deep-scrub starts 2018-06-14 10:58:33.411769 7fbd2fa80700 0 log_channel(cluster) log [DBG] : 15.5f deep-scrub ok The scrub on PG 14.1 took pretty much exactly 10 minutes, the others only about 5 seconds. It matches the value of "osd scrub finalize thread timeout" which is currently set to 10 minutes, however I'm not sure if it's related or just a coincidence. It's not just this PG, but there's a bunch of them, also on difference nodes and OSD's. PG dump for this problematic PG is as follows: PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 14.1 10573 0 0 0 0 0 1579 1579 active+clean 2018-06-14 15:47:32.832222 1215'1291261 1215:7062174 [3,8,20] 3 [3,8,20] 3 1179'1288697 2018-06-14 10:55:53.326320 1179'1288697 2018-06-14 10:55:53.326320 0 During the longer running deep-scrub actions we're also running into performance problems. Any idea what's going wrong? Thanks Sander _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |