I have no experience with perf and the package is not installed. I will take a look at it, thanks. Jan > On 11 Jun 2015, at 13:48, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > > Hi Jan, > > Can you get perf top running? It should show you where the OSDs are spinning... > > Cheers, Dan > > On Thu, Jun 11, 2015 at 11:21 AM, Jan Schermer <jan@xxxxxxxxxxx> wrote: >> Hi, >> hoping someone can point me in the right direction. >> >> Some of my OSDs have a larger CPU usage (and ops latencies) than others. If I restart the OSD everything runs nicely for some time, then it creeps up. >> >> 1) most of my OSDs have ~40% CPU (core) usage (user+sys), some are closer to 80%. Restarting means the offending OSDs only use 40% again. >> 2) average latencies and CPU usage on the host are the same - so it’s not caused by the host that the OSD is running on >> 3) I can’t say exactly when or how the issue happens. I can’t even say if it’s the same OSDs. It seems it either happens when something heavy happens in a cluster (like dropping very old snapshots, rebalancing) and then doesn’t come back, or maybe it happens slowly over time and I can’t find it in the graphs. Looking at the graphs it seems to be the former. >> >> I have just one suspicion and that is the “fd cache size” - we have it set to 16384 but the open fds suggest there are more open files for the osd process (over 17K fds) - it varies by some hundreds between the osds. Maybe some are just slightly over the limit and the misses cause this? Restarting the OSD clears them (~2K) and they increase over time. I increased it to 32768 yesterday and it consistently nice now, but it might take another few days to manifest… >> Could this explain it? Any other tips? >> >> Thanks >> >> Jan >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |