Hello, On Wed, 27 Jul 2016 10:21:34 +0200 Kenneth Waegeman wrote: > Hi all, > > When our OSD hosts are running for some time, we start see increased > usage of swap on a number of them. Some OSDs don't use swap for weeks, > while others has a full (4G) swap, and start filling swap again after we > did a swapoff/swapon. Obvious first question would be, are all these hosts really the same, HW, SW and configuration wise? > We have 8 8TB OSDS and 2 cache SSDs on each hosts, and 80GB of Memory. How full are these OSDs? I'm interested in # of files, not space, so a "df -i" should give us some idea. 80GB is an odd number, how are the DIMMs distributed among the CPU(s)? > There is still about 15-20GB memory available when this happens. Running > Centos7; How do you define free memory? Not used at all? I'd expect any Ceph storage server to use all "free" RAM for SLAB and pagecache very quickly, at the latest after the first deep scrub. If it is really unused AND your system is swapping, something odd is going on indeed, maybe something NUMA related that prevents part of your memory from being used. Of course this could also be an issue with your CentOS kernel, I'm definitely not seeing anything like this on any of my machines. > We had swapiness set to 0. I wouldn't set it lower than 1. Also any other tuning settings, like vm/vfs_cache_pressure and vm/min_free_kbytes? >There is no client io right now, > only scrubbing. some OSDs are using 20-80% of cpu. > Sounds high for pure CPU usage, unless that includes IOWAIT. Christian > Has somebody seen this behaviour? It doesn't have to be bad, but what > could explain some hosts keep on swapping, and others don't? > Could this be some issue? > > Thanks !! > > Kenneth > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |