Hi Sam, What happens if you just disable swap altogether? i.e., with `swapoff -a` --Lincoln On Tue, 2018-01-23 at 19:54 +0000, Samuel Taylor Liston wrote: > We have a 9 - node (16 - 8TB OSDs per node) running jewel on centos > 7.4. The OSDs are configured with encryption. The cluster is > accessed via two - RGWs and there are 3 - mon servers. The data > pool is using 6+3 erasure coding. > > About 2 weeks ago I found two of the nine servers wedged and had to > hard power cycle them to get them back. In this hard reboot 22 - > OSDs came back with either a corrupted encryption or data > partitions. These OSDs were removed and recreated, and the resultant > rebalance moved along just fine for about a week. At the end of that > week two different nodes were unresponsive complaining of page > allocation failures. This is when I realized the nodes were heavy > into swap. These nodes were configured with 64GB of RAM as a cost > saving going against the 1GB per 1TB recommendation. We have since > then doubled the RAM in each of the nodes giving each of them more > than the 1GB per 1TB ratio. > > The issue I am running into is that these nodes are still swapping; a > lot, and over time becoming unresponsive, or throwing page allocation > failures. As an example, “free” will show 15GB of RAM usage (out of > 128GB) and 32GB of swap. I have configured swappiness to 0 and and > also turned up the vm.min_free_kbytes to 4GB to try to keep the > kernel happy, and yet I am still filling up swap. It only occurs > when the OSDs have mounted partitions and ceph-osd daemons active. > > Anyone have an idea where this swap usage might be coming from? > Thanks for any insight, > > Sam Liston (sam.liston@xxxxxxxx) > ==================================== > Center for High Performance Computing > 155 S. 1452 E. Rm 405 > Salt Lake City, Utah 84112 (801)232-6932 > ==================================== > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |