On Wed, Dec 20, 2017 at 12:15 PM, Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: > On Wed, Dec 20, 2017 at 11:37 AM, Tejun Heo <tj@xxxxxxxxxx> wrote: >> Hello, Shakeel. >> >> On Tue, Dec 19, 2017 at 02:39:19PM -0800, Shakeel Butt wrote: >>> Suppose a user wants to run multiple instances of a specific job on >>> different datacenters and s/he has budget of 100MiB for each instance. >>> The instances are schduled on the requested datacenters and the >>> scheduler has set the memory limit of those instances to 100MiB. Now, >>> some datacenters have swap deployed, so, there, let's say, the swap >>> limit of those instances are set according to swap medium >>> availability. In this setting the user will see inconsistent memcg OOM >>> behavior. Some of the instances see OOMs at 100MiB usage (suppose only >>> anon memory) while some will see OOMs way above 100MiB due to swap. >>> So, the user is required to know the internal knowledge of datacenters >>> (like which has swap or not and swap type) and has to set the limits >>> accordingly and thus increase the chance of config bugs. >> >> I don't understand how this invariant is useful across different >> backing swap devices and availability. e.g. Our OOM decisions are >> currently not great in that the kernel can easily thrash for a very >> long time without making actual progresses. If you combine that with >> widely varying types and availability of swaps, > > The kernel never swaps out on hitting memsw limit. So, the varying > types and availability of swaps becomes invariant to the memcg OOM > behavior of the job. > >> whether something is >> OOMing or not doesn't really tell you much. The workload could be >> running completely fine or have been thrashing without making any >> meaningful forward progress for the past 15 mins. >> >> Given that whether or not swap exists, how much is avialable and how >> fast the backing swap device is all highly influential parameters in >> how the workload behaves, I don't see what having sum of memory + swap >> as an invariant actually buys. And, even that essentially meaningless >> invariant doesn't really exist - the performance of the swap device >> absolutely affects when the OOM killer would kick in. >> > > No, as I previously explained, the swap types and availability will be > transparent to the memcg OOM killer and memcg memory reclaim behavior. > >> So, I don't see how the sum of memory+swap makes it possible to ignore >> the swap type and availability. Can you please explain that further? >> >>> Also different types and sizes of swap mediums in data center will >>> further complicates the configuration. One datacenter might have SSD >>> as a swap, another might be doing swap on zram and third might be >>> doing swap on nvdimm. Each can have different size and can be assigned >>> to jobs differently. So, it is possible that the instances of the same >>> job might be assigned different swap limit on different datacenters. >> >> Sure, but what does memswap achieve? >> > > 1. memswap provides consistent memcg OOM killer and memcg memory > reclaim behavior independent to swap. > 2. With memswap, the job owners do not have to think or worry about swaps. When I say OOM and memory reclaim behavior, I specifically mean memcg oom-kill and memcg memory reclaim behavior. These are different from global oom-killer and global memory reclaim behaviors. The global behaviors will be affected by the types and availability of swaps and the jobs can suffer differently based on swap types and availability on hitting global OOM scenario. -- To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
