On Mon, Jul 16, 2018 at 11:09 AM, Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: > On Sun, Jul 15, 2018 at 6:50 PM Yafang Shao <laoar.shao@xxxxxxxxx> wrote: >> >> On Sun, Jul 15, 2018 at 11:04 PM, Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: >> > On Sun, Jul 15, 2018 at 1:02 AM Yafang Shao <laoar.shao@xxxxxxxxx> wrote: >> >> >> >> On Sun, Jul 15, 2018 at 2:34 PM, Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: >> >> > On Sat, Jul 14, 2018 at 10:26 PM Yafang Shao <laoar.shao@xxxxxxxxx> wrote: >> >> >> >> >> >> On Sun, Jul 15, 2018 at 12:25 PM, Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: >> >> >> > On Sat, Jul 14, 2018 at 7:10 PM Yafang Shao <laoar.shao@xxxxxxxxx> wrote: >> >> >> >> >> >> >> >> On Sat, Jul 14, 2018 at 11:38 PM, Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: >> >> >> >> > On Sat, Jul 14, 2018 at 1:32 AM Yafang Shao <laoar.shao@xxxxxxxxx> wrote: >> >> >> >> >> >> >> >> >> >> try_charge maybe executed in packet receive path, which is in interrupt >> >> >> >> >> context. >> >> >> >> >> In this situation, the 'current' is the interrupted task, which may has >> >> >> >> >> no relation to the rx softirq, So it is nonsense to use 'current'. >> >> >> >> >> >> >> >> >> > >> >> >> >> > Have you actually seen this occurring? >> >> >> >> >> >> >> >> Hi Shakeel, >> >> >> >> >> >> >> >> I'm trying to produce this issue, but haven't find it occur yet. >> >> >> >> >> >> >> >> > I am not very familiar with the >> >> >> >> > network code but I can think of two ways try_charge() can be called >> >> >> >> > from network code. Either through kmem charging or through >> >> >> >> > mem_cgroup_charge_skmem() and both locations correctly handle >> >> >> >> > interrupt context. >> >> >> >> > >> >> >> >> >> >> >> >> Why do you say that mem_cgroup_charge_skmem() correctly hanle >> >> >> >> interrupt context ? >> >> >> >> >> >> >> >> Let me show you why mem_cgroup_charge_skmem isn't hanling interrupt >> >> >> >> context correctly. >> >> >> >> >> >> >> >> mem_cgroup_charge_skmem() is calling try_charge() twice. >> >> >> >> The first one is with GFP_NOWAIT as the gfp_mask, and the second one >> >> >> >> is with (GFP_NOWAIT | __GFP_NOFAIL) as the gfp_mask. >> >> >> >> >> >> >> >> If page_counter_try_charge() failes at the first time, -ENOMEM is returned. >> >> >> >> Then mem_cgroup_charge_skmem() will call try_charge() once more with >> >> >> >> __GFP_NOFAIL set, and this time if If page_counter_try_charge() failes >> >> >> >> again the ' >> >> >> >> force' label in try_charge() will be executed and 0 is returned. >> >> >> >> >> >> >> >> No matter what, the 'current' will be used and touched, that is >> >> >> >> meaning mem_cgroup_charge_skmem() isn't hanling the interrupt context >> >> >> >> correctly. >> >> >> >> >> >> >> > >> >> >> > Hi Yafang, >> >> >> > >> >> >> > If you check mem_cgroup_charge_skmem(), the memcg passed is not >> >> >> > 'current' but is from the sock object i.e. sk->sk_memcg for which the >> >> >> > network buffer is allocated for. >> >> >> > >> >> >> >> >> >> That's correct, the memcg if from the sock object. >> >> >> But the point is, in this situation why 'current' is used in try_charge() ? >> >> >> As 'current' is not related with the memcg, which is just a interrupted task. >> >> >> >> >> > >> >> > Hmm so you mean the behavior of memcg charging in the interrupt >> >> > context depends on the state of the interrupted task. >> >> >> >> Yes. >> >> >> >> > As you have >> >> > noted, mem_cgroup_charge_skmem() tries charging again with >> >> > __GFP_NOFAIL and the charge succeeds. Basically the memcg charging by >> >> > mem_cgroup_charge_skmem() will always succeed irrespective of the >> >> > state of the interrupted task. However mem_cgroup_charge_skmem() can >> >> > return true if the interrupted task was exiting or a fatal signal is >> >> > pending or oom victim or reclaiming memory. Can you please explain why >> >> > this is bad? >> >> > >> >> >> >> Let me show you the possible issues cause by this behavoir. >> >> 1. In mem_cgroup_oom(), some members in 'current' is set. >> >> That means an innocent task will be in task_in_memcg_oom state. >> >> But this task may be in a different memcg, I mean the memcg of >> >> the 'current' may be differenct with the sk->sk_memcg. >> >> Then when this innocent 'current' do try_charge it will hit "if >> >> (unlikely(task_in_memcg_oom(current)))" and -ENOMEM is returned, >> >> While there're maybe some free memory (or some memory could be freed ) >> >> in the memcg of the innocent 'task'. >> >> >> > >> > No memory will be freed as try_charge() is in interrupt context. >> > >> >> I mean when this interrupted 'current' is running, that's in process context. >> In process context it should call try_to_free_mem_cgroup_pages() to >> free some memory, >> but it will hit "if (unlikely(task_in_memcg_oom(current)))" before as >> it is set in the interrupt context. >> >> That's an obviously issue. Do you understand ? >> > > Not really. I couldn't find where current->memcg_in_oom can be set in > the interrupt context. > You are right. current->memcg_in_oom can't be set in the interrupt context. >> >> 2. If the interrupted task was exiting or a fatal signal is pending >> >> or oom victim, >> >> it will directly goto force and 0 is returned, and then >> >> mem_cgroup_charge_skmem() will return true. >> >> But mem_cgroup_charge_skmem() maybe need to try the second time >> >> and return false. >> >> >> >> That are all unexpected behavoir. >> >> >> > >> > Yes, this is inconsistent behavior. Can you explain how this will >> > affect network traffic? Basically mem_cgroup_charge_skmem() was >> > supposed to return false but sometime based on the interrupted task, >> > mem_cgroup_charge_skmem() returns true. How is this behavior bad for >> > network traffic? >> > >> >> You could see the funtion __sk_mem_raise_allocated(). >> If mem_cgroup_charge_skmem() return false, it will goto >> suppress_allocation and uncharge skmem, >> while when mem_cgroup_charge_skmem() return true, it will charge skmem >> sucessfully. >> >> The consequence behavior is sk_rmem_schedule may fail while it should sucess. >> And then it will call tcp_prune_queue() and tcp collapse may take a long time. >> > > Is that a good thing or bad? > From what I understand with your change > if charge fails, sk_rmem_schedule will always fail. However without > your change the interrupted task's state might help sk_rmem_schedule > to pass. I am all for consistent behavior but I wanted to make sure if > that is what you are aiming for. > Yes, with this change it will always fail. Without this change it may sucess depends on the interrupted task's state. My previous statement makes some mistake. I have no clear idea it is bad or good. That's why I'm trying to produce the issue now. But I think that we should avoid this unexpected behavior due to state of the random interrupted task. > Anyways, from what I remember Facebook is using the cgroup-v2's tcpmem > accounting. Johannes or Roman can shed some light if they have > observed this issue in production and might have opinion on how to > solve it. > > thanks, > Shakeel
