On Thu, Mar 21, 2024 at 2:28 AM Chengming Zhou <chengming.zhou@xxxxxxxxx> wrote:
>
> On 2024/3/21 14:36, Zhongkun He wrote:
> > On Thu, Mar 21, 2024 at 1:24 PM Chengming Zhou <chengming.zhou@xxxxxxxxx> wrote:
> >>
> >> On 2024/3/21 13:09, Zhongkun He wrote:
> >>> On Thu, Mar 21, 2024 at 12:42 PM Chengming Zhou
> >>> <chengming.zhou@xxxxxxxxx> wrote:
> >>>>
> >>>> On 2024/3/21 12:34, Zhongkun He wrote:
> >>>>> Hey folks,
> >>>>>
> >>>>> Recently, I tested the zswap with memory reclaiming in the mainline
> >>>>> (6.8) and found a memory corruption issue related to exclusive loads.
> >>>>
> >>>> Is this fix included? 13ddaf26be32 ("mm/swap: fix race when skipping swapcache")
> >>>> This fix avoids concurrent swapin using the same swap entry.
> >>>>
> >>>
> >>> Yes, This fix avoids concurrent swapin from different cpu, but the
> >>> reported issue occurs
> >>> on the same cpu.
> >>
> >> I think you may misunderstand the race description in this fix changelog,
> >> the CPU0 and CPU1 just mean two concurrent threads, not real two CPUs.
> >>
> >> Could you verify if the problem still exists with this fix?
> >
> > Yes，I'm sure the problem still exists with this patch.
> > There is some debug info, not mainline.
> >
> > bpftrace -e'k:swap_readpage {printf("%lld, %lld,%ld,%ld,%ld\n%s",
> > ((struct page *)arg0)->private,nsecs,tid,pid,cpu,kstack)}' --include
> > linux/mm_types.h
>
> Ok, this problem seems only happen on SWP_SYNCHRONOUS_IO swap backends,
> which now include zram, ramdisk, pmem, nvdimm.
>
> It maybe not good to use zswap on these swap backends?
My gut reaction is to say yes, but I'll refrain from making sweeping
statements about backends I'm not too familiar with. Let's see:
1. zram: I don't even know why we're putting a compressed cache... in
front of a compressed faux swap device? Ramdisk == other in-memory
swap backend right?
2. I looked it up, and it seemed SWP_SYNCHRONOUS_IO was introduced for
fast swap storage (see the original patch series [1]). If this is the
case, one could argue there are diminishing returns for applying zswap
on top of this.
[1]: https://lore.kernel.org/linux-mm/1505886205-9671-1-git-send-email-minchan@xxxxxxxxxx/
>
> The problem here is the page fault handler tries to skip swapcache to
> swapin the folio (swap entry count == 1), but then it can't install folio
> to pte entry since some changes happened such as concurrent fork of entry.
>
> Maybe we should writeback that folio in this special case.
But yes, if this is simple maybe we can do this first to fix the bug?
>
> >
> > offset nsecs tid pid cpu
> > 2140659, 595771411052,15045,15045,6
> > swap_readpage+1
> > do_swap_page+2135
> > handle_mm_fault+2426
> > do_user_addr_fault+462
> > do_page_fault+48
> > async_page_fault+62
> >
> > offset nsecs tid pid cpu
> > 2140659, 595771424445,15045,15045,6
> > swap_readpage+1
> > do_swap_page+2135
> > handle_mm_fault+2426
> > do_user_addr_fault+462
> > do_page_fault+48
> > async_page_fault+62
> >
> > -------------------------------
> > There are two page faults with the same tid and offset in 13393 nsecs.
> >
> >>
> >>>
> >>> Thanks.
> >>>
> >>>> Thanks.
> >>>>
> >>>>>
> >>>>>
> >>>>> root@**:/sys/fs/cgroup/zz# stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep
> >>>>> stress: info: [31753] dispatching hogs: 0 cpu, 0 io, 5 vm, 0 hdd
> >>>>> stress: FAIL: [31758] (522) memory corruption at: 0x7f347ed1a010
> >>>>> stress: FAIL: [31753] (394) <-- worker 31758 returned error 1
> >>>>> stress: WARN: [31753] (396) now reaping child worker processes
> >>>>> stress: FAIL: [31753] (451) failed run completed in 14s
> >>>>>
> >>>>>
> >>>>> 1. Test step(the frequency of memory reclaiming has been accelerated):
> >>>>> -------------------------
> >>>>> a. set up the zswap, zram and cgroup V2
> >>>>> b. echo 0 > /sys/kernel/mm/lru_gen/enabled
> >>>>> (Increase the probability of problems occurring)
> >>>>> c. mkdir /sys/fs/cgroup/zz
> >>>>> echo $$ > /sys/fs/cgroup/zz/cgroup.procs
> >>>>> cd /sys/fs/cgroup/zz/
> >>>>> stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep
> >>>>>
> >>>>> e. in other shell:
> >>>>> while :;do for i in {1..5};do echo 20g >
> >>>>> /sys/fs/cgroup/zz/memory.reclaim & done;sleep 1;done
> >>>>>
> >>>>> 2. Root cause:
> >>>>> --------------------------
> >>>>> With a small probability, the page fault will occur twice with the
> >>>>> original pte, even if a new pte has been successfully set.
> >>>>> Unfortunately, zswap_entry has been released during the first page fault
> >>>>> with exclusive loads, so zswap_load will fail, and there is no corresponding
> >>>>> data in swap space, memory corruption occurs.
> >>>>>
> >>>>> bpftrace -e'k:zswap_load {printf("%lld, %lld\n", ((struct page
> >>>>> *)arg0)->private,nsecs)}'
> >>>>> --include linux/mm_types.h > a.txt
> >>>>>
> >>>>> look up the same index:
> >>>>>
> >>>>> index nsecs
> >>>>> 1318876, 8976040736819
> >>>>> 1318876, 8976040746078
> >>>>>
> >>>>> 4123110, 8976234682970
> >>>>> 4123110, 8976234689736
> >>>>>
> >>>>> 2268896, 8976660124792
> >>>>> 2268896, 8976660130607
> >>>>>
> >>>>> 4634105, 8976662117938
> >>>>> 4634105, 8976662127596
> >>>>>
> >>>>> 3. Solution
> >>>>>
> >>>>> Should we free zswap_entry in batches so that zswap_entry will be
> >>>>> valid when the next page fault occurs with the
> >>>>> original pte? It would be great if there are other better solutions.
> >>>>>
> >>>>
