Re: Umem Questions

[Magnus Karlsson <magnus.karlsson@xxxxxxxxx>]

On Thu, May 20, 2021 at 2:40 PM Magnus Karlsson
<magnus.karlsson@xxxxxxxxx> wrote:
>
> On Thu, May 20, 2021 at 8:14 AM Magnus Karlsson
> <magnus.karlsson@xxxxxxxxx> wrote:
> >
> > On Wed, May 19, 2021 at 11:09 PM Dan Siemon <dan@xxxxxxxxxxxxx> wrote:
> > >
> > >
> > > > > 2) It looks like there is a limit of 2GB on the maximum Umem size?
> > > > > I've
> > > > > tried with and without huge pages. Is this fundamental? How hard
> > > > > would
> > > > > it be to increase this?
> > > >
> > > > This was news to me. Do you know where in the xdp_umem_reg code it
> > > > complains about this? I guess it is xsk_umem__create() that fails,
> > > > or?
> > > > The only limit I see from a basic inspection of the code is that the
> > > > number of packet buffers cannot be larger than a u32 (4G). But you
> > > > are
> > > > not close to that limit.
> > >
> > > Yes, the failure is in xsk_umem__create(). I don't know where
> > > specifically but there are a couple spots in kernel side of that which
> > > return ENOMEM which is the return value.
>
> I think I have found it.
>
> static int xdp_umem_pin_pages(struct xdp_umem *umem, unsigned long address)
> {
>   unsigned int gup_flags = FOLL_WRITE;
> long npgs;
>         int err;
>
>         umem->pgs = kcalloc(umem->npgs, sizeof(*umem->pgs),
>                             GFP_KERNEL | __GFP_NOWARN);
>         if (!umem->pgs)
>                 return -ENOMEM;
>
> This structure becomes too large to allocate with kcalloc(). It needs
> to be turned into a kvcalloc() so that it can use vmalloc instead for
> requests that are this large. Will spin a patch.
>
> Thanks: Magnus

Could you please check if this one fixes it for you?

diff --git a/net/xdp/xdp_umem.c b/net/xdp/xdp_umem.c
index 56a28a686988..f01ef6bda390 100644
--- a/net/xdp/xdp_umem.c
+++ b/net/xdp/xdp_umem.c
@@ -27,7 +27,7 @@ static void xdp_umem_unpin_pages(struct xdp_umem *umem)
 {
        unpin_user_pages_dirty_lock(umem->pgs, umem->npgs, true);

-       kfree(umem->pgs);
+       kvfree(umem->pgs);
        umem->pgs = NULL;
 }

@@ -99,8 +99,7 @@ static int xdp_umem_pin_pages(struct xdp_umem *umem,
unsigned long address)
        long npgs;
        int err;

-       umem->pgs = kcalloc(umem->npgs, sizeof(*umem->pgs),
-                           GFP_KERNEL | __GFP_NOWARN);
+       umem->pgs = kvcalloc(umem->npgs, sizeof(*umem->pgs),
GFP_KERNEL | __GFP_NOWARN);
        if (!umem->pgs)
                return -ENOMEM;

@@ -123,7 +122,7 @@ static int xdp_umem_pin_pages(struct xdp_umem
*umem, unsigned long address)
 out_pin:
        xdp_umem_unpin_pages(umem);
 out_pgs:
-       kfree(umem->pgs);
+       kvfree(umem->pgs);
        umem->pgs = NULL;
        return err;
 }


> > Can you issue a "ulimit -a" on your system and share the result? Just
> > to verify that there is no per process limit that kicks in.