Re: GlusterFS FUSE client leaks summary — part I

[Soumya Koduri <skoduri@xxxxxxxxxx>]

On 01/31/2016 03:05 PM, Oleksandr Natalenko wrote:
> Unfortunately, this patch doesn't help.
>
> RAM usage on "find" finish is ~9G.
>
> Here is statedump before drop_caches: https://gist.github.com/
> fc1647de0982ab447e20

[mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
size=706766688
num_allocs=2454051

> And after drop_caches: https://gist.github.com/5eab63bc13f78787ed19

[mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
size=550996416
num_allocs=1913182

There isn't much significant drop in inode contexts. One of the reasons 
could be because of dentrys holding a refcount on the inodes which shall 
result in inodes not getting purged even after fuse_forget.


pool-name=fuse:dentry_t
hot-count=32761

if  '32761' is the current active dentry count, it still doesn't seem to 
match up to inode count.

Thanks,
Soumya
> And here is Valgrind output: https://gist.github.com/2490aeac448320d98596
>
> On субота, 30 січня 2016 р. 22:56:37 EET Xavier Hernandez wrote:
> > There's another inode leak caused by an incorrect counting of
> > lookups on directory reads.
> >
> > Here's a patch that solves the problem for
> > 3.7:
> >
> > http://review.gluster.org/13324
> >
> > Hopefully with this patch the
> > memory leaks should disapear.
> >
> > Xavi
> >
> > On 29.01.2016 19:09, Oleksandr
> >
> > Natalenko wrote:
> > > Here is intermediate summary of current memory
> >
> > leaks in FUSE client
> >
> > > investigation.
> > >
> > > I use GlusterFS v3.7.6
> >
> > release with the following patches:
> > > ===
> >
> > > Kaleb S KEITHLEY (1):
> > fuse: use-after-free fix in fuse-bridge, revisited
> >
> > > Pranith Kumar K
> >
> > (1):
> > > mount/fuse: Fix use-after-free crash
> >
> > > Soumya Koduri (3):
> > gfapi: Fix inode nlookup counts
> >
> > > inode: Retire the inodes from the lru
> >
> > list in inode_table_destroy
> >
> > > upcall: free the xdr* allocations
> > > ===
> > >
> > >
> > > With those patches we got API leaks fixed (I hope, brief tests show
> >
> > that) and
> >
> > > got rid of "kernel notifier loop terminated" message.
> >
> > Nevertheless, FUSE
> >
> > > client still leaks.
> > >
> > > I have several test
> >
> > volumes with several million of small files (100K…2M in
> >
> > > average). I
> >
> > do 2 types of FUSE client testing:
> > > 1) find /mnt/volume -type d
> > > 2)
> >
> > rsync -av -H /mnt/source_volume/* /mnt/target_volume/
> >
> > > And most
> >
> > up-to-date results are shown below:
> > > === find /mnt/volume -type d
> >
> > ===
> >
> > > Memory consumption: ~4G
> >
> > > Statedump:
> > https://gist.github.com/10cde83c63f1b4f1dd7a
> >
> > > Valgrind:
> > https://gist.github.com/097afb01ebb2c5e9e78d
> >
> > > I guess,
> >
> > fuse-bridge/fuse-resolve. related.
> >
> > > === rsync -av -H
> >
> > /mnt/source_volume/* /mnt/target_volume/ ===
> >
> > > Memory consumption:
> > ~3.3...4G
> >
> > > Statedump (target volume):
> > https://gist.github.com/31e43110eaa4da663435
> >
> > > Valgrind (target volume):
> > https://gist.github.com/f8e0151a6878cacc9b1a
> >
> > > I guess,
> >
> > DHT-related.
> >
> > > Give me more patches to test :).
> >
> > _______________________________________________
> >
> > > Gluster-devel mailing
> >
> > list
> >
> > > Gluster-devel@xxxxxxxxxxx
> >
> > http://www.gluster.org/mailman/listinfo/gluster-devel
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel