Thanks for the report Artem,
Looks like the issue is about cache warming up. Specially, I suspect rsync doing a 'readdir(), stat(), file operations' loop, where as when a find or ls is issued, we get 'readdirp()' request, which contains the stat information along with entries, which also makes sure cache is up-to-date (at md-cache layer).
Note that this is just a off-the memory hypothesis, We surely need to analyse and debug more thoroughly for a proper explanation. Some one in my team would look at it soon.
Regards,
Amar
On Mon, Feb 5, 2018 at 7:25 AM, Vlad Kopylov <vladkopy@xxxxxxxxx> wrote:
You mounting it to the local bricks?
struggling with same performance issues
try using this volume setting
http://lists.gluster.org/pipermail/gluster-users/2018- January/033397.html
performance.stat-prefetch: on might be it
seems like when it gets to cache it is fast - those stat fetch which
seem to come from .gluster are slow
> ______________________________
On Sun, Feb 4, 2018 at 3:45 AM, Artem Russakovskii <archon810@xxxxxxxxx> wrote:
> An update, and a very interesting one!
>
> After I started stracing rsync, all I could see was lstat calls, quite slow
> ones, over and over, which is expected.
>
> For example: lstat("uploads/2016/10/nexus2cee_DSC05339_thumb- 161x107.jpg",
> {st_mode=S_IFREG|0664, st_size=4043, ...}) = 0
>
> I googled around and found
> https://gist.github.com/nh2/1836415489e2132cf85ed3832105fc , which isc1
> seeing this exact issue with gluster, rsync and xfs.
>
> Here's the craziest finding so far. If while rsync is running (or right
> before), I run /bin/ls or find on the same gluster dirs, it immediately
> speeds up rsync by a factor of 100 or maybe even 1000. It's absolutely
> insane.
>
> I'm stracing the rsync run, and the slow lstat calls flood in at an
> incredible speed as soon as ls or find run. Several hundred of files per
> minute (excruciatingly slow) becomes thousands or even tens of thousands of
> files a second.
>
> What do you make of this?
>
>
_________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
Amar Tumballi (amarts)
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
