Re: Readdirp (ls -l) Performance Improvement

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Felix,

Thanks for your mail. I will test it more to make sure that it doesn't break anything. Also I have added a configuration key for easier switching to the older code in case if there is any problem. If you can help me in any manner in testing or performance numbers, please let me know.


Regards

Rafi KC

On 27/05/20 12:37 pm, Felix Kölzow wrote:

Dear Rafi,


thanks for your effort. I think this is of great interest of many gluster users. Thus, I would really encourage you to

test and to further improve this feature. Maybe it is beneficial to create a certain guideline which things should be tested

to make this feature really ready for productive use.


Thanks in advance.

Felix

On 27/05/2020 07:56, RAFI KC wrote:

Hi All,

I have been working on POC to improve readdirp performance improvement. At the end of the experiment, The results are showing promising result in performance, overall there is a 104% improvement for full filesystem crawl compared to the existing solution. Here is the short test numbers. The tests were carried out in 16*3 setup with 1.5 Million dentries (Both files and dir). The system also contains some empty directories. In the result the proposed solution is 287% faster than the plane volume and 104% faster than the parallel-readdir based solution.


Configuration

Plain volume

Parallel-readdir

Proposed Solution

FS Crawl Time in Seconds

16497.523

8717.872

4261.401

In short, the basic idea behind the proposal is the efficient managing of readdir buffer in gluster along with prefetching the dentries for intelligent switch-over to the next buffer. The detailed problem description, deign description and results are available in the doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit


If anybody can help with the testing on a different kind of workloads, I would be very happy to assist. If wanted to test the patch and run a performance test on your setup, I could help with back-porting the patch to the version of your choice.


https://review.gluster.org/24469

https://review.gluster.org/24470


Regards

Rafi KC


________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux