Interesting development work here, if anyone is interested. -JM ----- Forwarded Message ----- From: "Brian Foster (Code Review)" <root@xxxxxxxxxxxxxxx> Sent: Friday, February 15, 2013 9:21:41 AM Subject: Change in glusterfs[master]: performance/readdir-ahead: translator prototype Brian Foster has uploaded a new change for review. Change subject: performance/readdir-ahead: translator prototype ...................................................................... performance/readdir-ahead: translator prototype This is a prototype translator to explore the potential performance benefits of directory prefetching on sequential directory reads. It is incomplete and only tested with a focus on performance of the specific use case of single-threaded, sequential directory reads. General thoughts on approach/design are appreciated. While directory read performance could still be considered slow with this functionality, preliminary results show a measurable improvement: Notes: - The test directory structures are one or two levels deep. 'ls -R' is used to ensure lookups are generated during the traversal. - The biggest issue in performance of gluster readdirp tests is the ineffectiveness of md-cache during the initial traversal. Inodes are not linked until the completion of the first pass, causing lookups to dominate the test and increase the time to complete by orders of magnitude. This is a separate issue with fixes in progress. Thus, all tests that follow are run after an initial 'ls -R' to ensure inodes are linked into the inode table and md-cache is working. Test Description: - 2-brick DHT volume, all translators other than md-cache disabled. - gid-timeout=2 on client mount. - Command: time ls -R /mnt/<largedir> | wc -l Results: --- 100k files (1 dir) - baseline - ~13s - readdir-ahead - ~6s --- 200k files (2 dirs, 100k per dir) - baseline - ~28s - readdir-ahead - ~12s --- 1m files (10 dirs, 100k per dir) - baseline - 2m32s - readdir-ahead - 1m13s Change-Id: Ieceb9e1eb47d1d5b5af8da2bf03839537364653f Signed-off-by: Brian Foster <bfoster@xxxxxxxxxx> --- M configure.ac M xlators/performance/Makefile.am A xlators/performance/readdir-ahead/Makefile.am A xlators/performance/readdir-ahead/src/Makefile.am A xlators/performance/readdir-ahead/src/readdir-ahead-mem-types.h A xlators/performance/readdir-ahead/src/readdir-ahead.c A xlators/performance/readdir-ahead/src/readdir-ahead.h 7 files changed, 552 insertions(+), 1 deletion(-) git pull ssh://git.gluster.org/glusterfs refs/changes/19/4519/1 -- To view, visit http://review.gluster.org/4519 To unsubscribe, visit http://review.gluster.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ieceb9e1eb47d1d5b5af8da2bf03839537364653f Gerrit-PatchSet: 1 Gerrit-Project: glusterfs Gerrit-Branch: master Gerrit-Owner: Brian Foster <bfoster@xxxxxxxxxx>
