Hello
I need hints about where to look at for poor readdir performance
on NetBSD.
The volume is 4x2 distributed/replicated (gluster volume info at the end of
this message). Sometimes, the first listing of a large directroy is oddly
long. For instance, ls on a directoty containing 96 entries took 39 seconds
to complete. A second listing gets a decent time of 0.7s.
profiling data included below shows a large time is spent in READDIRP
for brick baril:/export/wd0e.
Kernel trace shows that on the slow brick, all items in the listed
directory get the following treatment:
stat()
lgetxattr(trusted.glusterfs.mdata) => does not exist
llistxattr()
That does not happen on other bricks, which just call get directory
content, without further inspection.
Any hint? I suspect it considers it has to self-heal something, but
how is this decided?
# gluster volume info
Volume Name: gfs
Type: Distributed-Replicate
Volume ID: 45f4dc1b-344f-41a7-be82-c02e57a4b555
Status: Started
Snapshot Count: 0
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: bidon:/export/wd0e_tmp
Brick2: baril:/export/wd0e
Brick3: bidon:/export/WD-WX11D741AXCA-v2
Brick4: baril:/export/gfs1a
Brick5: bidon:/export/wd2e
Brick6: baril:/export/wd2e
Brick7: bidon:/export/wd3e_tmp
Brick8: baril:/export/wd3e_tmp
Options Reconfigured:
performance.cache-size: 128MB
performance.cache-max-file-size: 0
storage.gfid2path: off
cluster.self-heal-daemon: enable
nfs.disable: true
diagnostics.brick-log-level: INFO
performance.parallel-readdir: on
cluster.readdir-optimize: on
# gluster volume profile gfs info
Brick: bidon:/export/wd3e_tmp
-----------------------------
Cumulative Stats:
Block Size: 512b+ 2048b+ 8192b+
No. of Reads: 1 0 16
No. of Writes: 0 3 1
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 3 18 0
No. of Writes: 1 14 2
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 43
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 148 FORGET
0.00 0.00 us 0.00 us 0.00 us 148 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66884 RELEASEDIR
1.59 52.41 us 52.41 us 52.41 us 1 GETXATTR
3.96 65.33 us 41.03 us 89.64 us 2 STATFS
4.55 74.97 us 45.48 us 104.45 us 2 OPENDIR
10.16 335.15 us 335.15 us 335.15 us 1 READDIRP
79.74 219.11 us 95.56 us 427.04 us 12 LOOKUP
Duration: 138813 seconds
Data Read: 795226 bytes
Data Written: 6319409 bytes
Interval 0 Stats:
Block Size: 512b+ 2048b+ 8192b+
No. of Reads: 1 0 16
No. of Writes: 0 3 1
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 3 18 0
No. of Writes: 1 14 2
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 43
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 148 FORGET
0.00 0.00 us 0.00 us 0.00 us 148 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66884 RELEASEDIR
1.59 52.41 us 52.41 us 52.41 us 1 GETXATTR
3.96 65.33 us 41.03 us 89.64 us 2 STATFS
4.55 74.97 us 45.48 us 104.45 us 2 OPENDIR
10.16 335.15 us 335.15 us 335.15 us 1 READDIRP
79.74 219.11 us 95.56 us 427.04 us 12 LOOKUP
Duration: 138813 seconds
Data Read: 795226 bytes
Data Written: 6319409 bytes
Brick: bidon:/export/wd2e
-------------------------
Cumulative Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 5 1 6
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 134 FORGET
0.00 0.00 us 0.00 us 0.00 us 135 RELEASE
0.00 0.00 us 0.00 us 0.00 us 74499 RELEASEDIR
0.35 71.17 us 71.17 us 71.17 us 1 GETXATTR
0.58 58.37 us 53.90 us 62.84 us 2 OPENDIR
0.60 60.38 us 41.31 us 79.44 us 2 STATFS
1.89 379.35 us 379.35 us 379.35 us 1 READDIRP
96.58 1763.96 us 141.55 us 16854.47 us 11 LOOKUP
Duration: 138814 seconds
Data Read: 262144 bytes
Data Written: 0 bytes
Interval 0 Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 5 1 6
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 134 FORGET
0.00 0.00 us 0.00 us 0.00 us 135 RELEASE
0.00 0.00 us 0.00 us 0.00 us 74499 RELEASEDIR
0.35 71.17 us 71.17 us 71.17 us 1 GETXATTR
0.58 58.37 us 53.90 us 62.84 us 2 OPENDIR
0.60 60.38 us 41.31 us 79.44 us 2 STATFS
1.89 379.35 us 379.35 us 379.35 us 1 READDIRP
96.58 1763.96 us 141.55 us 16854.47 us 11 LOOKUP
Duration: 138814 seconds
Data Read: 262144 bytes
Data Written: 0 bytes
Brick: bidon:/export/wd0e_tmp
-----------------------------
Cumulative Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 5 1 6
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 FORGET
0.00 0.00 us 0.00 us 0.00 us 124 RELEASE
0.00 0.00 us 0.00 us 0.00 us 17347 RELEASEDIR
2.82 99.39 us 99.39 us 99.39 us 1 GETXATTR
4.29 75.51 us 73.75 us 77.28 us 2 OPENDIR
5.59 98.52 us 43.81 us 153.22 us 2 STATFS
10.54 371.27 us 371.27 us 371.27 us 1 READDIRP
76.77 225.44 us 122.50 us 684.71 us 12 LOOKUP
Duration: 138816 seconds
Data Read: 262144 bytes
Data Written: 0 bytes
Interval 0 Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 5 1 6
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 FORGET
0.00 0.00 us 0.00 us 0.00 us 124 RELEASE
0.00 0.00 us 0.00 us 0.00 us 17347 RELEASEDIR
2.82 99.39 us 99.39 us 99.39 us 1 GETXATTR
4.29 75.51 us 73.75 us 77.28 us 2 OPENDIR
5.59 98.52 us 43.81 us 153.22 us 2 STATFS
10.54 371.27 us 371.27 us 371.27 us 1 READDIRP
76.77 225.44 us 122.50 us 684.71 us 12 LOOKUP
Duration: 138816 seconds
Data Read: 262144 bytes
Data Written: 0 bytes
Brick: bidon:/export/WD-WX11D741AXCA-v2
---------------------------------------
Cumulative Stats:
Block Size: 256b+ 2048b+ 8192b+
No. of Reads: 1 4 1504
No. of Writes: 0 197941 2627877
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 15 6705 0
No. of Writes: 86163 2 2
Block Size: 131072b+
No. of Reads: 64860
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 182 FORGET
0.00 0.00 us 0.00 us 0.00 us 219 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66888 RELEASEDIR
0.04 58.40 us 51.64 us 65.16 us 2 OPENDIR
0.08 246.24 us 246.24 us 246.24 us 1 GETXATTR
0.32 979.61 us 979.61 us 979.61 us 1 UNLINK
1.13 251.23 us 123.88 us 485.58 us 14 LOOKUP
2.46 1906.00 us 15.69 us 7573.23 us 4 ENTRYLK
6.20 9608.96 us 41.02 us 19176.90 us 2 STATFS
16.05 49754.55 us 49754.55 us 49754.55 us 1 CREATE
73.72 228492.97 us 228492.97 us 228492.97 us 1 READDIRP
Duration: 138815 seconds
Data Read: 8733744996 bytes
Data Written: 28674746937 bytes
Interval 0 Stats:
Block Size: 256b+ 2048b+ 8192b+
No. of Reads: 1 4 1504
No. of Writes: 0 197941 2627877
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 15 6705 0
No. of Writes: 86163 2 2
Block Size: 131072b+
No. of Reads: 64860
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 182 FORGET
0.00 0.00 us 0.00 us 0.00 us 219 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66888 RELEASEDIR
0.04 58.40 us 51.64 us 65.16 us 2 OPENDIR
0.08 246.24 us 246.24 us 246.24 us 1 GETXATTR
0.32 979.61 us 979.61 us 979.61 us 1 UNLINK
1.13 251.23 us 123.88 us 485.58 us 14 LOOKUP
2.46 1906.00 us 15.69 us 7573.23 us 4 ENTRYLK
6.20 9608.96 us 41.02 us 19176.90 us 2 STATFS
16.05 49754.55 us 49754.55 us 49754.55 us 1 CREATE
73.72 228492.97 us 228492.97 us 228492.97 us 1 READDIRP
Duration: 138815 seconds
Data Read: 8733744996 bytes
Data Written: 28674746937 bytes
Brick: baril:/export/wd3e_tmp
-----------------------------
Cumulative Stats:
Block Size: 2048b+ 8192b+ 16384b+
No. of Reads: 0 10 2
No. of Writes: 3 1 1
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 20 0 0
No. of Writes: 14 2 43
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 148 FORGET
0.00 0.00 us 0.00 us 0.00 us 148 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66884 RELEASEDIR
0.03 46.99 us 46.99 us 46.99 us 1 GETXATTR
0.04 37.97 us 37.83 us 38.12 us 2 STATFS
0.05 45.43 us 44.62 us 46.24 us 2 OPENDIR
32.77 57479.55 us 57479.55 us 57479.55 us 1 READDIRP
67.11 9810.81 us 122.08 us 38620.88 us 12 LOOKUP
Duration: 138821 seconds
Data Read: 786432 bytes
Data Written: 6319409 bytes
Interval 0 Stats:
Block Size: 2048b+ 8192b+ 16384b+
No. of Reads: 0 10 2
No. of Writes: 3 1 1
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 20 0 0
No. of Writes: 14 2 43
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 148 FORGET
0.00 0.00 us 0.00 us 0.00 us 148 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66884 RELEASEDIR
0.03 46.99 us 46.99 us 46.99 us 1 GETXATTR
0.04 37.97 us 37.83 us 38.12 us 2 STATFS
0.05 45.43 us 44.62 us 46.24 us 2 OPENDIR
32.77 57479.55 us 57479.55 us 57479.55 us 1 READDIRP
67.11 9810.81 us 122.08 us 38620.88 us 12 LOOKUP
Duration: 138821 seconds
Data Read: 786432 bytes
Data Written: 6319409 bytes
Brick: baril:/export/gfs1a
--------------------------
Cumulative Stats:
Block Size: 2048b+ 8192b+ 16384b+
No. of Reads: 0 20 7
No. of Writes: 197941 2627878 86163
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 30 0 0
No. of Writes: 2 2 64860
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 182 FORGET
0.00 0.00 us 0.00 us 0.00 us 219 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66885 RELEASEDIR
0.06 50.84 us 50.84 us 50.84 us 1 GETXATTR
0.09 18.55 us 16.41 us 20.18 us 4 ENTRYLK
0.11 48.97 us 44.84 us 53.10 us 2 OPENDIR
0.12 51.16 us 37.81 us 64.51 us 2 STATFS
1.78 1518.16 us 1518.16 us 1518.16 us 1 UNLINK
3.50 213.42 us 118.62 us 342.69 us 14 LOOKUP
46.17 39364.15 us 39364.15 us 39364.15 us 1 CREATE
48.16 41060.82 us 41060.82 us 41060.82 us 1 READDIRP
Duration: 138823 seconds
Data Read: 1326977 bytes
Data Written: 37176085049 bytes
Interval 0 Stats:
Block Size: 2048b+ 8192b+ 16384b+
No. of Reads: 0 20 7
No. of Writes: 197941 2627878 86163
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 30 0 0
No. of Writes: 2 2 64860
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 182 FORGET
0.00 0.00 us 0.00 us 0.00 us 219 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66885 RELEASEDIR
0.06 50.84 us 50.84 us 50.84 us 1 GETXATTR
0.09 18.55 us 16.41 us 20.18 us 4 ENTRYLK
0.11 48.97 us 44.84 us 53.10 us 2 OPENDIR
0.12 51.16 us 37.81 us 64.51 us 2 STATFS
1.78 1518.16 us 1518.16 us 1518.16 us 1 UNLINK
3.50 213.42 us 118.62 us 342.69 us 14 LOOKUP
46.17 39364.15 us 39364.15 us 39364.15 us 1 CREATE
48.16 41060.82 us 41060.82 us 41060.82 us 1 READDIRP
Duration: 138823 seconds
Data Read: 1326977 bytes
Data Written: 37176085049 bytes
Brick: baril:/export/wd0e
-------------------------
Cumulative Stats:
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 9 5 2
No. of Writes: 0 0 0
Block Size: 32768b+
No. of Reads: 4463
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 127 FORGET
0.00 0.00 us 0.00 us 0.00 us 127 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66931 RELEASEDIR
0.00 61.70 us 61.70 us 61.70 us 1 GETXATTR
0.00 44.23 us 40.46 us 47.99 us 2 STATFS
0.00 66.94 us 53.19 us 80.69 us 2 OPENDIR
0.01 214.48 us 132.89 us 350.46 us 12 LOOKUP
99.99 19333949.34 us 808361.68 us 37859537.00 us 2 READDIRP
Duration: 138824 seconds
Data Read: 146374656 bytes
Data Written: 0 bytes
Interval 0 Stats:
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 9 5 2
No. of Writes: 0 0 0
Block Size: 32768b+
No. of Reads: 4463
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 127 FORGET
0.00 0.00 us 0.00 us 0.00 us 127 RELEASE
0.00 0.00 us 0.00 us 0.00 us 66931 RELEASEDIR
0.00 61.70 us 61.70 us 61.70 us 1 GETXATTR
0.00 44.23 us 40.46 us 47.99 us 2 STATFS
0.00 66.94 us 53.19 us 80.69 us 2 OPENDIR
0.01 214.48 us 132.89 us 350.46 us 12 LOOKUP
99.99 19333949.34 us 808361.68 us 37859537.00 us 2 READDIRP
Duration: 138824 seconds
Data Read: 146374656 bytes
Data Written: 0 bytes
Brick: baril:/export/wd2e
-------------------------
Cumulative Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 10 2 20
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 134 FORGET
0.00 0.00 us 0.00 us 0.00 us 135 RELEASE
0.00 0.00 us 0.00 us 0.00 us 67991 RELEASEDIR
0.07 52.79 us 52.79 us 52.79 us 1 GETXATTR
0.10 39.25 us 38.38 us 40.13 us 2 STATFS
0.13 51.45 us 45.36 us 57.55 us 2 OPENDIR
20.28 1406.18 us 133.88 us 13190.43 us 11 LOOKUP
79.41 60571.65 us 60571.65 us 60571.65 us 1 READDIRP
Duration: 138822 seconds
Data Read: 786432 bytes
Data Written: 0 bytes
Interval 0 Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 10 2 20
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 134 FORGET
0.00 0.00 us 0.00 us 0.00 us 135 RELEASE
0.00 0.00 us 0.00 us 0.00 us 67991 RELEASEDIR
0.07 52.79 us 52.79 us 52.79 us 1 GETXATTR
0.10 39.25 us 38.38 us 40.13 us 2 STATFS
0.13 51.45 us 45.36 us 57.55 us 2 OPENDIR
20.28 1406.18 us 133.88 us 13190.43 us 11 LOOKUP
79.41 60571.65 us 60571.65 us 60571.65 us 1 READDIRP
Duration: 138822 seconds
Data Read: 786432 bytes
Data Written: 0 bytes
--
Emmanuel Dreyfus
manu@xxxxxxxxxx
_______________________________________________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel
