| Howdy all, We're experiencing terrible small file performance when copying or moving files on gluster clients. In the example below, Gluster is taking 6mins~ to copy 128MB / 21,000 files sideways on a client, doing the same thing on NFS (which I know is a totally different solution etc. etc.) takes approximately 10-15 seconds(!). Any advice for tuning the volume or XFS settings would be greatly appreciated. Hopefully I've included enough relevant information below. ## Gluster Client root@gluster-client:/mnt/gluster_perf_test/ # du -sh . 127M . root@gluster-client:/mnt/gluster_perf_test/ # find . -type f | wc -l 21791 root@gluster-client:/mnt/gluster_perf_test/ # du 9584toto9584.txt 4 9584toto9584.txt root@gluster-client:/mnt/gluster_perf_test/ # time cp -a private private_perf_test real 5m51.862s user 0m0.862s sys 0m8.334s root@gluster-client:/mnt/gluster_perf_test/ # time rm -rf private_perf_test/ real 0m49.702s user 0m0.087s sys 0m0.958s ## Hosts - 16x Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz per Gluster host / client - Storage: iSCSI provisioned (via 10Gbit DAC/Fibre), SSD disk, 50K R/RW 4k IOP/s, 400MB/s per Gluster host - Volumes are replicated across two hosts and one arbiter only host - Networking is 10Gbit DAC/Fibre between Gluster hosts and clients - 18GB DDR4 ECC memory ## Volume Info root@gluster-host-01:~ # gluster pool list UUID Hostname State ad02970b-e2aa-4ca8-998c-bd10d5970faa gluster-host-02.fqdn Connected ea116a94-c19e-48db-b108-0be3ae622e2e gluster-host-03.fqdn Connected 2e855c25-e7ac-4ff6-be85-e8bcc6f45ee4 localhost Connected root@gluster-host-01:~ # gluster volume info uat_storage Volume Name: uat_storage Type: Replicate Volume ID: 7918f1c5-5031-47b8-b054-56f6f0c569a2 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: gluster-host-01.fqdn:/mnt/gluster-storage/uat_storage Brick2: gluster-host-02.fqdn:/mnt/gluster-storage/uat_storage Brick3: gluster-host-03.fqdn:/mnt/gluster-storage/uat_storage (arbiter) Options Reconfigured: performance.rda-cache-limit: 256MB network.inode-lru-limit: 50000 server.outstanding-rpc-limit: 256 performance.client-io-threads: true nfs.disable: on transport.address-family: inet client.event-threads: 8 cluster.eager-lock: true cluster.favorite-child-policy: size cluster.lookup-optimize: true cluster.readdir-optimize: true cluster.use-compound-fops: true diagnostics.brick-log-level: ERROR diagnostics.client-log-level: ERROR features.cache-invalidation-timeout: 600 features.cache-invalidation: true network.ping-timeout: 15 performance.cache-invalidation: true performance.cache-max-file-size: 6MB performance.cache-refresh-timeout: 60 performance.cache-size: 1024MB performance.io-thread-count: 16 performance.md-cache-timeout: 600 performance.stat-prefetch: true performance.write-behind-window-size: 256MB server.event-threads: 8 transport.listen-backlog: 2048 root@gluster-host-01:~ # xfs_info /dev/mapper/gluster-storage-unlocked meta-data="" isize=512 agcount=4, agsize=196607360 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0 data = bsize=4096 blocks=786429440, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=8192 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=383998, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 -- Sam McLeod (protoporpoise on IRC) https://smcleod.net https://twitter.com/s_mcleod Words are my own opinions and do not necessarily represent those of my employer or partners. |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
