Hello, thanks for your answer! Yes, I am using sharding. That the 1 Gigabit NIC is the limit was my second thought as well, because when I run this "performance" test on all three nodes at the same time, I get roughly around ~80MB/sec, which is close(r) to the theoretical maximum throughput of the NICs. I was hoping that there is a way to more "loosen" the strict synchronous approach of writing files to the other nodes to gain write performance (and loose some security) so that files are being written faster locally. But it seems like there is no way around this (other than the way you mentioned w/ the new Thin Arbiter) and if the files would be written asynchronously that would create other problems (like that the destination must "poll" until a file appears). So, the way to go for me would be to switch to 10GBit/sec NICs (maybe even with Bonding) to get a better write performance that is closer to the speed of the NVMe disks. In the mean-time, I have changed the software architecture a bit and let the application that does the writes, write directly to the local NVMe disk and then asynchronously move the file(s) from the "buffer" to the gluster volume. This helps to prevent that the application process is blocked for a too long time. Thanks for your help and explanation! Marc > On 11/9/2020 12:59 PM, Marc Jakobs wrote: >> I have a GlusterFS Volume on three Linux Servers (Ubuntu 20.04LTS) which >> are connected via 1GBit/sec NIC with each other over a dedicated switch. >> >> Every server has a NVMe disk which is used for the GlusterFS Volume >> called "data". > > So I assume you have a simple replica 3 setup. > > Are you using sharding? > > >> I have mounted the Volume like this >> >> mount -t glusterfs -o direct-io-mode=disable 127.0.0.1:/data /mnt/test/ >> >> so it does not even go over the local NIC but instead over the loopback >> device. > > You are Network constrained. > > Your mount is direct, but if you have replica 3 the data still has to > travel to the other two gluster bricks and that is occurring over a > single 1 Gbit/s ethernet port which would have a maximum throughput of > 125 MB/s. > > Since you have two streams going out that is roughly 62+ MB/s assuming > full replica 3. > > My understanding is that gluster doesn't acknowledge a write until its > been written to at least one of the replicas ( I am sure others will > jump in and correct me). So 60 MB/s under those circumstances is what I > would expect to see. > > You can improve things by using an arbiter and supposedly the new Thin > Arbiter is even faster (though I haven't tried it), but you lose a > little safety The arbiter node only receives the metadata so it can > referee on split-brain decisions, freeing up more BW for the actually > data replica node. > > A huge improvement would be if you were to bond two or more Gbit/s > ports. Round-Robin teamd is really easy to setup, or use the traditional > bonding in its various flavors. You probably have some spare NIC cards > lying around so its usually a 'freebie' > > Of course best case would be to make the jump to 10Gb/s kit. > > -wk > ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
