-----Original Mail----- From: Jung Young Seok [jung.youngseok at gmail.com] Sent: 22.08.13 - 19:09:32 To: Maik Kulbe [info at linux-web-development.de] Cc: gluster-users at gluster.org Subject: Re: Writing performance gets slower when writethreadsare many > 2013/8/23 Maik Kulbe <info at linux-web-development.de> > > -----Original Mail----- > From: Jung Young Seok [jung.youngseok at gmail.com] > Sent: 22.08.13 - 17:19:32 > To: gluster-users at gluster.org > > Subject: Writing performance gets slower when write > threadsare many > > Dear gluster users, > > We have set up? glusterfs 3.4 on cent 6. 2 Gluster FS servers are > replicated each (2brick x 2). > Glusterfs is mounted on WAS as fuse. > > We realized when there are more thread to write files on gluster fs, > it > gets slower on writing. > > We've set 250 thread to write 1M file each repeatedly. In this case we > get > under 30 TPS. No NW bandwidth problem. > Is there any way to avoid performance degradation when there are lots > of > write threads? > > That depends on a lot of factors - what interconnect are you using? 1GE? > 10GE? RDMA? What are your bricks built on? SATA? SSD? RAID? > > My first guess would be that interconnect latency and a kernel maximum > for some connection count could be the problem. > > ? > Thank you for the comments, > We use 1GE for storage network. When I checked the net stream using iftop, > it never reached 1G. (only was under 500Mbps. Since we have 2 glusterfs > servers, 250Mb for first brick 250Mb for second brick) > When I scp a file to Gluster servers from WAS?directly?, ?it reached > 80MB/s each. (used less encrypted option) > About HDD,? > First server is on SAS 10k RAID1. > Second server is on SATA 10k RAID1. Have you benchmarked both network and harddisks without gluster? I would especially look at the network timings and there behaviour with multiple connections between clients and gluster servers. From my experience I would say too high latencies could very quickly become a bottleneck for the whole system. > Those are for test. We'll use SATA RAID1 both finally to reduce cost. > We are using CentOS 6.4. Since we use vanilla installation, most of kernel > parameters are defaults. > If you give me any option that I can test, it would be great. There is an article on that in the Gluster docs: http://www.gluster.org/community/documentation/index.php/Linux_Kernel_Tuning You can also look at the performance options that Gluster provides. Those are in the Gluster administration guide: http://www.gluster.org/wp-content/uploads/2012/05/Gluster_File_System-3.3.0-Administration_Guide-en-US.pdf - there are some caching mechanisms, etc. that might increase your write rate. > Thank you. > ? > > Look forward any help. > Thanks. > > Regards, > Youngseok jung
