What do you mean with "small files"? 16k ? 160k? 16mb? Do you know any workaround or any other software for this? Mee too i'm trying to create a clustered storage for many small file 2012/8/10 Philip Poten <philip.poten at gmail.com> > Hi Ivan, > > that's because Gluster has really bad "many small files" performance > due to it's architecture. > > On all stat() calls (which rsync is doing plenty of), all replicas are > being checked for integrity. > > regards, > Philip > > 2012/8/10 Ivan Dimitrov <dobber at amln.net>: > > So I stopped a node to check the BIOS and after it went up, the rebalance > > kicked in. I was looking for those kind of speeds on a normal write. The > > rebalance is much faster than my rsync/cp. > > > > > https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%202.04.09%20PM.png > > > > Best Regards > > Ivan Dimitrov > > > > > > On 8/10/12 1:23 PM, Ivan Dimitrov wrote: > >> > >> Hello > >> What am I doing wrong?!? > >> > >> I have a test setup with 4 identical servers with 2 disks each in > >> distribute-replicate 2. All servers are connected to a GB switch. > >> > >> I am experiencing really slow speeds at anything I do. Slow write, slow > >> read, not to mention random write/reads. > >> > >> Here is an example: > >> random-files is a directory with 32768 files with average size 16kb. > >> [root at gltclient]:~# rsync -a /root/speedtest/random-files/ > >> /home/gltvolume/ > >> ^^ This will take more than 3 hours. > >> > >> On any of the servers if I do "iostat" the disks are not loaded at all: > >> > >> > https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png > >> > >> This is similar result for all servers. > >> > >> Here is an example of simple "ls" command on the content. > >> [root at gltclient]:~# unalias ls > >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls /home/gltvolume/ > | wc > >> -l > >> 2.81 seconds > >> 5393 > >> > >> almost 3 seconds to display 5000 files?!?! When they are 32,000, the ls > >> will take around 35-45 seconds. > >> > >> This directory is on local disk: > >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls > >> /root/speedtest/random-files/ | wc -l > >> 1.45 seconds > >> 32768 > >> > >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" cat /home/gltvolume/* > >> >/dev/null > >> 190.50 seconds > >> > >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" du -sh > /home/gltvolume/ > >> 126M /home/gltvolume/ > >> 75.23 seconds > >> > >> > >> Here is the volume information. > >> > >> [root at glt1]:~# gluster volume info > >> > >> Volume Name: gltvolume > >> Type: Distributed-Replicate > >> Volume ID: 16edd852-8d23-41da-924d-710b753bb374 > >> Status: Started > >> Number of Bricks: 4 x 2 = 8 > >> Transport-type: tcp > >> Bricks: > >> Brick1: 1.1.74.246:/home/sda3 > >> Brick2: glt2.network.net:/home/sda3 > >> Brick3: 1.1.74.246:/home/sdb1 > >> Brick4: glt2.network.net:/home/sdb1 > >> Brick5: glt3.network.net:/home/sda3 > >> Brick6: gltclient.network.net:/home/sda3 > >> Brick7: glt3.network.net:/home/sdb1 > >> Brick8: gltclient.network.net:/home/sdb1 > >> Options Reconfigured: > >> performance.io-thread-count: 32 > >> performance.cache-size: 256MB > >> cluster.self-heal-daemon: on > >> > >> > >> [root at glt1]:~# gluster volume status all detail > >> Status of volume: gltvolume > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick 1.1.74.246:/home/sda3 > >> Port : 24009 > >> Online : Y > >> Pid : 1479 > >> File System : ext4 > >> Device : /dev/sda3 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 179.3GB > >> Total Disk Space : 179.7GB > >> Inode Count : 11968512 > >> Free Inodes : 11901550 > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick glt2.network.net:/home/sda3 > >> Port : 24009 > >> Online : Y > >> Pid : 1589 > >> File System : ext4 > >> Device : /dev/sda3 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 179.3GB > >> Total Disk Space : 179.7GB > >> Inode Count : 11968512 > >> Free Inodes : 11901550 > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick 1.1.74.246:/home/sdb1 > >> Port : 24010 > >> Online : Y > >> Pid : 1485 > >> File System : ext4 > >> Device : /dev/sdb1 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 228.8GB > >> Total Disk Space : 229.2GB > >> Inode Count : 15269888 > >> Free Inodes : 15202933 > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick glt2.network.net:/home/sdb1 > >> Port : 24010 > >> Online : Y > >> Pid : 1595 > >> File System : ext4 > >> Device : /dev/sdb1 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 228.8GB > >> Total Disk Space : 229.2GB > >> Inode Count : 15269888 > >> Free Inodes : 15202933 > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick glt3.network.net:/home/sda3 > >> Port : 24009 > >> Online : Y > >> Pid : 28963 > >> File System : ext4 > >> Device : /dev/sda3 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 179.3GB > >> Total Disk Space : 179.7GB > >> Inode Count : 11968512 > >> Free Inodes : 11906058 > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick gltclient.network.net:/home/sda3 > >> Port : 24009 > >> Online : Y > >> Pid : 3145 > >> File System : ext4 > >> Device : /dev/sda3 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 179.3GB > >> Total Disk Space : 179.7GB > >> Inode Count : 11968512 > >> Free Inodes : 11906058 > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick glt3.network.net:/home/sdb1 > >> Port : 24010 > >> Online : Y > >> Pid : 28969 > >> File System : ext4 > >> Device : /dev/sdb1 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 228.8GB > >> Total Disk Space : 229.2GB > >> Inode Count : 15269888 > >> Free Inodes : 15207375 > >> > >> > ------------------------------------------------------------------------------ > >> Brick : Brick gltclient.network.net:/home/sdb1 > >> Port : 24010 > >> Online : Y > >> Pid : 3151 > >> File System : ext4 > >> Device : /dev/sdb1 > >> Mount Options : rw,noatime > >> Inode Size : 256 > >> Disk Space Free : 228.8GB > >> Total Disk Space : 229.2GB > >> Inode Count : 15269888 > >> Free Inodes : 15207375 > >> _______________________________________________ > >> Gluster-users mailing list > >> Gluster-users at gluster.org > >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > >> > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20120811/34a7bbd1/attachment.htm>
