Hello What am I doing wrong?!? I have a test setup with 4 identical servers with 2 disks each in distribute-replicate 2. All servers are connected to a GB switch. I am experiencing really slow speeds at anything I do. Slow write, slow read, not to mention random write/reads. Here is an example: random-files is a directory with 32768 files with average size 16kb. [root at gltclient]:~# rsync -a /root/speedtest/random-files/ /home/gltvolume/ ^^ This will take more than 3 hours. On any of the servers if I do "iostat" the disks are not loaded at all: https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png This is similar result for all servers. Here is an example of simple "ls" command on the content. [root at gltclient]:~# unalias ls [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls /home/gltvolume/ | wc -l 2.81 seconds 5393 almost 3 seconds to display 5000 files?!?! When they are 32,000, the ls will take around 35-45 seconds. This directory is on local disk: [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls /root/speedtest/random-files/ | wc -l 1.45 seconds 32768 [root at gltclient]:~# /usr/bin/time -f "%e seconds" cat /home/gltvolume/* >/dev/null 190.50 seconds [root at gltclient]:~# /usr/bin/time -f "%e seconds" du -sh /home/gltvolume/ 126M /home/gltvolume/ 75.23 seconds Here is the volume information. [root at glt1]:~# gluster volume info Volume Name: gltvolume Type: Distributed-Replicate Volume ID: 16edd852-8d23-41da-924d-710b753bb374 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 1.1.74.246:/home/sda3 Brick2: glt2.network.net:/home/sda3 Brick3: 1.1.74.246:/home/sdb1 Brick4: glt2.network.net:/home/sdb1 Brick5: glt3.network.net:/home/sda3 Brick6: gltclient.network.net:/home/sda3 Brick7: glt3.network.net:/home/sdb1 Brick8: gltclient.network.net:/home/sdb1 Options Reconfigured: performance.io-thread-count: 32 performance.cache-size: 256MB cluster.self-heal-daemon: on [root at glt1]:~# gluster volume status all detail Status of volume: gltvolume ------------------------------------------------------------------------------ Brick : Brick 1.1.74.246:/home/sda3 Port : 24009 Online : Y Pid : 1479 File System : ext4 Device : /dev/sda3 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11901550 ------------------------------------------------------------------------------ Brick : Brick glt2.network.net:/home/sda3 Port : 24009 Online : Y Pid : 1589 File System : ext4 Device : /dev/sda3 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11901550 ------------------------------------------------------------------------------ Brick : Brick 1.1.74.246:/home/sdb1 Port : 24010 Online : Y Pid : 1485 File System : ext4 Device : /dev/sdb1 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 228.8GB Total Disk Space : 229.2GB Inode Count : 15269888 Free Inodes : 15202933 ------------------------------------------------------------------------------ Brick : Brick glt2.network.net:/home/sdb1 Port : 24010 Online : Y Pid : 1595 File System : ext4 Device : /dev/sdb1 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 228.8GB Total Disk Space : 229.2GB Inode Count : 15269888 Free Inodes : 15202933 ------------------------------------------------------------------------------ Brick : Brick glt3.network.net:/home/sda3 Port : 24009 Online : Y Pid : 28963 File System : ext4 Device : /dev/sda3 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11906058 ------------------------------------------------------------------------------ Brick : Brick gltclient.network.net:/home/sda3 Port : 24009 Online : Y Pid : 3145 File System : ext4 Device : /dev/sda3 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11906058 ------------------------------------------------------------------------------ Brick : Brick glt3.network.net:/home/sdb1 Port : 24010 Online : Y Pid : 28969 File System : ext4 Device : /dev/sdb1 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 228.8GB Total Disk Space : 229.2GB Inode Count : 15269888 Free Inodes : 15207375 ------------------------------------------------------------------------------ Brick : Brick gltclient.network.net:/home/sdb1 Port : 24010 Online : Y Pid : 3151 File System : ext4 Device : /dev/sdb1 Mount Options : rw,noatime Inode Size : 256 Disk Space Free : 228.8GB Total Disk Space : 229.2GB Inode Count : 15269888 Free Inodes : 15207375
