Hi Vijay,
Please find the volume info here :-
[root@cpu01 ~]# gluster volume info
Volume Name: ds01
Type: Distributed-Replicate
Volume ID: 369d3fdc-c8eb-46b7-a33e-0a49f2451ff6
Status: Started
Number of Bricks: 48 x 2 = 96
Transport-type: tcp
Bricks:
Brick1: cpu01:/bricks/1/vol1
Brick2: cpu02:/bricks/1/vol1
Brick3: cpu03:/bricks/1/vol1
Brick4: cpu04:/bricks/1/vol1
Brick5: cpu01:/bricks/2/vol1
Brick6: cpu02:/bricks/2/vol1
Brick7: cpu03:/bricks/2/vol1
Brick8: cpu04:/bricks/2/vol1
Brick9: cpu01:/bricks/3/vol1
Brick10: cpu02:/bricks/3/vol1
Brick11: cpu03:/bricks/3/vol1
Brick12: cpu04:/bricks/3/vol1
Brick13: cpu01:/bricks/4/vol1
Brick14: cpu02:/bricks/4/vol1
Brick15: cpu03:/bricks/4/vol1
Brick16: cpu04:/bricks/4/vol1
Brick17: cpu01:/bricks/5/vol1
Brick18: cpu02:/bricks/5/vol1
Brick19: cpu03:/bricks/5/vol1
Brick20: cpu04:/bricks/5/vol1
Brick21: cpu01:/bricks/6/vol1
Brick22: cpu02:/bricks/6/vol1
Brick23: cpu03:/bricks/6/vol1
Brick24: cpu04:/bricks/6/vol1
Brick25: cpu01:/bricks/7/vol1
Brick26: cpu02:/bricks/7/vol1
Brick27: cpu03:/bricks/7/vol1
Brick28: cpu04:/bricks/7/vol1
Brick29: cpu01:/bricks/8/vol1
Brick30: cpu02:/bricks/8/vol1
Brick31: cpu03:/bricks/8/vol1
Brick32: cpu04:/bricks/8/vol1
Brick33: cpu01:/bricks/9/vol1
Brick34: cpu02:/bricks/9/vol1
Brick35: cpu03:/bricks/9/vol1
Brick36: cpu04:/bricks/9/vol1
Brick37: cpu01:/bricks/10/vol1
Brick38: cpu02:/bricks/10/vol1
Brick39: cpu03:/bricks/10/vol1
Brick40: cpu04:/bricks/10/vol1
Brick41: cpu01:/bricks/11/vol1
Brick42: cpu02:/bricks/11/vol1
Brick43: cpu03:/bricks/11/vol1
Brick44: cpu04:/bricks/11/vol1
Brick45: cpu01:/bricks/12/vol1
Brick46: cpu02:/bricks/12/vol1
Brick47: cpu03:/bricks/12/vol1
Brick48: cpu04:/bricks/12/vol1
Brick49: cpu01:/bricks/13/vol1
Brick50: cpu02:/bricks/13/vol1
Brick51: cpu03:/bricks/13/vol1
Brick52: cpu04:/bricks/13/vol1
Brick53: cpu01:/bricks/14/vol1
Brick54: cpu02:/bricks/14/vol1
Brick55: cpu03:/bricks/14/vol1
Brick56: cpu04:/bricks/14/vol1
Brick57: cpu01:/bricks/15/vol1
Brick58: cpu02:/bricks/15/vol1
Brick59: cpu03:/bricks/15/vol1
Brick60: cpu04:/bricks/15/vol1
Brick61: cpu01:/bricks/16/vol1
Brick62: cpu02:/bricks/16/vol1
Brick63: cpu03:/bricks/16/vol1
Brick64: cpu04:/bricks/16/vol1
Brick65: cpu01:/bricks/17/vol1
Brick66: cpu02:/bricks/17/vol1
Brick67: cpu03:/bricks/17/vol1
Brick68: cpu04:/bricks/17/vol1
Brick69: cpu01:/bricks/18/vol1
Brick70: cpu02:/bricks/18/vol1
Brick71: cpu03:/bricks/18/vol1
Brick72: cpu04:/bricks/18/vol1
Brick73: cpu01:/bricks/19/vol1
Brick74: cpu02:/bricks/19/vol1
Brick75: cpu03:/bricks/19/vol1
Brick76: cpu04:/bricks/19/vol1
Brick77: cpu01:/bricks/20/vol1
Brick78: cpu02:/bricks/20/vol1
Brick79: cpu03:/bricks/20/vol1
Brick80: cpu04:/bricks/20/vol1
Brick81: cpu01:/bricks/21/vol1
Brick82: cpu02:/bricks/21/vol1
Brick83: cpu03:/bricks/21/vol1
Brick84: cpu04:/bricks/21/vol1
Brick85: cpu01:/bricks/22/vol1
Brick86: cpu02:/bricks/22/vol1
Brick87: cpu03:/bricks/22/vol1
Brick88: cpu04:/bricks/22/vol1
Brick89: cpu01:/bricks/23/vol1
Brick90: cpu02:/bricks/23/vol1
Brick91: cpu03:/bricks/23/vol1
Brick92: cpu04:/bricks/23/vol1
Brick93: cpu01:/bricks/24/vol1
Brick94: cpu02:/bricks/24/vol1
Brick95: cpu03:/bricks/24/vol1
Brick96: cpu04:/bricks/24/vol1
Options Reconfigured:
nfs.disable: off
user.cifs: enable
auth.allow: 10.10.0.*
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
server.allow-insecure: on
[root@cpu01 ~]#
Thanks,
punit
On Tue, Feb 17, 2015 at 6:16 AM, Ben Turner <bturner@xxxxxxxxxx> wrote:
----- Original Message -----
> From: "Joe Julian" <joe@xxxxxxxxxxxxxxxx>
> To: "Punit Dambiwal" <hypunit@xxxxxxxxx>, gluster-users@xxxxxxxxxxx, "Humble Devassy Chirammal"
> <humble.devassy@xxxxxxxxx>
> Sent: Monday, February 16, 2015 3:32:31 PM
> Subject: Re: Gluster performance on the small files
>
>
> On 02/12/2015 10:58 PM, Punit Dambiwal wrote:
>
>
>
> Hi,
>
> I have seen the gluster performance is dead slow on the small files...even i
> am using the SSD....it's too bad performance....even i am getting better
> performance in my SAN with normal SATA disk...
>
> I am using distributed replicated glusterfs with replica count=2...i have all
> SSD disks on the brick...
>
>
>
> root@vm3:~# dd bs=64k count=4k if=/dev/zero of=test oflag=dsync
>
> 4096+0 records in
>
> 4096+0 records out
>
> 268435456 bytes (268 MB) copied, 57.3145 s, 4.7 MB/s
>
This seems pretty slow, even if you are using gigabit. Here is what I get:
[root@gqac031 smallfile]# dd bs=64k count=4k if=/dev/zero of=/gluster-emptyvol/test oflag=dsync
4096+0 records in
4096+0 records out
268435456 bytes (268 MB) copied, 10.5965 s, 25.3 MB/s
FYI this is on my 2 node pure replica + spinning disks(RAID 6, this is not setup for smallfile workloads. For smallfile I normally use RAID 10) + 10G.
The single threaded DD process is defiantly a bottle neck here, the power in distributed systems is doing things in parallel across clients / threads. You may want to try smallfile:
http://www.gluster.org/community/documentation/index.php/Performance_Testing
Smallfile command used - python /small-files/smallfile/smallfile_cli.py --operation create --threads 8 --file-size 64 --files 10000 --top /gluster-emptyvol/ --pause 1000 --host-set "client1, client2"
total threads = 16
total files = 157100
total data ="" 9.589 GB
98.19% of requested files processed, minimum is 70.00
41.271602 sec elapsed time
3806.491454 files/sec
3806.491454 IOPS
237.905716 MB/sec
If you wanted to do something similar with DD you could do:
<my script>
for i in `seq 1..4`
do
dd bs=64k count=4k if=/dev/zero of=/gluster-emptyvol/test$i oflag=dsync &
done
for pid in $(pidof dd); do
while kill -0 "$pid"; do
sleep 0.1
done
done
# time myscript.sh
Then do the math to figure out the MB / sec of the system.
-b
>
>
> root@vm3:~# dd bs=64k count=4k if=/dev/zero of=test conv=fdatasync
>
> 4096+0 records in
>
> 4096+0 records out
>
> 268435456 bytes (268 MB) copied, 1.80093 s, 149 MB/s
>
>
>
> How small is your VM image? The image is the file that GlusterFS is serving,
> not the small files within it. Perhaps the filesystem you're using within
> your VM is inefficient with regard to how it handles disk writes.
>
> I believe your concept of "small file" performance is misunderstood, as is
> often the case with this phrase. The "small file" issue has to do with the
> overhead of finding and checking the validity of any file, but with a small
> file the percentage of time doing those checks is proportionally greater.
> With your VM image, that file is already open. There are no self-heal checks
> or lookups that are happening in your tests, so that overhead is not the
> problem.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
