Re: Problems to work with mounted directory in Gluster 3.2.7 -> switch to 3.2.4 ; -)

[BGM <bernhard.glomm@xxxxxxxxxxx>]

well, note:

- you don't need zfs on the hardeware machines, xfs or ext3 or ext4 would do it too

- for production you wouldn't use a glusterfs on top of a glusterfs but rather giving the vm access to a real blockdevice, like a whole harddisk or at least a partition of it although migration of the vm wouldn't be possible than...

therefor: a VM as a glusterserver might not be the best idea.

- remember to peer probe the glusterserver partner from both sides! as mentioned below

for a first setup you should be fine with that.

regards

On 19.02.2014, at 19:32, Targino Silveira <targinosilveira@xxxxxxxxx> wrote:

Thanks Bernhard I will do this.

Regards, 

Targino Silveira
+55-85-8626-7297
www.twitter.com/targinosilveira

2014-02-19 14:43 GMT-03:00 Bernhard Glomm <bernhard.glomm@xxxxxxxxxxx>:

I would strongly recommend to restart fresh with gluster 3.2.4 from 

http://download.gluster.org/pub/gluster/glusterfs/3.4/

It works totally fine for me.

(reinstall the vms as slim as possible if you can.)

As a quick howto consider this:

- We have 2 Hardware machines (just desktop machines for dev-env)

- both running zol

- create a zpool and zfs filesystem

- create a gluster replica 2 volume between hostA and hostB

- installe 3 VM vmachine0{4,5,6}

- vmachine0{4,5} each have a 100GB diskimage file as /dev/vdb which also resides on the glustervolume

- create ext3 filesystem on vmachine0{4,5}:/dev/vdb1

- create gluster replica 2 between vmachine04 and vmachine05 as shown below

(!!!obviously nobody would do that in any serious environment,

just to show that even a setup like that _would_ be possible!!!)

- run some benchmarks on that volume and compare the results to other 

So:

root@vmachine04[/0]:~ # mkdir -p /srv/vdb1/gf_brick

root@vmachine04[/0]:~ # mount /dev/vdb1 /srv/vdb1/

root@vmachine04[/0]:~ # gluster peer probe vmachine05

peer probe: success

# now switch over to vmachine05 and do

root@vmachine05[/1]:~ # mkdir -p /srv/vdb1/gf_brick

root@vmachine05[/1]:~ # mount /dev/vdb1 /srv/vdb1/

root@vmachine05[/1]:~ # gluster peer probe vmachine04

peer probe: success

root@vmachine05[/1]:~ # gluster peer probe vmachine04

peer probe: success: host vmachine04 port 24007 already in peer list

# the peer probe from BOTH sides ist often forgotten 

# switch back to vmachine04 and continue with

root@vmachine04[/0]:~ # gluster peer status

Number of Peers: 1

Hostname: vmachine05

Port: 24007

Uuid: 085a1489-dabf-40bb-90c1-fbfe66539953

State: Peer in Cluster (Connected)

root@vmachine04[/0]:~ # gluster volume info layer_cake_volume

Volume Name: layer_cake_volume

Type: Replicate

Volume ID: ef5299db-2896-4631-a2a8-d0082c1b25be

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: vmachine04:/srv/vdb1/gf_brick

Brick2: vmachine05:/srv/vdb1/gf_brick

root@vmachine04[/0]:~ # gluster volume status layer_cake_volume

Status of volume: layer_cake_volume

Gluster process                                         Port    Online  Pid

------------------------------------------------------------------------------

Brick vmachine04:/srv/vdb1/gf_brick                         49152   Y       12778

Brick vmachine05:/srv/vdb1/gf_brick                         49152   Y       16307

NFS Server on localhost                                 2049    Y       12790

Self-heal Daemon on localhost                           N/A     Y       12791

NFS Server on vmachine05                                    2049    Y       16320

Self-heal Daemon on vmachine05                              N/A     Y       16319

There are no active volume tasks

# set any option you might like

root@vmachine04[/1]:~ # gluster volume set layer_cake_volume network.remote-dio enable

volume set: success

# go to vmachine06 and mount the volume

root@vmachine06[/1]:~ # mkdir /srv/layer_cake

root@vmachine06[/1]:~ # mount -t glusterfs -o backupvolfile-server=vmachine05 vmachine04:/layer_cake_volume /srv/layer_cake

root@vmachine06[/1]:~ # mount

vmachine04:/layer_cake_volume on /srv/layer_cake type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

root@vmachine06[/1]:~ # df -h

Filesystem                 Size  Used Avail Use% Mounted on

...

vmachine04:/layer_cake_volume   97G  188M   92G   1% /srv/layer_cake

All fine and stable

# now let's see how it tastes

# note this is postmark on  / NOT on the glustermounted layer_cake_volume!

# that postmark results might be available tomorrow ;-)))

root@vmachine06[/1]:~ # postmark

PostMark v1.51 : 8/14/01

pm>set transactions 500000

pm>set number 200000

pm>set subdirectories 10000

pm>run

Creating subdirectories...Done

Creating files...Done

Performing transactions..........Done

Deleting files...Done

Deleting subdirectories...Done

Time:

        2314 seconds total

        2214 seconds of transactions (225 per second)

Files:

        450096 created (194 per second)

                Creation alone: 200000 files (4166 per second)

                Mixed with transactions: 250096 files (112 per second)

        249584 read (112 per second)

        250081 appended (112 per second)

        450096 deleted (194 per second)

                Deletion alone: 200192 files (3849 per second)

                Mixed with transactions: 249904 files (112 per second)

Data:

        1456.29 megabytes read (644.44 kilobytes per second)

        2715.89 megabytes written (1.17 megabytes per second)

# reference

# running postmark on the hardware machine directly on zfs

#

#           /test # postmark

#           PostMark v1.51 : 8/14/01

#           pm>set transactions 500000

#           pm>set number 200000

#           pm>set subdirectories 10000

#           pm>run

#           Creating subdirectories...Done

#           Creating files...Done

#           Performing transactions..........Done

#           Deleting files...Done

#           Deleting subdirectories...Done

#           Time:

#           605 seconds total

#           549 seconds of transactions (910 per second)

#

#           Files:

#           450096 created (743 per second)

#           Creation alone: 200000 files (4255 per second)

#           Mixed with transactions: 250096 files (455 per second)

#           249584 read (454 per second)

#           250081 appended (455 per second)

#           450096 deleted (743 per second)

#           Deletion alone: 200192 files (22243 per second)

#           Mixed with transactions: 249904 files (455 per second)

#

#           Data:

#           1456.29 megabytes read (2.41 megabytes per second)

#           2715.89 megabytes written (4.49 megabytes per second)

dbench -D /srv/layer_cake 5

 Operation      Count    AvgLat    MaxLat

 ----------------------------------------

 NTCreateX     195815     5.159   333.296

 Close         143870     0.793    93.619

 Rename          8310    10.922   123.096

 Unlink         39525     2.428   203.753

 Qpathinfo     177736     2.551   220.605

 Qfileinfo      31030     2.057   175.565

 Qfsinfo        32545     1.393   174.045

 Sfileinfo      15967     2.691   129.028

 Find           68664     9.629   185.739

 WriteX         96860     0.841   108.863

 ReadX         307834     0.511   213.602

 LockX            642     1.511    10.578

 UnlockX          642     1.541    10.137

 Flush          13712    12.853   405.383

Throughput 10.1832 MB/sec  5 clients  5 procs  max_latency=405.405 ms

# reference

dbench -D /tmp 5

# reference

dbench -D /tmp 5

 Operation      Count    AvgLat    MaxLat

 ----------------------------------------

 NTCreateX    3817455     0.119   499.847

 Close        2804160     0.005    16.000

 Rename        161655     0.322   459.790

 Unlink        770906     0.556   762.314

 Deltree           92    20.647    81.619

 Mkdir             46     0.003     0.012

 Qpathinfo    3460227     0.017    18.388

 Qfileinfo     606258     0.003    11.652

 Qfsinfo       634444     0.006    14.976

 Sfileinfo     310990     0.155   604.585

 Find         1337732     0.056    18.466

 WriteX       1902611     0.245   503.604

 ReadX        5984135     0.008    16.154

 LockX          12430     0.008     9.111

 UnlockX        12430     0.004     4.551

 Flush         267557     4.505   902.093

Throughput 199.664 MB/sec  5 clients  5 procs  max_latency=902.099 ms

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users