Re: GlusterFS 3.6.2, volume from 4 to 8 bricks & CPU went sky high

[Sander Zijlstra <sander.zijlstra@xxxxxxxxxxx>]

Ho jiri

Looking in the logs I also think that a self heal is triggered as I see numerous entries with dht_heal regarding volume relay out activities.

So yes that might be the cause, but my personal impression was that this would only be started by a rebalance, so I expect that it's caused because the application "owncloud" is constantly scanning the file system once users log in; currently something like 2500 a day ....

Thanks for the input

Met vriendelijke groet / kind regards,

Sander Zijlstra

| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | 

T +31(0)6 43 99 12 47 | sander.zijlstra@xxxxxxxxxxx | www.surfsara.nl |

Op 09 Apr 2015 om 15:15 heeft Jiri Hoogeveen <j.hoogeveen@xxxxxxxxxxxxxxxx> het volgende geschreven:

Hi Sander,

It sounds to me, that it triggered the self-healing, which will do a scan on the bricks. Depending on the number of files on the brick, it can use a lot of CPU.

Does the logs say anything useful?

 

Grtz, 

Jiri  Hoogeveen

On 09 Apr 2015, at 14:18, Sander Zijlstra <sander.zijlstra@xxxxxxxxxxx> wrote:

LS,

We have a GlusterFS cluster which consists of 4 nodes with one brick each and a distributed-replicated volume of 72 TB.

Today I extended the cluster to 8 machines and added new bricks to the volume, so it now contains 8 bricks.

I didn’t start the rebalance yet to limit the impact during the day but to my surprise all glusterfsd process went sky high and performance was really really bad. So effectively I cause downtime to our storage service while I didn’t anticipated this, hence I didn’t do any rebalance yet.

Can somebody explain to me why adding bricks to a volume causes this high CPU usage? I can imagine the meta data needed to be synced but if this is so heavy, why can’t I tune this?

This is my current volume setup:

Volume Name: gv0

Type: Distributed-Replicate

Volume ID: 0322f20f-e507-492b-91db-cb4c953a24eb

Status: Started

Number of Bricks: 4 x 2 = 8

Transport-type: tcp

Bricks:

Brick1: s-s35-06:/glusterfs/bricks/brick1/brick

Brick2: s-s35-07:/glusterfs/bricks/brick1/brick

Brick3: s-s35-08:/glusterfs/bricks/brick1/brick

Brick4: s-s35-09:/glusterfs/bricks/brick1/brick

Brick5: v39-app-01:/glusterfs/bricks/brick1/gv0

Brick6: v39-app-02:/glusterfs/bricks/brick1/gv0

Brick7: v39-app-03:/glusterfs/bricks/brick1/gv0

Brick8: v39-app-04:/glusterfs/bricks/brick1/gv0

Options Reconfigured:

performance.cache-size: 256MB

nfs.disable: on

geo-replication.indexing: off

geo-replication.ignore-pid-check: on

changelog.changelog: on

performance.io-thread-count: 32

performance.write-behind-window-size: 5MB

Met vriendelijke groet / kind regards,

Sander Zijlstra

| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijlstra@xxxxxxxxxxx | www.surfsara.nl |

Regular day off on friday

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users