Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 version

Mauro Tridici <mauro.tridici@xxxxxxx> · Thu, 4 Oct 2018 15:31:36 +0200

Hi Nithya,

thank you very much.
This is the current “gluster volume info” output after removing bricks (and after peer detach command).

[root@s01 ~]# gluster volume info

Volume Name: tier2
Type: Distributed-Disperse
Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c
Status: Started
Snapshot Count: 0
Number of Bricks: 6 x (4 + 2) = 36
Transport-type: tcp
Bricks:
Brick1: s01-stg:/gluster/mnt1/brick
Brick2: s02-stg:/gluster/mnt1/brick
Brick3: s03-stg:/gluster/mnt1/brick
Brick4: s01-stg:/gluster/mnt2/brick
Brick5: s02-stg:/gluster/mnt2/brick
Brick6: s03-stg:/gluster/mnt2/brick
Brick7: s01-stg:/gluster/mnt3/brick
Brick8: s02-stg:/gluster/mnt3/brick
Brick9: s03-stg:/gluster/mnt3/brick
Brick10: s01-stg:/gluster/mnt4/brick
Brick11: s02-stg:/gluster/mnt4/brick
Brick12: s03-stg:/gluster/mnt4/brick
Brick13: s01-stg:/gluster/mnt5/brick
Brick14: s02-stg:/gluster/mnt5/brick
Brick15: s03-stg:/gluster/mnt5/brick
Brick16: s01-stg:/gluster/mnt6/brick
Brick17: s02-stg:/gluster/mnt6/brick
Brick18: s03-stg:/gluster/mnt6/brick
Brick19: s01-stg:/gluster/mnt7/brick
Brick20: s02-stg:/gluster/mnt7/brick
Brick21: s03-stg:/gluster/mnt7/brick
Brick22: s01-stg:/gluster/mnt8/brick
Brick23: s02-stg:/gluster/mnt8/brick
Brick24: s03-stg:/gluster/mnt8/brick
Brick25: s01-stg:/gluster/mnt9/brick
Brick26: s02-stg:/gluster/mnt9/brick
Brick27: s03-stg:/gluster/mnt9/brick
Brick28: s01-stg:/gluster/mnt10/brick
Brick29: s02-stg:/gluster/mnt10/brick
Brick30: s03-stg:/gluster/mnt10/brick
Brick31: s01-stg:/gluster/mnt11/brick
Brick32: s02-stg:/gluster/mnt11/brick
Brick33: s03-stg:/gluster/mnt11/brick
Brick34: s01-stg:/gluster/mnt12/brick
Brick35: s02-stg:/gluster/mnt12/brick
Brick36: s03-stg:/gluster/mnt12/brick
Options Reconfigured:
network.ping-timeout: 0
features.scrub: Active
features.bitrot: on
features.inode-quota: on
features.quota: on
performance.client-io-threads: on
cluster.min-free-disk: 10
cluster.quorum-type: auto
transport.address-family: inet
nfs.disable: on
server.event-threads: 4
client.event-threads: 4
cluster.lookup-optimize: on
performance.readdir-ahead: on
performance.parallel-readdir: off
cluster.readdir-optimize: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 50000
performance.io-cache: off
disperse.cpu-extensions: auto
performance.io-thread-count: 16
features.quota-deem-statfs: on
features.default-soft-limit: 90
cluster.server-quorum-type: server
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
cluster.brick-multiplex: on
cluster.server-quorum-ratio: 51%

Regards,
Mauro

Il giorno 04 ott 2018, alle ore 15:22, Nithya Balachandran <nbalacha@xxxxxxxxxx> ha scritto:

Hi Mauro,

The files on s04 and s05 can be deleted safely as long as those bricks have been removed from the volume and their brick processes are not running.

.glusterfs/indices/xattrop/xattrop-* are links to files that need to be healed.
.glusterfs/quarantine/stub-00000000-0000-0000-0000-000000000008 links to files that bitrot (if enabled)says are corrupted. (none in this case)

I will get back to you on s06. Can you please provide the output of gluster volume info again?

Regards,
Nithya

On 4 October 2018 at 13:47, Mauro Tridici <mauro.tridici@xxxxxxx> wrote:

Dear Ashish, Dear Nithya,

I’m writing this message only to summarize and simplify the information about the "not migrated” files left on removed bricks on server s04, s05 and s06.
In attachment, you can find 3 files (1 file for each server) containing the “not migrated” files lists and related brick number.

In particular:
s04 and s05 bricks contain only not migrated files in hidden directories “/gluster/mnt#/brick/.glusterfs" (I could delete them, doesn’t it?)
s06 bricks contain 
not migrated files in hidden directories “/gluster/mnt#/brick/.glusterfs”;
not migrated files with size equal to 0;
not migrated files with size greater than 0.

I think it was necessary to collect and summarize information to simplify your analysis.
Thank you very much,
Mauro

-------------------------
Mauro Tridici

Fondazione CMCC
CMCC Supercomputing Center
presso Complesso Ecotekne - Università del Salento -
Strada Prov.le Lecce - Monteroni sn
73100 Lecce  IT
http://www.cmcc.it

mobile: (+39) 327 5630841
email: mauro.tridici@xxxxxxx
https://it.linkedin.com/in/mauro-tridici-5977238b

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users