Re: Number of shards in .shard directory

[Krutika Dhananjay <kdhananj@xxxxxxxxxx>]

Hmm I don't have an answer to that question right now because I'm no on-disk fs expert.

As far as gluster is concerned, I don't actually see any issue except for the following:

as of today, gluster's entry self-heal algorithm crawls the whole parent directory and compares the

source and sink to figure out what files need to be created/deleted on the sink brick.

If the shards were out into multiple directories as opposed to just /.shard, entry self-heal would take

lesser time and resources to complete in certain cases since the crawling could possibly be confined to only those few directories and they would all within them contain fewer shards.

But with granular entry self-heal feature which is going to be introduced in 3.8, this will also be a non-issue since

there AFR will record the exact shards that were created/deleted while a brick is down and only sync them to the sink brick once it is back up.

https://github.com/gluster/glusterfs-specs/blob/master/accepted/granular-entry-self-healing.md

-Krutika

-Krutika

On Sat, Apr 16, 2016 at 5:38 AM, Lindsay Mathieson <lindsay.mathieson@xxxxxxxxx> wrote:

Looking at migrating all our VM's to sharded gluster storage, 2.5TB of data in total. Underlying storage is zfs (on linux).

Small scale testing has been going well but one thing I am unsure of is whether a *large* number of small files in one directory is going to be a problem. If I migrate everything to a 4MB shard size there is going to be 655360 files. Will that be a problem for gluster? or for zfs?

--
Lindsay Mathieson

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users