Hi, I am using 26x8TB disks per server. There are 60 servers in gluster cluster. Each disk is a brick and configuration is 16+4 EC, 9PB single volume. Clients are using fuse mounts. Even with 1-2K files in a directory, ls from clients takes ~60 secs. So If you are sensitive to metadata operations, I suggest another approach... On Wed, Nov 9, 2016 at 1:05 PM, Frank Rothenstein <f.rothenstein@xxxxxxxxxxxxxxxxxx> wrote: > As you said you want to have 3 or 4 replicas, so i would use the zfs > knowledge and build 1 zpool per node with whatever config you know is > fastest on this kind of hardware and as safe as you need (stripe, > mirror, raidz1..3 - resilvering zfs is faster than healing gluster, I > think) . 1 node -> 1 brick (per gluster volume). > > Frank > Am Dienstag, den 08.11.2016, 19:19 +0000 schrieb Thomas Wakefield: >> We haven’t decided how the JBODS would be configured. They would >> likely be SAS attached without a raid controller for improved >> performance. I run large ZFS arrays this way, but only in single >> server NFS setups right now. >> Mounting each hard drive as it’s own brick would probably give the >> most usable space, but would need scripting to manage building all >> the bricks. But does Gluster handle 1000’s of small bricks? >> >> >> >> > On Nov 8, 2016, at 9:18 AM, Frank Rothenstein <f.rothenstein@bodden >> > -kliniken.de> wrote: >> > >> > Hi Thomas, >> > >> > thats a huge storage. >> > What I can say from my usecase - dont use Gluster directly if the >> > files >> > are small. I dont know, if the file count matters, but if the files >> > are >> > small (few KiB), Gluster takes ages to remove for example. Doing >> > the >> > same in a VM with e.g. ext4 disk on the very same Gluster gives a >> > big >> > speedup. >> > There are many options for a new Gluster volume, like Lindsay >> > mentioned. >> > And there are other options, like Ceph, OrangeFS. >> > How do you want to use the JBODs? I dont think you would use every >> > single drive as a brick... How are these connected to the servers? >> > >> > Im only dealing with about 10TiB Gluster volumes, so by far not at >> > your >> > planned level, but I really would like to see some results, if you >> > go >> > for Gluster! >> > >> > Frank >> > >> > >> > Am Dienstag, den 08.11.2016, 13:49 +0000 schrieb Thomas Wakefield: >> > > I think we are leaning towards erasure coding with 3 or 4 >> > > copies. But open to suggestions. >> > > >> > > >> > > > On Nov 8, 2016, at 8:43 AM, Lindsay Mathieson <lindsay.mathieso >> > > > n@gm >> > > > ail.com> wrote: >> > > > >> > > > On 8/11/2016 11:38 PM, Thomas Wakefield wrote: >> > > > > High Performance Computing, we have a small cluster on campus >> > > > > of >> > > > > about 50 linux compute servers. >> > > > > >> > > > >> > > > D'oh! I should have thought of that. >> > > > >> > > > >> > > > Are you looking at replication (2 or 3)/disperse or pure >> > > > disperse? >> > > > >> > > > -- >> > > > Lindsay Mathieson >> > > > >> > > >> > > _______________________________________________ >> > > Gluster-users mailing list >> > > Gluster-users@xxxxxxxxxxx >> > > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> > >> > >> > >> > >> > ___________________________________________________________________ >> > ___________ >> > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH >> > Sandhufe 2 >> > 18311 Ribnitz-Damgarten >> > >> > Telefon: 03821-700-0 >> > Fax: 03821-700-240 >> > >> > E-Mail: info@xxxxxxxxxxxxxxxxxx Internet: http://www.bodden-klini >> > ken.de >> > >> > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer- >> > Nr.: 079/133/40188 >> > Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. >> > Falko Milski >> > >> > Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten >> > Adressaten bestimmt. Wenn Sie nicht der vorge- >> > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, >> > beachten Sie bitte, dass jede Form der Veröf- >> > fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E- >> > Mail unzulässig ist. Wir bitten Sie, sofort den >> > Absender zu informieren und die E-Mail zu löschen. >> > >> > >> > Bodden-Kliniken Ribnitz-Damgarten GmbH 2016 >> > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus *** >> > >> >> > > > > > > ______________________________________________________________________________ > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH > Sandhufe 2 > 18311 Ribnitz-Damgarten > > Telefon: 03821-700-0 > Fax: 03821-700-240 > > E-Mail: info@xxxxxxxxxxxxxxxxxx Internet: http://www.bodden-kliniken.de > > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.: 079/133/40188 > Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko Milski > > Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorge- > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten Sie bitte, dass jede Form der Veröf- > fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail unzulässig ist. Wir bitten Sie, sofort den > Absender zu informieren und die E-Mail zu löschen. > > > Bodden-Kliniken Ribnitz-Damgarten GmbH 2016 > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus *** > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
