Thanks for sharing. It looks like 3.5 million files are directly
placed under /export/read-write/g0{1,2} on 2 servers. I am assuming
performance stays the same as no. of files increase.
Couple of more questions:
Did you also consider using RAID 0 over these 2 disks on the same host?
Do you know what happens when a new node is added. Since gluster uses
elastic hashing does it still work or does new node throws off the
hashing algorithm. I am wondering how gluster is able to retrieve old
files after adding new node because now it's hashing accross new
nodes.
Thanks again and keep us posted!
I will be starting similar testing in next 2 weeks also.
On Wed, Mar 9, 2011 at 10:53 AM, Burnash, James <jburnash at knight.com> wrote:
> I'm running ext4 on CentOS 5.5. Each storage server has two bricks, each in their own directory, mounted as straight partitions. No LVM used in this config.
>
> I did find that I had to keep the ext4 format and mount options simple to avoid crashes that I encountered with more "tuned" configs.
>
> # create ext4 read-write storage
>
> yum -y install e4fsprogs e4fsprogs-libs e4fsprogs-devel dmapi
>
> create a single partition on each disk, maximum size, primary type
>
> time mke4fs -F -L g01 -v -j /dev/cciss/c1d0p1
> time mke4fs -F -L g02 -v -j /dev/cciss/c2d0p1
>
> mkdir -p /export/read-write/g0{1,2}
> mount -t ext4 /dev/cciss/c1d0p1 /export/read-write/g01
> mount
> mount -t ext4 /dev/cciss/c2d0p1 /export/read-write/g02
>
> fgrep read-write /etc/fstab
> /dev/cciss/c1d0p1 ? ? ? /export/read-write/g01 ?ext4 ? ?defaults ? ? ? ?0 0
> /dev/cciss/c2d0p1 ? ? ? /export/read-write/g02 ?ext4 ? ?defaults ? ? ? ?0 0
>
> James Burnash, Unix Engineering
>
> -----Original Message-----
> From: Mohit Anchlia [mailto:mohitanchlia at gmail.com]
> Sent: Wednesday, March 09, 2011 1:15 PM
> To: Burnash, James; gluster-users at gluster.org
> Subject: Re: Files per directory
>
> Thanks! Is this on ext3 or ext4? Are all these files in mount
> directory or they are in sub directories. On glusterfs does it matter
> if all the files are placed in same directory? Generally from what
> I've seen in the past is that multiple no. of subdirs are recommended
> to improve performance.
>
> On Wed, Mar 9, 2011 at 9:59 AM, Burnash, James <jburnash at knight.com> wrote:
>> I'm going through a rebalance operation now on my "small" Glusterfs storage pool - 2 servers, 4 bricks, 30TB of total storage, 175 native Glusterfs clients.
>>
>> Current files checked is at 3.5 million - a lot of those are in the 1-1.5GB size range.
>>
>> Hopefully that is of some help - more stats to follow as I get a chance to document them ...
>>
>> James Burnash, Unix Engineering
>>
>>
>> -----Original Message-----
>> From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Mohit Anchlia
>> Sent: Wednesday, March 09, 2011 12:47 PM
>> To: gluster-users at gluster.org
>> Subject: Re: Files per directory
>>
>> It will be good if I can get some suggestion from people who already
>> have millions of files on glusterFS.
>>
>> On Mon, Mar 7, 2011 at 4:11 PM, Mohit Anchlia <mohitanchlia at gmail.com> wrote:
>>> Is there any recommendation about how many files should be stored in
>>> one directory in glusterFS? In my experience I've seen spreading files
>>> accross many directories helps, but I am not sure if it's same with
>>> glusterFS.
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>> DISCLAIMER:
>> This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission.
>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
>>
>
