Sorry for reviving an old thread, but could I get some input on this, pretty please? ext4 has 256-byte inodes by default (at least according to docs) but the fragment below says: OPTION(filestore_max_inline_xattr_size_other, OPT_U32, 512) The default 512b is too much if the inode is just 256b, so shouldn’t that be 256b in case people use the default ext4 inode size? Anyway, is it better to format ext4 with larger inodes (say 2048b) and set filestore_max_inline_xattr_size_other=1536, or leave it at defaults? (As I understand it, on ext4 xattrs ale limited to one block, inode size + something can spill to one different inode - maybe someone knows better). Is filestore_max_inline_xattr_size and absolute limit, or is it filestore_max_inline_xattr_size*filestore_max_inline_xattrs in reality? Does OSD do the sane thing if for some reason the xattrs do not fit? What are the performance implications of storing the xattrs in leveldb? And lastly - what size of xattrs should I really expect if all I use is RBD for OpenStack instances? (No radosgw, no cephfs, but heavy on rbd image and pool snapshots). This overhead is quite large My plan so far is to format the drives like this: mkfs.ext4 -I 2048 -b 4096 -i 524288 -E stride=32,stripe-width=256 (2048b inode, 4096b block size, one inode for 512k of space and set filestore_max_inline_xattr_size_other=1536 Does that make sense? Thanks! Jan > On 02 Jul 2015, at 12:18, Jan Schermer <jan@xxxxxxxxxxx> wrote: > > Does anyone have a known-good set of parameters for ext4? I want to try it as well but I’m a bit worried what happnes if I get it wrong. > > Thanks > > Jan > > > >> On 02 Jul 2015, at 09:40, Nick Fisk <nick@xxxxxxxxxx> wrote: >> >>> -----Original Message----- >>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of >>> Christian Balzer >>> Sent: 02 July 2015 02:23 >>> To: Ceph Users >>> Subject: Re: xattrs vs omap >>> >>> On Thu, 2 Jul 2015 00:36:18 +0000 Somnath Roy wrote: >>> >>>> It is replaced with the following config option.. >>>> >>>> // Use omap for xattrs for attrs over >>>> // filestore_max_inline_xattr_size or >>>> OPTION(filestore_max_inline_xattr_size, OPT_U32, 0) //Override >>>> OPTION(filestore_max_inline_xattr_size_xfs, OPT_U32, 65536) >>>> OPTION(filestore_max_inline_xattr_size_btrfs, OPT_U32, 2048) >>>> OPTION(filestore_max_inline_xattr_size_other, OPT_U32, 512) >>>> >>>> // for more than filestore_max_inline_xattrs attrs >>>> OPTION(filestore_max_inline_xattrs, OPT_U32, 0) //Override >>>> OPTION(filestore_max_inline_xattrs_xfs, OPT_U32, 10) >>>> OPTION(filestore_max_inline_xattrs_btrfs, OPT_U32, 10) >>>> OPTION(filestore_max_inline_xattrs_other, OPT_U32, 2) >>>> >>>> >>>> If these limits crossed, xattrs will be stored in omap.. >>>> >>> Sounds fair. >>> >>> Since I only use RBD I don't think it will ever exceed this. >> >> Possibly, see my thread about performance difference between new and old >> pools. Still not quite sure what's going on, but for some reasons some of >> the objects behind RBD's have larger xattrs which is causing really poor >> performance. >> >>> >>> Thanks, >>> >>> Chibi >>>> For ext4, you can use either filestore_max*_other or >>>> filestore_max_inline_xattrs/ filestore_max_inline_xattr_size. I any >>>> case, later two will override everything. >>>> >>>> Thanks & Regards >>>> Somnath >>>> >>>> -----Original Message----- >>>> From: Christian Balzer [mailto:chibi@xxxxxxx] >>>> Sent: Wednesday, July 01, 2015 5:26 PM >>>> To: Ceph Users >>>> Cc: Somnath Roy >>>> Subject: Re: xattrs vs omap >>>> >>>> >>>> Hello, >>>> >>>> On Wed, 1 Jul 2015 15:24:13 +0000 Somnath Roy wrote: >>>> >>>>> It doesn't matter, I think filestore_xattr_use_omap is a 'noop' and >>>>> not used in the Hammer. >>>>> >>>> Then what was this functionality replaced with, esp. considering EXT4 >>>> based OSDs? >>>> >>>> Chibi >>>>> Thanks & Regards >>>>> Somnath >>>>> >>>>> -----Original Message----- >>>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On >>>>> Behalf Of Adam Tygart Sent: Wednesday, July 01, 2015 8:20 AM >>>>> To: Ceph Users >>>>> Subject: xattrs vs omap >>>>> >>>>> Hello all, >>>>> >>>>> I've got a coworker who put "filestore_xattr_use_omap = true" in the >>>>> ceph.conf when we first started building the cluster. Now he can't >>>>> remember why. He thinks it may be a holdover from our first Ceph >>>>> cluster (running dumpling on ext4, iirc). >>>>> >>>>> In the newly built cluster, we are using XFS with 2048 byte inodes, >>>>> running Ceph 0.94.2. It currently has production data in it. >>>>> >>>>> From my reading of other threads, it looks like this is probably not >>>>> something you want set to true (at least on XFS), due to performance >>>>> implications. Is this something you can change on a running cluster? >>>>> Is it worth the hassle? >>>>> >>>>> Thanks, >>>>> Adam >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> ________________________________ >>>>> >>>>> PLEASE NOTE: The information contained in this electronic mail >>>>> message is intended only for the use of the designated recipient(s) >>>>> named above. If the reader of this message is not the intended >>>>> recipient, you are hereby notified that you have received this >>>>> message in error and that any review, dissemination, distribution, >>>>> or copying of this message is strictly prohibited. If you have >>>>> received this communication in error, please notify the sender by >>>>> telephone or e-mail (as shown above) immediately and destroy any and >>>>> all copies of this message in your possession (whether hard copies >>>>> or electronically stored copies). >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>> >>>> >>>> -- >>>> Christian Balzer Network/Systems Engineer >>>> chibi@xxxxxxx Global OnLine Japan/Fusion Communications >>>> http://www.gol.com/ >>>> >>>> ________________________________ >>>> >>>> PLEASE NOTE: The information contained in this electronic mail message >>>> is intended only for the use of the designated recipient(s) named above. >>>> If the reader of this message is not the intended recipient, you are >>>> hereby notified that you have received this message in error and that >>>> any review, dissemination, distribution, or copying of this message is >>>> strictly prohibited. If you have received this communication in error, >>>> please notify the sender by telephone or e-mail (as shown above) >>>> immediately and destroy any and all copies of this message in your >>>> possession (whether hard copies or electronically stored copies). >>>> >>>> >>> >>> >>> -- >>> Christian Balzer Network/Systems Engineer >>> chibi@xxxxxxx Global OnLine Japan/Fusion Communications >>> http://www.gol.com/ >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |