Hi David,
I'm surprised your message didn't get any echo yet. I guess it depends on how many files your OSDs get to store on filesystem which depends essentialy on use cases.
We're having similar issues with a 144 osd cluster running 2 pools. Each one holds 100 M objects.One is replication x3 (256 PGs) and the other is EC k=5, m=4 (512 PGs).
That's 300 M + 900 M = 1.2 B files stored on XFS filesystem.
We're observing that our PGs subfolders only holds around 120 files each when they should holds around 320 (we're using default split / merge values).
All objetcs were created when cluster was running Hammer. We're now running Jewel (RHCS 2.0 actually).
We ran some tests on a Jewel backup infrastructure. Split happens at around 320 files per directory, as expected.
We have no idea why we're not seeing 320 files per PG subfolder on our production cluster pools.
Everything we read suggests to raise the filestore_merge_threshold and filestore_split_multiple values to 40 / 8 :
https://www.redhat.com/en/files/resources/en-rhst-cephstorage-supermicro-INC0270868_v2_0715.pdf
https://bugzilla.redhat.com/show_bug.cgi?id=1219974
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041179.html
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-September/012987.html
We now need to merge directories (when you need to split apparently :-)
We will do so, by increasing the filestore_merge_threshold in 10 units steps until maybe 120 to lower it back to 40.
Between each steps we'll run 'rados bench' (in cleanup mode) on both pools to generate enough deletes operations to trigger merges operations on each PGs.
By running the 'rados bench' at night our clients won't be much impacted by blocked requests.
Running this on you cluster would also provoke split when rados bench writes to the pools.
Also, note that you can set merge and split values to a specific OSD in ceph.conf ([osd.123]) so you can see how the OSD reorganizes the PGs tree when running a 'rados bench'.
Regarding the OSDs flapping, does this happen when scrubbing ? You may hit the Jewel scrubbing bug Sage reported like 3 weeks ago (look for 'stalls caused by scrub on jewel').
It's fixed in 10.2.4 and waiting for QA to make it to RHCS >= 2.0
We are impacted by this bug because we have a lot of objects (200k) per PGs with, I think, bad split / merge values. Lowering vfs_cache_pressure to 1 might also help to avoid the flapping.
Regards,
Frederic Nass,
Université de Lorraine.
----- Le 27 Sep 16, à 0:42, David Turner <david.turner@xxxxxxxxxxxxxxxx> a écrit :
We are running on Hammer 0.94.7 and have had very bad experiences with PG folders splitting a sub-directory further. OSDs being marked out, hundreds of blocked requests, etc. We have modified our settings and watched the behavior match the ceph documentation for splitting, but right now the subfolders are splitting outside of what the documentation says they should.
filestore_split_multiple * abs(filestore_merge_threshold) * 16
Our filestore_merge_threshold is set to 40. When we had our filestore_split_multiple set to 8, we were splitting subfolders when a subfolder had (8 * 40 * 16 = ) 5120 objects in the directory. In a different cluster we had to push that back again with elevated settings and the subfolders split when they had (16 * 40 * 16 = ) 10240 objects.
We have another cluster that we're working with that is splitting at a value that seems to be a hardcoded maximum. The settings are (32 * 40 * 16 = ) 20480 objects before it should split, but it seems to be splitting subfolders at 12800 objects.
Normally I would expect this number to be a power of 2, but we recently found another hardcoded maximum of the object map only allowing RBD's with a maximum 256,000,000 objects in them. The 12800 matches that as being a base 2 followed by a set of zero's to be the hardcoded maximum.
Has anyone else encountered what seems to be a hardcoded maximum here? Are we missing a setting elsewhere that is capping us, or diminishing our value? Much more to the point, though, is there any way to mitigate how painful it is to split subfolders in PGs? So far it seems like the only way we can do it is to push up the setting to later drop it back down during a week that we plan to have our cluster plagued with blocked requests all while cranking our osd_heartbeat_grace so that we don't have flapping osds.
A little more about our setup is that we have 32x 4TB HGST drives with 4x 200GB Intel DC3710 journals (8 drives per journal), dual hyper-threaded octa-core Xeon (32 virtual cores), 192GB memory, 10Gb redundant network... per storage node.
David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: 385.224.2943
If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
