FWIW, We still had a couple of example cbt configuration files in the
github repo that used -n 64k until Jens pointed it out earlier this
week. That's now been fixed, so that should at least help.
Unfortunately there' still some old threads on the mailing list that
list it as a potential performance tuning:
http://www.spinics.net/lists/ceph-devel/msg11236.html
Hopefully future readers of the list will see this thread first before
reading that old one.
Mark
On 02/18/2016 08:10 AM, Dan van der Ster wrote:
Hi,
Thanks for linking to a current update on this problem [1] [2]. I
really hope that new Ceph installations aren't still following that
old advice... it's been known to be a problem for around a year and a
half [3].
That said, the "-n size=64k" wisdom was really prevalent a few years
ago, and I wonder how many old clusters are at risk today. I manage a
sufficiently large enough number of affected OSDs that I'll be willing
to try all other possibilities before reformatting them [4]. Today
they're rock solid stable on EL6 (with hammer), but the jewel release
is getting closer and that's when we'll need to upgrade to EL7. (I've
already upgraded one host to 7 and haven't seen any problems yet, but
that one sample doesn't offer much comfort for the rest.) Anyway, it's
great to hear that there's a patch in the works... Dave deserves
infinite thanks if this gets resolved.
Cheers, Dan
[1] http://tracker.ceph.com/issues/6301
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1278992
[3] https://github.com/redhat-cip/puppet-ceph/commit/b9407efd4a8a25d452e493fb48ea048e4d36e070
[4] https://access.redhat.com/solutions/1597523
On Thu, Feb 18, 2016 at 1:14 PM, Jens Rosenboom <j.rosenboom@xxxxxxxx> wrote:
Various people have noticed performance problems and sporadic kernel
log messages like
kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)
with their Ceph clusters. We have seen this in one of our clusters
ourselves, but not been able to reproduce it in a lab environment
until recently. While trying to setup a benchmark for comparing the
effect of varying the bucket shard count, I suddenly started seeing
the same issues again and they seemed to be reproducible even with the
latest upstream kernel.
The test setup comprised 8 nodes with 2 SSDs as OSDs each. The
messages started to appear after writing 16kb sized objects with
cosbench using 32 workers for about 2 hours and soon after that the
OSDs started dying because of suicide timeout.
So we went ahead and tried running a kernel patched with [1], but this
had only partial success, so I posted these results to the XFS mailing
list. The response by Dave Chinner led to some important result:
Creating the file system with the option "-n size=64k" was the
culprit. Repeating the tests with sizes <=16k did not show any issues
and the performance for this particular test even turned out to be
better with simply letting the directory block size stay at the
default value of 4k.
In case you are seeing similar issues, you may want to check the
directory block size of your file system, you can use xfs_info for
that. The bad news is that the parameter cannot be changed for an
existing file system, so you will need to reformat everything.
And the morale is: Do not blindly trust configuration settings to be
helpful, even if their use seems to be widely spread and is looking
reasonable at first.
[1] http://oss.sgi.com/pipermail/xfs/2016-January/046308.html
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
