Hi,
We recently tried to switch to kernel 3.13 (from 3.5) using the Ubuntu
Precise backported kernels (e.g. package linux-generic-lts-trusty), as
we're having problems with memory fragmentation with the 3.5 kernel
(3.8 was worse!). After upgrading to 3.13, some - but not all - OSDs
fail to start with the following error:
http://p.rig.gr/view/raw/a512debf
I can confirm that the journal symlink is present and is pointing to
the correct device. I've done an xfs_repair on the data disk FWIW, but
no joy there either. Thanks to 'oomkiller' on IRC I manually flushed
the OSD journal (no joy) and recreated the journal (no joy). Still
giving me errors (http://p.rig.gr/view/raw/8a036747).
It was after this we started playing around with starting the OSDs
manually, and it seems that we can only get around 15-18 OSDs (out of
32) started before they start giving us these errors. If we stop all
the OSDs and try to start one of the previously failing ones, it
starts fine. Because of this, I'm suspecting there is some kind of
limit imposed upon the 3.13 kernel that wasn't in place in 3.8 or 3.5,
though I thought I'd ask here first before trying to get an answer out
of the Ubuntu folk.
Reverting to kernel 3.5 resolved all the issues and all OSDs start
normally - so this is what we've had to do for the time being.
ceph.conf: http://p.rig.gr/view/raw/96d5ac73
Ceph was installed using ceph-deploy initially, but upgraded using
apt-get. We're running Ubuntu Precise (12.04) and the problem kernel
is the backported Trusty LTS kernel (3.13 - linux-generic-lts-trusty).
I'll admit that we went 3.5 -> 3.13 at the same time as we upgraded
0.80.1 -> 0.80.4, which in hindsight was probably a silly move.
Thanks
Dane
