Try increasing /proc/sys/fs/aio-max-nr and see if that helps? It's teh io_setup syscall that is failing. sage On Tue, 22 Jul 2014, Dane Elwell wrote: > Hi, > > We recently tried to switch to kernel 3.13 (from 3.5) using the Ubuntu > Precise backported kernels (e.g. package linux-generic-lts-trusty), as > we're having problems with memory fragmentation with the 3.5 kernel > (3.8 was worse!). After upgrading to 3.13, some - but not all - OSDs > fail to start with the following error: > > http://p.rig.gr/view/raw/a512debf > > I can confirm that the journal symlink is present and is pointing to > the correct device. I've done an xfs_repair on the data disk FWIW, but > no joy there either. Thanks to 'oomkiller' on IRC I manually flushed > the OSD journal (no joy) and recreated the journal (no joy). Still > giving me errors (http://p.rig.gr/view/raw/8a036747). > > It was after this we started playing around with starting the OSDs > manually, and it seems that we can only get around 15-18 OSDs (out of > 32) started before they start giving us these errors. If we stop all > the OSDs and try to start one of the previously failing ones, it > starts fine. Because of this, I'm suspecting there is some kind of > limit imposed upon the 3.13 kernel that wasn't in place in 3.8 or 3.5, > though I thought I'd ask here first before trying to get an answer out > of the Ubuntu folk. > > Reverting to kernel 3.5 resolved all the issues and all OSDs start > normally - so this is what we've had to do for the time being. > > ceph.conf: http://p.rig.gr/view/raw/96d5ac73 > > Ceph was installed using ceph-deploy initially, but upgraded using > apt-get. We're running Ubuntu Precise (12.04) and the problem kernel > is the backported Trusty LTS kernel (3.13 - linux-generic-lts-trusty). > I'll admit that we went 3.5 -> 3.13 at the same time as we upgraded > 0.80.1 -> 0.80.4, which in hindsight was probably a silly move. > > Thanks > > Dane > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
 |