Hi Ilya, Short of building a 3.14 kernel from scratch, are there any centos/EL kernels of 3.14 but less than 3.15? Is the fix in 3.15 yet? I just installed 3.15.8. All the best, ~ Christopher Al On Sat, Aug 2, 2014 at 11:20 AM, Ilya Dryomov <ilya.dryomov at inktank.com> wrote: > On Sat, Aug 2, 2014 at 10:03 PM, Christopher O'Connell > <cjo at sendfaster.com> wrote: > > On Sat, Aug 2, 2014 at 6:27 AM, Ilya Dryomov <ilya.dryomov at inktank.com> > > wrote: > >> > >> On Sat, Aug 2, 2014 at 1:41 AM, Christopher O'Connell > >> <cjo at sendfaster.com> wrote: > >> > So I've been having a seemingly similar problem and while trying to > >> > follow > >> > the steps in this thread, things have gone very south for me. > >> > >> Show me where in this thread have I said to set tunables to optimal ;) > >> optimal (== firefly for firefly) is actually the opposite of what you > >> are going to need. > > > > > > So what should tunables be set to? Optimal? > > Ordinarily yes, but not if you are going to use older kernels. In that > case you'd want "default" or "legacy". > > > > >> > >> > >> > > >> > Kernal on OSDs and MONs: 2.6.32-431.20.3.0.1.el6.centos.plus.x86_64 #1 > >> > SMP > >> > Wed Jul 16 21:27:52 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > > >> > Kernal on RBD host: 3.2.0-61-generic #93-Ubuntu SMP Fri May 2 21:31:50 > >> > UTC > >> > 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > > >> > All are running 0.80.5 > >> > >> Is this a new firefly cluster or was it created before firely > >> (specifically before v0.78) and then upgraded? > > > > > > It was created before 0.78 and upgraded. It has also been expanded > several > > times. > > > >> > >> > >> > > >> > I updated the tunables as per this article > >> > > >> > > http://cephnotes.ksperis.com/blog/2014/01/16/set-tunables-optimal-on-ceph-crushmap > >> > > >> > Here's what's happening: > >> > > >> > 1) On the rbd client node, trying to map rbd produces > >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring > >> > /etc/ceph/mia1.client.admin.keyring map poolname > >> > > >> > rbd: add failed: (5) Input/output error > >> > > >> > Dmesg: > >> > > >> > [331172.147289] libceph: mon0 10.103.11.132:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 20042040000 > >> > [331172.154059] libceph: mon0 10.103.11.132:6789 missing required > >> > protocol > >> > features > >> > [331182.176604] libceph: mon1 10.103.11.141:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 20042040000 > >> > [331182.183535] libceph: mon1 10.103.11.141:6789 missing required > >> > protocol > >> > features > >> > [331192.192630] libceph: mon2 10.103.11.152:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 20042040000 > >> > [331192.199810] libceph: mon2 10.103.11.152:6789 missing required > >> > protocol > >> > features > >> > [331202.209324] libceph: mon0 10.103.11.132:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 20042040000 > >> > [331202.216957] libceph: mon0 10.103.11.132:6789 missing required > >> > protocol > >> > features > >> > [331212.224540] libceph: mon0 10.103.11.132:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 20042040000 > >> > [331212.232276] libceph: mon0 10.103.11.132:6789 missing required > >> > protocol > >> > features > >> > [331222.240605] libceph: mon2 10.103.11.152:6789 feature set > mismatch, > >> > my 2 > >> > < server's 20042040002, missing 20042040000 > >> > [331222.248660] libceph: mon2 10.103.11.152:6789 missing required > >> > protocol > >> > features > >> > > >> > However, running > >> > $ sudo rbd --conf /etc/ceph/mia1.conf --keyring > >> > /etc/ceph/mia1.client.admin.keyring ls > >> > poolname > >> > > >> > works fine and shows the expected pool name. > >> > > >> > 2) On the monitor where I ran the command to update the tunables, I > can > >> > no > >> > longer run the ceph console: > >> > $ ceph -c /etc/ceph/mia1.conf --keyring > >> > /etc/ceph/mia1.client.admin.keyring > >> > 2014-08-01 17:32:05.026960 7f21943d2700 0 -- 10.103.11.132:0/1030058 > >> > >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 > l=1 > >> > c=0x7f21900286a0).connect protocol feature mismatch, my fffffffff < > peer > >> > 20fffffffff missing 20000000000 > >> > 2014-08-01 17:32:05.027024 7f21943d2700 0 -- 10.103.11.132:0/1030058 > >> > >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42360 s=1 pgs=0 cs=0 > l=1 > >> > c=0x7f21900286a0).fault > >> > 2014-08-01 17:32:05.027544 7f21943d2700 0 -- 10.103.11.132:0/1030058 > >> > >> > 10.103.11.141:6789/0 pipe(0x7f2190028440 sd=3 :42361 s=1 pgs=0 cs=0 > l=1 > >> > c=0x7f21900286a0).connect protocol feature mismatch, my fffffffff < > peer > >> > 20fffffffff missing 20000000000 > >> > > >> > and it just keeps spitting out a similar message. However I *can* run > >> > the > >> > ceph console and execute basic commands (status, at the very least) > from > >> > other nodes. > >> > >> What does ceph -s from those other nodes say? Check versions of all > >> monitors with > >> > >> ceph daemon mon.<id> version > > > > > > So with some suggestions from people on IRC last night, it seems that > > several nodes didn't get librados upgraded, but still had 0.72. I'm not > > entirely sure how this happened, but I had to use yum-transaction to sort > > out the fact that python-librados went away for 0.80, and it's quite > > possible that I made a mistake and didn't upgrade these libraries. After > > manually getting all of the libraries up to date the problems went away. > > > >> > >> > >> > > >> > At this point, I'm reluctant to continue without some advice from > >> > someone > >> > else. I can certainly try upgrading the kernal on the rbd client, but > >> > I'm > >> > worried I may just make things worse. > >> > >> Upgrading the kernel won't make things worse, it's just a client. I'm > >> pretty sure we can make this work with 3.2, but if you actually plan on > >> using krbd for anything serious, I'd recommend an upgrade to 3.14. > >> > >> 3.13 will do too, if you don't plan on having any erasure pools in your > >> cluster. > > > > > > I went ahead and upgraded to 3.15 and it sorted out the problems with the > > client. > > I hate to tell you this, but due to a subtle change in kernel's low > level primitives, rbd in 3.15 is prone to deadlocks. It will be fixed > in future 3.15 stable releases, but a couple people have already run > into them and they are very reproducible under higher than average > loads, so you might want to downgrade to 3.14 and do > > ceph osd getcrushmap -o /tmp/crush > crushtool -i /tmp/crush --set-chooseleaf_vary_r 0 -o /tmp/crush.new > ceph osd setcrushmap -i /tmp/crush.new > > to make "optimal" work with 3.14. > > Thanks, > > Ilya > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140802/a8e5fc72/attachment.htm>
 |