Hi Greg, > On 24 Nov 2014, at 22:01, Gregory Farnum <greg@xxxxxxxxxxx> wrote: > > On Thu, Nov 20, 2014 at 9:08 AM, Dan van der Ster > <daniel.vanderster@xxxxxxx> wrote: >> Hi all, >> What is compatibility/incompatibility of dumpling clients to talk to firefly >> and giant clusters? > > We sadly don't have a good matrix about this yet, but in general you > should assume that anything which changed the way the data is > physically placed on the cluster will prevent them from communicating; > if you don't enable those features then they should remain compatible. It would be good to have such a compat matrix, as I was confused, probably others are confused, and if I’m not wrong, even you are confused ... see below. > In particular > >> I know that tunables=firefly will prevent dumpling >> clients from talking to a firefly cluster, but how about the existence or >> not of erasure pools? > > As you mention, updating the tunables will prevent old clients from > accessing them (although that shouldn't be the case in future now that > they're all set by the crush map for later interpretation). Erasure > pools are a special case (precisely because people had issues with > them) and you should be able to communicate with a cluster that has EC > pools while using old clients That’s what we’d hoped, but alas we get the same error mentioned here: http://tracker.ceph.com/issues/8178 In our case (0.67.11 clients talking to the latest firefly gitbuilder build) we get: protocol feature mismatch, my 407ffffffff < peer 417ffffffff missing 1000000000 By adding an EC pool, we lose connectivity for dumpling clients to even the replicated pools. The good news is that when we remove the EC pool, the 1000000000 feature bit is removed so dumpling clients can connect again. But nevertheless it leaves open the possibility of accidentally breaking the users’ access. So this means we should upgrade all clients (quite a few qemu-kvm processes) to the firefly librbd before upgrading the cluster to firefly, to be 100% safe. > — but, no: > >> Can a dumpling client talk to a Firefly/Giant erasure >> pool if the tunables are still dumping? > > Definitely not. EC pools use a slightly different CRUSH algorithm than > the old clients could, among many other things. Oops, duh! I’d convinced myself that it might be possible, since the EC work is coordinated by the primary OSD anyway. But I forgot that things like failover to other OSDs could never work without client knowledge of EC rules. Cheers, Dan _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |