rbd kernel client fencing

[Chaofan Yu <chaofanyu@xxxxxxxxxxx>]

Hi list,

  I wonder someone can help with rbd kernel client fencing (aimed to avoid simultaneously rbd map on different hosts).

I know the exclusive rbd image feature is added later to avoid manual rbd lock CLIs. But want to know previous blacklist solution.

The official workflow I’ve got is listed below (without exclusive rbd feature) :

 - identify old rbd lock holder (rbd lock list <img>)
 - blacklist old owner (ceph osd blacklist add <addr>)
 - break old rbd lock (rbd lock remove <img> <lockid> <addr>)
 - lock rbd image on new host (rbd lock add <img> <lockid>)
 - map rbd image on new host

The blacklisted entry identified by entity_addr_t (ip, port, nonce).

However as far as I know, ceph kernel client will do socket reconnection if connection failed. So I wonder in this scenario it won’t work:

1. old client network down for a while

2. perform below steps on new host to achieve failover

- identify old rbd lock holder (rbd lock list <img>)

 - blacklist old owner (ceph osd blacklist add <addr>)
 - break old rbd lock (rbd lock remove <img> <lockid> <addr>)
 - lock rbd image on new host (rbd lock add <img> <lockid>)
 - map rbd image on new host

3. old client network come back and reconnect to osds with new created socket client, i.e. new (ip, port,nonce) turple

as a result both new and old client can write to same rbd image, which might potentially cause the data corruption.

So does this mean if kernel client does not support exclusive-lock image feature, fencing is not possible ?

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com