On 03/14/2018 01:06 PM, Maxim Patlasov wrote: > On Sun, Mar 11, 2018 at 5:10 PM, Mike Christie <mchristi@xxxxxxxxxx > <mailto:mchristi@xxxxxxxxxx>> wrote: > > On 03/11/2018 08:54 AM, shadow_lin wrote: > > Hi Jason, > > How the old target gateway is blacklisted? Is it a feature of the target > > gateway(which can support active/passive multipath) should provide or is > > it only by rbd excusive lock? > > I think excusive lock only let one client can write to rbd at the same > > time,but another client can obtain the lock later when the lock is released. > > For the case where we had the lock and it got taken: > > If IO was blocked, then unjammed and it has already passed the target > level checks then the IO will be failed by the OSD due to the > blacklisting. When we get IO errors from ceph indicating we are > blacklisted the tcmu rbd layer will fail the IO indicating the state > change and that the IO can be retried. We will also tell the target > layer rbd does not have the lock anymore and to just stop the iscsi > connection while we clean up the blacklisting, running commands and > update our state. > > > Mike, can you please give more details on how you tell the target layer > rbd does not have the lock and to stop iscsi connection. Which > tcmu-runner/kernel-target functions are used for that? For this case it would be tcmu_rbd_handle_blacklisted_cmd. Note for failback type of test, we might not hit that error if the initiator does a RTPG before it sends IO. In that case we would see tcmu_update_dev_lock_state get run first and the iscsi connection would not be dropped. > > In fact, I performed an experiment with three stale write requests stuck > on blacklisted gateway, and one of them managed to overwrite newer data. What is the test exactly? What OS for the initiator? What kernel were you using and are you using the upstream tools/libs or the RHCS ones? Can you run your tests and send the initiator side kernel logs and on the iscsi targets send the /var/log/tcmu-runner.log with debugging in enabled. To do that open /etc/tcmu/tcmu.conf on the iscsi target nodes and set log_level = 5 If that is too much output drop it to level 4. > I followed all instructions from > http://docs.ceph.com/docs/master/rbd/iscsi-target-cli-manual-install/ > and http://docs.ceph.com/docs/master/rbd/iscsi-target-cli/, so I'm > interested what I'm missing... You used the initiator settings in http://docs.ceph.com/docs/master/rbd/iscsi-initiators/ too right? > > Thanks, > Maxim > > Thanks, > Maxim > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |