On 03/11/2018 08:54 AM, shadow_lin wrote: > Hi Jason, > How the old target gateway is blacklisted? Is it a feature of the target > gateway(which can support active/passive multipath) should provide or is > it only by rbd excusive lock? > I think excusive lock only let one client can write to rbd at the same > time,but another client can obtain the lock later when the lock is released. For the case where we had the lock and it got taken: If IO was blocked, then unjammed and it has already passed the target level checks then the IO will be failed by the OSD due to the blacklisting. When we get IO errors from ceph indicating we are blacklisted the tcmu rbd layer will fail the IO indicating the state change and that the IO can be retried. We will also tell the target layer rbd does not have the lock anymore and to just stop the iscsi connection while we clean up the blacklisting, running commands and update our state. The case where the initiator switched on us while we were grabbing the lock is similar: After we grab the lock and before we start sending IO to the rbd/ceph layers, we will have flushed IO in various queues similar to above but a little less invasively and tested the iscsi connection to make sure it is not stuck on the network. If the path is still the good one, then the initaitor will retry the IOs on it. If the iscsi connection has been dropped, then the iscsi layer detects this and just drops IO during the flush. So, if the failover timers have fired and the multipath layer is already using a new path then the IO is not going to be running on multiple paths. > > 2018-03-11 > ------------------------------------------------------------------------ > shadowlin > > ------------------------------------------------------------------------ > > *发件人：*Jason Dillaman <jdillama@xxxxxxxxxx> > *发送时间：*2018-03-11 07:46 > *主题：*Re: Re: iSCSI Multipath (Load Balancing) vs RBD > Exclusive Lock > *收件人：*"shadow_lin"<shadow_lin@xxxxxxx> > *抄送：*"Mike Christie"<mchristi@xxxxxxxxxx>,"Lazuardi > Nasution"<mrxlazuardin@xxxxxxxxx>,"Ceph > Users"<ceph-users@xxxxxxxxxxxxxx> > > On Sat, Mar 10, 2018 at 10:11 AM, shadow_lin <shadow_lin@xxxxxxx> wrote: > > Hi Jason, > > > >>As discussed in this thread, for active/passive, upon initiator > >>failover, we used the RBD exclusive-lock feature to blacklist the old > >>"active" iSCSI target gateway so that it cannot talk w/ the Ceph > >>cluster before new writes are accepted on the new target gateway. > > > > I can get during the new active target gateway was talking to rbd the old > > active target gateway cannot write because of the RBD exclusive-lock > > But after the new target gateway done the writes,if the old target gateway > > had some blocked io during the failover,cant it then get the lock and > > overwrite the new writes? > > Negative -- it's blacklisted so it cannot talk to the cluster. > > > PS: > > Petasan say they can do active/active iscsi with patched suse kernel. > > I'll let them comment on these corner cases. > > > 2018-03-10 > > ________________________________ > > shadowlin > > > > ________________________________ > > > > 发件人：Jason Dillaman <jdillama@xxxxxxxxxx> > > 发送时间：2018-03-10 21:40 > > 主题：Re: [ceph- > users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock > > 收件人："shadow_lin"<shadow_lin@xxxxxxx> > > 抄送："Mike Christie"<mchristi@xxxxxxxxxx>,"Lazuardi > > Nasution"<mrxlazuardin@xxxxxxxxx>,"Ceph Users"<ceph-users@xxxxxxxxxxxxxx> > > > > On Sat, Mar 10, 2018 at 7:42 AM, shadow_lin <shadow_lin@xxxxxxx> wrote: > >> Hi Mike, > >> So for now only suse kernel with target_rbd_core and tcmu-runner can run > >> active/passive multipath safely? > > > > Negative, the LIO / tcmu-runner implementation documented here [1] is > > safe for active/passive. > > > >> I am a newbie to iscsi. I think the stuck io get excuted cause overwrite > >> problem can happen with both active/active and active/passive. > >> What makes the active/passive safer than active/active? > > > > As discussed in this thread, for active/passive, upon initiator > > failover, we used the RBD exclusive-lock feature to blacklist the old > > "active" iSCSI target gateway so that it cannot talk w/ the Ceph > > cluster before new writes are accepted on the new target gateway. > > > >> What mechanism should be implement to avoid the problem with > >> active/passive > >> and active/active multipath? > > > > Active/passive it solved as discussed above. For active/active, we > > don't have a solution that is known safe under all failure conditions. > > If LIO supported MCS (multiple connections per session) instead of > > just MPIO (multipath IO), the initiator would provide enough context > > to the target to detect IOs from a failover situation. > > > >> 2018-03-10 > >> ________________________________ > >> shadowlin > >> > >> ________________________________ > >> > >> 发件人：Mike Christie <mchristi@xxxxxxxxxx> > >> 发送时间：2018-03-09 00:54 > >> 主题：Re: [ceph- > users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock > >> 收件人："shadow_lin"<shadow_lin@xxxxxxx>,"Lazuardi > >> Nasution"<mrxlazuardin@xxxxxxxxx>,"Ceph Users"<ceph-users@xxxxxxxxxxxxxx> > >> 抄送： > >> > >> On 03/07/2018 09:24 AM, shadow_lin wrote: > >>> Hi Christie, > >>> Is it safe to use active/passive multipath with krbd with exclusive lock > >>> for lio/tgt/scst/tcmu? > >> > >> No. We tried to use lio and krbd initially, but there is a issue where > >> IO might get stuck in the target/block layer and get executed after new > >> IO. So for lio, tgt and tcmu it is not safe as is right now. We could > >> add some code tcmu's file_example handler which can be used with krbd so > >> it works like the rbd one. > >> > >> I do know enough about SCST right now. > >> > >> > >>> Is it safe to use active/active multipath If use suse kernel with > >>> target_core_rbd? > >>> Thanks. > >>> > >>> 2018-03-07 > >>> ------------------------------------------------------------------------ > >>> shadowlin > >>> > >>> ------------------------------------------------------------------------ > >>> > >>> *发件人：*Mike Christie <mchristi@xxxxxxxxxx> > >>> *发送时间：*2018-03-07 03:51 > >>> *主题：*Re: [ceph- > users] iSCSI Multipath (Load Balancing) vs RBD > >>> Exclusive Lock > >>> *收件人：*"Lazuardi Nasution"<mrxlazuardin@xxxxxxxxx>,"Ceph > >>> Users"<ceph-users@xxxxxxxxxxxxxx> > >>> *抄送：* > >>> > >>> On 03/06/2018 01:17 PM, Lazuardi Nasution wrote: > >>> > Hi, > >>> > > >>> > I want to do load balanced multipathing (multiple iSCSI > >>> gateway/exporter > >>> > nodes) of iSCSI backed with RBD images. Should I disable exclusive > >>> lock > >>> > feature? What if I don't disable that feature? I'm using TGT > >>> (manual > >>> > way) since I get so many CPU stuck error messages when I was using > >>> LIO. > >>> > > >>> > >>> You are using LIO/TGT with krbd right? > >>> > >>> You cannot or shouldn't do active/active multipathing. If you have > >>> the > >>> lock enabled then it bounces between paths for each IO and will be > >>> slow. > >>> If you do not have it enabled then you can end up with stale IO > >>> overwriting current data. > >>> > >>> > >>> > >>> > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > [1] http://docs.ceph.com/docs/master/rbd/iscsi-overview/ > > > > -- > > Jason > > > > -- > Jason > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |