On 03/08/2018 10:59 AM, Lazuardi Nasution wrote: > Hi Mike, > > Since I have moved from LIO to TGT, I can do full ALUA (active/active) > of multiple gateways. Of course I have to disable any write back cache > at any level (RBD cache and TGT cache). It seem to be safe to disable > exclusive lock since each RBD image is accessed only by single client > and as long as I know mostly ALUA use RR of I/O path. It might be possible if you have configured your timers correctly but I do not think anyone has figured it all out yet. Here is a simple but long example of the problem. Sorry for the length, but I want to make sure people know the risks. You have 2 iscsi target nodes and 1 iscsi initiator connected to both doing active/active over them. To make it really easy to hit, the iscsi initiator should be connected to the target with a different nic port or network than what is being used for ceph traffic. 1. Prep the data. Just clear the first sector of your iscsi disk. On the initiator system do: dd if=/dev/zero of=/dev/sdb count=1 ofile=direct 2. Kill the network/port for one of the iscsi targets ceph traffic. So for example on target node 1 pull its cable for ceph traffic if you set it up where iscsi and ceph use different physical ports. iSCSI traffic should be unaffected for this test. 3. Write some new data over the sector we just wrote in #1. This will get sent from the initiator to the target ok, but get stuck in the rbd/ceph layer since that network is down: dd if=somefile of=/dev/sdb count=1 ofile=direct ifile=direct 4. The initiator's eh timers will fire and that will fail and will the command will get failed and retired on the other path. After that dd in #3 completes run: dd if=someotherfile of=/dev/sdb count=1 ofile=direct ifile=direct This should execute quickly since it goes through the good iscsi and ceph path right away. 5. Now plug the cable back in and wait for maybe 30 seconds for the network to come back up and the stuck command to run. 6. Now do dd if=/dev/sdb of=somenewfile count=1 ifile=direct ofile=direct The data is going to be the data sent in step 3 and not the new data in step 4. To get around this issue you could try to set the krbd osd_request_timeout to a value shorter than the initiator side failover time out (for multipath-tools/open-iscsi in linux this would be fast_io_fail_tmo/replacement timeout) + the various TMF/EH but also account for the transport related timers that might short circut/bypass the TMF based EH. One problem with trying to rely on configuring that is handling all the corner cases. So you have: - Transport (nop) timer or SCSI/TMF command timer set so the fast_io_fail/replacement timer starts at N seconds and then fires at M. - It is a really bad connection so it takes N - 1 seconds to get the SCSI command from the initiator to target. - At the N second mark the iscsi connection is dropped the fast_io_fail/replacement timer is started. For the easy case, the SCSI command is sent directly to krbd and so if osd_request_timeout is less than M seconds then the command will be failed in time and we would not hit the problem above. If something happens in the target stack like the SCSI command gets stuck/queued then your osd_request_timeout value might be too short. For example, if you were using tgt/lio right now and this was a COMPARE_AND_WRITE, the READ part might take osd_request_timeout - 1 seconds, and then the write part might take osd_request_timeout -1 seconds so you need to have your fast_io_fail long enough for that type of case. For tgt a WRITE_SAME command might be N WRITEs to krbd, so you need to make sure your queue depths are set so you do not end up with something similar as the CAW but where M WRITEs get executed and take osd_request_timeout -1 seconds then M more, etc and at some point the iscsi connection is lost so the failover timer had started. Some ceph requests also might be multiple requests. Maybe an overly paranoid case, but I still worry about because I do not want to mess up anyone's data, is that a disk on the iscsi target node goes flakey. In the target we do kmalloc(GFP_KERNEL) to execute a SCSI command, and that blocks trying to write data to the flakey disk. If the disk recovers and we can eventually recover, did you account for the recovery timers in that code path when configuring the failover and krbd timers. One other case we have been debating about is if krbd/librbd is able to put the ceph request on the wire but then the iscsi connection goes down, will the ceph request always get sent to the OSD before the initiator side failover timeouts have fired and it starts using a different target node. > Best regards, > > On Mar 8, 2018 11:54 PM, "Mike Christie" <mchristi@xxxxxxxxxx > <mailto:mchristi@xxxxxxxxxx>> wrote: > > On 03/07/2018 09:24 AM, shadow_lin wrote: > > Hi Christie, > > Is it safe to use active/passive multipath with krbd with > exclusive lock > > for lio/tgt/scst/tcmu? > > No. We tried to use lio and krbd initially, but there is a issue where > IO might get stuck in the target/block layer and get executed after new > IO. So for lio, tgt and tcmu it is not safe as is right now. We could > add some code tcmu's file_example handler which can be used with krbd so > it works like the rbd one. > > I do know enough about SCST right now. > > > > Is it safe to use active/active multipath If use suse kernel with > > target_core_rbd? > > Thanks. > > > > 2018-03-07 > > > ------------------------------------------------------------------------ > > shadowlin > > > > > ------------------------------------------------------------------------ > > > > *发件人：*Mike Christie <mchristi@xxxxxxxxxx > <mailto:mchristi@xxxxxxxxxx>> > > *发送时间：*2018-03-07 03:51 > > *主题：*Re: iSCSI Multipath (Load Balancing) vs RBD > > Exclusive Lock > > *收件人：*"Lazuardi Nasution"<mrxlazuardin@xxxxxxxxx > <mailto:mrxlazuardin@xxxxxxxxx>>,"Ceph > > Users"<ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx>> > > *抄送：* > > > > On 03/06/2018 01:17 PM, Lazuardi Nasution wrote: > > > Hi, > > > > > > I want to do load balanced multipathing (multiple iSCSI > gateway/exporter > > > nodes) of iSCSI backed with RBD images. Should I disable > exclusive lock > > > feature? What if I don't disable that feature? I'm using TGT > (manual > > > way) since I get so many CPU stuck error messages when I was > using LIO. > > > > > > > You are using LIO/TGT with krbd right? > > > > You cannot or shouldn't do active/active multipathing. If you > have the > > lock enabled then it bounces between paths for each IO and > will be slow. > > If you do not have it enabled then you can end up with stale IO > > overwriting current data. > > > > > > > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |