On Mon, 2015-12-21 at 01:18 -0800, Nicholas A. Bellinger wrote: > On Thu, 2015-12-17 at 11:37 -0500, Vedavyas Duggirala wrote: > > Hi, > > > > Thanks for your response, > > > > > Seeing these occasionally on a loaded system is not unusual. Seeing > > > them ongoing at multiple times a second as in your logs below, means > > > your backend is having trouble completing I/Os within that 5 second > > > latency requirement for ESX iSCSI. > > > > > > As mentioned in the thread above, you can try reducing the > > > default_cmdsn_depth or NodeACL cmdsn_depth on the target side, to reduce > > > the number of active I/Os each initiator can keep in flight in parallel. > > > > > > Since your using such a larger number of ESX hosts (~20) connected to a > > > single target, the default_cmdsn_depth = 64 value is certainly too high > > > for you. > > > > I have checked the backend storage configuration. The machine is > > serving luns off SSDs. I verified our stats, > > there was not a whole lot of IO happening at that time (20-100 IOPS). > > In fact, other that ESX health checks and heartbeats, > > nothing was writing to the disk. > > > > Disabling ATS for heartbeats on that lun, seems to help stability. It > > has been running fine without issues for three weeks, > > as opposed to a hang every week. > > > > I have the same issue happen on a smaller 8 node ESX cluster. There > > too disabling ATS for heartbeat helps. > > > > These results are without reducing default_cmdsn_depth. I will try > > that out in the next maintenance window. > > Disabling ATS means VMFS metadata updates are done using coarse grained > LUN locking with SCSI reservations, instead of fine grained VMFS extent > locks with ATS. > > That is, the amount of I/O bandwidth with ATS enabled is considerably > larger with multiple ESX hosts, because multiple ESX hosts can WRITE in > parallel to a single VMFS LUN, with ESX hosts owning different VMFS > extent. > > So the issue with your RBD backend configuration resulting in >= 5 > second I/O latencies with fined grained ATS locking doing lots of single > sector READ/WRITEs for VMFS extent ownership, is now effectively being > masked with coarse grained VMFS locking. > > As mentioned, the resulting RCU stalls you've observed with v3.18.y > still seem to be related to RBD not (ever..?) completing outstanding > bios back to IBLOCK. > Also WRT to ATS, there has been one recent upstream bug-fix yet to make it back to v3.18.y stable code: target: Fix race for SCF_COMPARE_AND_WRITE_POST checking https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/target?id=057085e522f8bf94c2e691a5b76880f68060f8ba I don't think you're actively triggering this specific bug, but it certainly wouldn't hurt to have it applied. --nab -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
