Gennadiy Nerubayev, on 07/26/2010 09:00 PM wrote:
On Mon, Jul 26, 2010 at 8:22 AM, Vladislav Bolkhovitin<vst@xxxxxxxx> wrote:
Gennadiy Nerubayev, on 07/24/2010 12:51 AM wrote:
The real life problem we can see in an active-active DRBD-setup. In
this
configuration 2 nodes act as a single SCST-powered SCSI device and they
both
run DRBD to keep their backstorage in-sync. The initiator uses them as
a
single multipath device in an active-active round-robin load-balancing
configuration, i.e. sends requests to both nodes in parallel, then DRBD
takes care to replicate the requests to the other node.
The problem is that sometimes DRBD complies about concurrent local
writes, like:
kernel: drbd0: scsi_tgt0[12503] Concurrent local write detected!
[DISCARD
L] new: 144072784s +8192; pending: 144072784s +8192
This message means that DRBD detected that both nodes received
overlapping writes on the same block(s) and DRBD can't figure out which
one
to store. This is possible only if the initiator sent the second write
request before the first one completed.
The topic of the discussion could well explain the cause of that. But,
unfortunately, people who reported it forgot to note which OS they run
on
the initiator, i.e. I can't say for sure it's Linux.
Sorry for the late chime in, but here's some more information of
potential interest as I've previously inquired about this to the drbd
mailing list:
1. It only happens when using blockio mode in IET or SCST. Fileio,
nv_cache, and write_through do not generate the warnings.
Some explanations for those who not familiar with the terminology:
- "Fileio" means Linux IO stack on the target receives IO via
vfs_readv()/vfs_writev()
- "NV_CACHE" means all the cache synchronization requests
(SYNCHRONIZE_CACHE, FUA) from the initiator are ignored
- "WRITE_THROUGH" means write through, i.e. the corresponding backend
file
for the device open with O_SYNC flag.
2. It happens on active/passive drbd clusters (on the active node
obviously), NOT active/active. In fact, I've found that doing round
robin on active/active is a Bad Idea (tm) even with a clustered
filesystem, until at least the target software is able to synchronize
the command state of either node.
3. Linux and ESX initiators can generate the warning, but I've so far
only been able to reliably reproduce it using a Windows initiator and
sqlio or iometer benchmarks. I'll be trying again using iometer when I
have the time.
4. It only happens using a random write io workload (any block size),
with initiator threads>1, OR initiator queue depth>1. The higher
either of those is, the more spammy the warnings become.
5. The transport does not matter (reproduced with iSCSI and SRP)
6. If DRBD is disconnected (primary/unknown), the warnings are not
generated. As soon as it's reconnected (primary/secondary), the
warnings will reappear.
It would be great if you prove or disprove our suspicions that Linux can
produce several write requests for the same blocks simultaneously. To be
sure we need:
1. The initiator is Linux. Windows and ESX are not needed for this
particular case.
2. If you are able to reproduce it, we will need full description of
which
application used on the initiator to generate the load and in which mode.
Target and DRBD configuration doesn't matter, you can use any.
I just tried, and this particular DRBD warning is not reproducible
with io (iometer) coming from a Linux initiator (2.6.30.10) The same
iometer parameters were used as on windows, and both the base device
as well as filesystem (ext3) were tested, both negative. I'll try a
few more tests, but it seems that this is a nonissue with a Linux
initiator.
OK, but to be completely sure, can you check also with other load
generators, than IOmeter, please? IOmeter on Linux is a lot less effective
than on Windows, because it uses sync IO, while we need big multi-IO load to
trigger the problem we are discussing, if it exists. Plus, to catch it we
need an FS on the initiator side, not using raw devices. So, something like
fio over files on FS or diskbench should be more appropriate. Please don't
use direct IO to avoid the bug Dave Chinner pointed us out.
I tried both fio and dbench, with the same results. With fio in
particular, I think I used pretty much every possible combination of
engines, directio, and sync settings with 8 threads, 32 queue depth
and random write workload.
Also, you mentioned above about that Linux can generate the warning. Can you
recall on which configuration, including the kernel version, the load
application and its configuration, you have seen it?
Sorry, after double checking, it's only ESX and Windows that generate
them. The majority of the ESX virtuals in question are Windows, though
I can see some indications of ESX servers that have Linux-only
virtuals generating one here and there. It's somewhat difficult to
tell historically, and I probably would not be able to determine what
those virtuals were running at the time.
OK, I see. A negative result is also a result. Now we know that Linux
(in contrast to VMware and Windows) works well in this area.
Thank you!
Vlad
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
