Nicholas, Jörn Thanks for your response on this. I apologize for the delay getting back to the list with my results. I am running iblock, in my specific case. While I see better results with fileio, I'd like to stick with iblock if possible. The NAS I am comparing against, which is performing surprisingly well, is also set up to use iblock. I had default_cmdsn_depth still at 16 (as did the NAS I am comparing against), but I did not see an improvement in my robocopy results in Windows 7 when I increased the number of outstanding commands to 64 or 128. That may be because the data transfer is using larger blocks (I believe it is configured for ~250K), and requiring fewer commands to transfer a given amount of data. Additionally, I had already had emulate_write_cache turned on. I did try with TCP_NODELAY enabled on the windows side. The change of performance was insignificant (maybe a few percent faster, but easily within the noise of my measurement). I spent a little time trying to characterize difference between my system and the comparison NAS. One thing I looked at, which may or may not indicate anything, was the Queue Depth in info for the LUN under configFS on both my system and the comparison system. I put together a little script to cat the contents of info to a configFS entry every second or so while the copy was commencing. On my system, I'd see the size of the "left" queue depth drop to around 103, at it's lowest point when writing from a Windows 7 system. It gets down to 81 when running from a WIndows 2008 system. I have a Max of 128 entries listed. On the comparison NAS system, the Max number of queue entries is 32, and the "left" queue depth never goes below 31. While there is little that probably can be explicitly derived from this, it would suggest to me that the comparison NAS system is able to process the writes coming in from the iSCSI target interface much faster than I am doing in my system, as the number of queue entries does not seem to stack up. Both the NAS and my system are using software RAID5 arrays, so I am now wondering if there is some interaction between my iSCSI setup and my mdraid setup that does not exist on the other NAS. If incoming write requests over iSCSI were being mapped to more physical disk accesses in my system, compared to the other NAS, I could certainly see this as a cause for the lower results. Does anyone have any thoughts on this? Thanks, Scott On Mon, Aug 19, 2013 at 1:36 PM, Nicholas A. Bellinger <nab@xxxxxxxxxxxxxxx> wrote: > Hi Scott, > > On Sun, 2013-08-18 at 22:43 -0600, Scott Hallowell wrote: >> I have been looking into a performance concern with the iSCSI target >> as compared to CIFS running on the same server. The expectation was >> that iSCSI should perform somewhat similarly to Samba. The test >> environments are Window 7 & Windows 2008 Initiators connecting to a >> target running on a Debian wheezy release (a 3.2.46 kernel). The test >> is a file copy from Windows to the Linux server. The source volume is >> a software Raid 0 running on Windows. The destination is an iSCSI LUN >> on a software Raid-5 array with 4 disks (2tb WD Reds). >> >> The write (from Windows to the CIFS/iSCSI volumes) is considerably >> slower (about half the rate) than the CIFS write. >> >> Specifically, I see 90+ MB/s writes with Samba on both the Windows 7 >> and WIndows 2008 machines (using robocopy and 5.7GB of data spread >> unevenly across about 30 files). >> >> Performing the same tests with iSCSI and what I believe to be the >> 2.0.8 version of the Windows iSCSI initiator, I am getting closer to >> 40-45MB/s. on Windows 7 and 65 MB/s on Windows 2008. >> >> To test the theory the issue was a Windows issue, I connected the >> Windows 7 initiator to a commercial SAN and repeated the same tests. >> I got results of around 87MB/s. The commercial SAN was configured >> similarly to my Linux server - Raid 5, 4 2tb WD Red disks, and has >> similar hardware (intel Atom processor, e1000e NICs, although less >> physical ram: 1GB vs 2GB). >> >> The results are fairly repeatable (+/- a couple of MB/s) and, at least >> with Windows 7, do not appear to suggest a specific issue with the >> Windows side of the equation. The CIFS performance would suggest (to >> me, at least) that there is not a basic networking problem, either. >> >> I've tried a number of different things in an attempt to affect the >> iSCSI performance: changing the disk scheduling (CFQ, Deadline, and >> noop), confirming write caching is on with hdparm, tweaking vm >> parameters in the kernel, tweaking TCP and adapter parameters (both in >> Linux and Windows), etc. Interestingly, the performance numbers do >> not seem to change by more than +/- 10%, aggregate, with enough >> variability in the results that I'd suggest the changes are >> essentially in the noise. I will note that I have not gone to >> 9000byte MTUs, but that seems irrelevant as the commercial SAN I >> compared against wasn't using that, either. >> >> I attempted to look at wireshark traces to identify any obvious >> patterns that might be had from the traffic. Unfortunately, the >> amount of data required before I was able to start seeing repeatable >> differences in the aggregate rates (>400MB of file transfers) combined >> with offloading and the significant amount of caching in Windows has >> made such an analysis a bit tricky. >> >> It seems to me that there is something mis-configured in a very basic >> way which is significantly limits the performance by a far more >> significant extreme than can be explained by simple tuning, but I am >> at a loss to understand what it is. >> >> I am hoping that this has a ring of familiarity with someone who can >> give me some pointers on where I need to focus my attention. >> > > I recommend pursuing a few different things.. > > First, you'll want to bump the default_cmdsn_depth from 16 to 64. This > is the maximum number of commands allowed in flight (per session) at any > given time. This can be changed with 'set attrib default_cmdsn_depth > 64' from within targetcli TPG context, or this can be changed on a per > NodeACL context basis if your not using TPG demo mode. > > The second is to try with write cache (buffered writes) enabled. By > default both IBLOCK and FILEIO are running without write cache enabled, > to favor strict data integrity during target power loss over backend > performance. IBLOCK itself can set the WriteCacheEnabled=1 bit via > emulate_write_cache, but all WRITEs are still going to be submitted + > completed to the underlying storage (which may also have a cache of it's > own) before acknowledgement. > > For FILEIO however, there is a buffered mode, which puts all WRITEs into > the buffer cache and acknowledges immediately, and let's VFS writeback > occur based upon /proc/sys/vm/dirty_[writeback,expire]_centisecs. This > can be enabled during FILEIO created in targetcli by setting > 'buffered=true', which depending upon your version of the target will > automatically set 'emulate_write_cache=1'. > > You can verify that buffered mode is enabled in configfs with the 'Mode' > output, which depending on your kernel version should look something > like: > > # cat /sys/kernel/config/target/core/fileio_0/test/info > Status: DEACTIVATED Execute/Max Queue Depth: 0/0 SectorSize: 512 MaxSectors: 1024 > TCM FILEIO ID: 0 File: /tmp/test Size: 1073741824 Mode: Buffered > > The third thing is to enable TCP_NODELAY on the windows side, which does > not enable this bit by default. This needs to be enabled on a per > interface basis in the registry, and should be easy enough to find on > google. > > Please let the list know your results. > > --nab > -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
