On Fri, Oct 11, 2013 at 5:28 PM, Nicholas A. Bellinger
<nab@xxxxxxxxxxxxxxx> wrote:
> On Thu, 2013-10-10 at 18:33 -0500, Xianghua Xiao wrote:
>>
>
> <SNIP>
>
>> I switched to fileio and now READ is at full speed, i.e. wirespeed of
>> 10Gbps, plus all cores are being used(though still only core0 takes
>> the MSI-X interrupts). However, WRITE slows down dramatically from
>> 415MB/s to 150MB/s, that really puzzles me.
>>
>
> From the Mode: O_DSYNC output below, the FILEIO device is *not* running
> in buffered I/O mode for WRITEs.
>
> Also, can you confirm using iostat -xm that the READs are coming from
> the backend device, and not from the buffer cache itself..?
>
> FYI, iostat -xm output also shosw the size of the I/Os being submitted
> to both the MD RAID devices, as well as to the underlying mpt2sas SCSI
> devices. This would be useful to see if the size of I/Os is different
> between backend drivers.
>
>>
>> I had the same setting, including :
>> echo 1 > /sys/kernel/config/target/core/fileio_$i/my_fileio
>> $i/attrib/emulate_write_cache for all endpoints.
>> At WRITE all cores are equally involved, they're just not fully loaded
>> at all.
>>
>
> Setting emulate_write_cache is the per device attribute for exposing the
> SCSI WriteCacheEnabled bit to the initiator, and does not directly
> enable buffered I/O operation for FILEIO backends.
>
>>
>> Here is the output of fileio:
>>
>> Rounding down aligned max_sectors from 32767 to 32760
>> Status: DEACTIVATED Max Queue Depth: 0 SectorSize: 512 HwMaxSectors: 32760
>> TCM FILEIO ID: 0 File: /dev/md12 Size: 0 Mode: O_DSYNC
>>
>> I'm still trying to get the optimal throughput for sequential
>> READ/WRITE at the moment. Reliability is secondary as I'm trying to
>> stress the system performance for now. Is there anything else that I
>> can tune for FILEIO (or IBLOCK)?
>
> Buffered FILEIO can be enabled with targetcli + rtslib, but not during
> setup with the legacy lio-utils code.
>
> The easiest way to do this is modifying /etc/target/tcm_start.sh after
> the configuration has been saved, and then including the
> ',fd_buffered_io=1' parameter for each FILEIO device like so:
>
> tcm_node --establishdev fileio_0/test fd_dev_name=/tmp/test,fd_dev_size=2147483648,fd_buffered_io=1
>
> From there restart the target, and verify the same output from
> /sys/kernel/config/target/core/$FILEIO_HBA/$FILEIO_DEV/info as above, which
> should now read:
>
> Status: DEACTIVATED Max Queue Depth: 0 SectorSize: 512 HwMaxSectors: 32760
> TCM FILEIO ID: 0 File: /dev/md12 Size: 0 Mode: Buffered-WCE
>
> --nab
>
After I added fd_buffered_io I'm getting very similar results as
SCST's fileio mode. There are still more to optimize but I'm happy
that I can see similar results now.
Thanks so much for all your helps!
Xiao
P.S:
Below is the output of 'iostat -xm'
avg-cpu: %user %nice %system %iowait %steal %idle
0.11 0.00 8.30 0.84 0.00 90.74
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 126.07 301.20 8.57 9.54 0.53 1.38
215.37 1.39 24.28 16.31 31.44 2.11 3.82
sdd 101.02 279.02 10.36 9.26 0.43 1.27
178.04 0.37 15.89 11.84 20.42 1.59 3.12
sdb 120.46 304.83 8.16 10.21 0.50 1.40
211.68 1.05 22.39 19.15 24.98 2.05 3.76
sdc 118.04 303.83 7.83 10.17 0.49 1.39
214.31 0.68 18.48 17.65 19.12 1.97 3.55
sdg 103.29 279.23 7.61 7.63 0.43 1.29
231.83 0.96 27.60 22.61 32.57 2.22 3.38
sdf 102.52 280.62 7.35 7.84 0.43 1.30
232.81 1.28 26.69 24.56 28.69 2.25 3.42
sde 105.69 279.50 6.39 7.50 0.44 1.29
255.00 0.62 19.53 21.54 17.83 2.36 3.28
sdh 104.62 279.54 7.09 7.62 0.44 1.29
240.62 0.38 16.00 19.97 12.30 2.15 3.16
mtdblock0 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 1.00 1.00 0.00 1.00 0.00
mtdblock1 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 1.83 1.83 0.00 1.83 0.01
mtdblock2 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 0.67 0.67 0.00 0.67 0.00
mtdblock3 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 0.83 0.83 0.00 0.83 0.00
mtdblock4 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 1.17 1.17 0.00 1.17 0.00
mtdblock5 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 2.83 2.83 0.00 2.83 0.01
mmcblk0 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 9.76 9.76 0.00 9.76 0.03
mmcblk0rpmb 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.09 2715.33 2715.33 0.00 2715.33 9.10
mmcblk0boot1 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 11.52 11.52 0.00 11.52 0.04
mmcblk0boot0 0.00 0.00 0.03 0.00 0.00 0.00
8.00 0.00 10.72 10.72 0.00 10.72 0.04
md0 0.00 0.00 36.03 84.17 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 36.03 85.24 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 36.02 84.30 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md3 0.00 0.00 36.02 84.48 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md4 0.00 0.00 59.24 133.29 0.23 0.52
7.97 0.00 0.00 0.00 0.00 0.00 0.00
md5 0.00 0.00 36.02 83.58 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md6 0.00 0.00 36.05 84.48 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md7 0.00 0.00 36.01 80.28 0.14 0.31
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md8 0.00 0.00 36.01 85.13 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md9 0.00 0.00 36.00 82.94 0.14 0.32
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md10 0.00 0.00 36.04 83.22 0.14 0.32
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md11 0.00 0.00 36.00 86.13 0.14 0.34
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md12 0.00 0.00 36.00 85.70 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md13 0.00 0.00 36.00 84.44 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md14 0.00 0.00 35.99 83.96 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md15 0.00 0.00 35.99 79.41 0.14 0.31
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md16 0.00 0.00 35.99 83.47 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md17 0.00 0.00 36.07 75.11 0.14 0.29
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md18 0.00 0.00 35.98 75.07 0.14 0.29
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md19 0.00 0.00 35.98 85.17 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md20 0.00 0.00 35.98 84.82 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md21 0.00 0.00 35.97 86.27 0.14 0.34
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md22 0.00 0.00 35.97 82.72 0.14 0.32
8.00 0.00 0.00 0.00 0.00 0.00 0.00
md23 0.00 0.00 35.94 84.84 0.14 0.33
8.00 0.00 0.00 0.00 0.00 0.00 0.00
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
