On Fri, 2013-10-11 at 21:04 -0500, xxiao wrote: > On Fri, Oct 11, 2013 at 5:28 PM, Nicholas A. Bellinger > <nab@xxxxxxxxxxxxxxx> wrote: > > On Thu, 2013-10-10 at 18:33 -0500, Xianghua Xiao wrote: > >> > > > > <SNIP> > > > >> I switched to fileio and now READ is at full speed, i.e. wirespeed of > >> 10Gbps, plus all cores are being used(though still only core0 takes > >> the MSI-X interrupts). However, WRITE slows down dramatically from > >> 415MB/s to 150MB/s, that really puzzles me. > >> > > > > From the Mode: O_DSYNC output below, the FILEIO device is *not* running > > in buffered I/O mode for WRITEs. > > > > Also, can you confirm using iostat -xm that the READs are coming from > > the backend device, and not from the buffer cache itself..? > > > > FYI, iostat -xm output also shosw the size of the I/Os being submitted > > to both the MD RAID devices, as well as to the underlying mpt2sas SCSI > > devices. This would be useful to see if the size of I/Os is different > > between backend drivers. > > > >> > >> I had the same setting, including : > >> echo 1 > /sys/kernel/config/target/core/fileio_$i/my_fileio > >> $i/attrib/emulate_write_cache for all endpoints. > >> At WRITE all cores are equally involved, they're just not fully loaded > >> at all. > >> > > > > Setting emulate_write_cache is the per device attribute for exposing the > > SCSI WriteCacheEnabled bit to the initiator, and does not directly > > enable buffered I/O operation for FILEIO backends. > > > >> > >> Here is the output of fileio: > >> > >> Rounding down aligned max_sectors from 32767 to 32760 > >> Status: DEACTIVATED Max Queue Depth: 0 SectorSize: 512 HwMaxSectors: 32760 > >> TCM FILEIO ID: 0 File: /dev/md12 Size: 0 Mode: O_DSYNC > >> > >> I'm still trying to get the optimal throughput for sequential > >> READ/WRITE at the moment. Reliability is secondary as I'm trying to > >> stress the system performance for now. Is there anything else that I > >> can tune for FILEIO (or IBLOCK)? > > > > Buffered FILEIO can be enabled with targetcli + rtslib, but not during > > setup with the legacy lio-utils code. > > > > The easiest way to do this is modifying /etc/target/tcm_start.sh after > > the configuration has been saved, and then including the > > ',fd_buffered_io=1' parameter for each FILEIO device like so: > > > > tcm_node --establishdev fileio_0/test fd_dev_name=/tmp/test,fd_dev_size=2147483648,fd_buffered_io=1 > > > > From there restart the target, and verify the same output from > > /sys/kernel/config/target/core/$FILEIO_HBA/$FILEIO_DEV/info as above, which > > should now read: > > > > Status: DEACTIVATED Max Queue Depth: 0 SectorSize: 512 HwMaxSectors: 32760 > > TCM FILEIO ID: 0 File: /dev/md12 Size: 0 Mode: Buffered-WCE > > > > --nab > > > > After I added fd_buffered_io I'm getting very similar results as > SCST's fileio mode. There are still more to optimize but I'm happy > that I can see similar results now. > > Thanks so much for all your helps! > Again to confirm, the SCST numbers that where previously quoted where all with buffered FILEIO then, right..? That would make sense, as I don't think 4x SATA 3 SSDs with 12 active partitions can sustain 1000 MB/sec of large block random I/O. --nab -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
