Hello Nicholas Using fedora here and LIO/ The same array is used in testing the RHEL and upstream kernels. Linux fedstorage 4.5.0-rc7+ #1 SMP Sun Mar 13 16:30:39 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux I have 3 NVME cards striped in an LVM configuration and I create the block devices using LVM and then provision using targetcli. I had not tuned anything on the LIO target because I was able to change the max_sectors_kb on the RHEL kernel. Looking at the I/O size I was issuing on the initiator I confirmed the 4MB I/O was working on the RHEL kernel. I realized of course it was possible the I/O could be broken down on the target by the time it handled the I/O but I was not focusing on that. When I started testing upstream because I bumped into the various sg_map issues this is when I found the different behavior on the LUNS on the initiator. Let me know if you think this is an array side tuning issue. optimal_sectors = 256 hw_max_sectors = 256 /sys/kernel/config/target/core/iblock_0/block-1 [root@fedstorage iblock_0]# cat hba_info HBA Index: 1 plugin: iblock version: v5.0 [root@fedstorage iblock_0]# cat hba_mode 0 [root@fedstorage iblock_0]# cd block-1/ /sys/kernel/config/target/core/iblock_0/block-1/attrib [root@fedstorage attrib]# for i in * > do > echo $i > cat $i > echo > done block_size 512 emulate_3pc 1 emulate_caw 1 emulate_dpo 1 emulate_fua_read 1 emulate_fua_write 1 emulate_model_alias 1 emulate_rest_reord 0 emulate_tas 1 emulate_tpu 0 emulate_tpws 0 emulate_ua_intlck_ctrl 0 emulate_write_cache 0 enforce_pr_isids 1 force_pr_aptpl 0 hw_block_size 512 hw_max_sectors ***** 256 hw_pi_prot_type 0 hw_queue_depth 128 is_nonrot 1 max_unmap_block_desc_count 1 max_unmap_lba_count 8388607 max_write_same_len 65535 optimal_sectors 256 ***** pi_prot_format 0 pi_prot_type 0 queue_depth 128 unmap_granularity 1 unmap_granularity_alignment 0 unmap_zeroes_data 0 [root@fedstorage ~]# targetcli ls o- / ........................................................................................................... [...] o- backstores ................................................................................................ [...] | o- block ................................................................................... [Storage Objects: 30] | | o- block-1 ................................................... [/dev/data/block-1 (9.0GiB) write-thru activated] | | o- block-2 ................................................... [/dev/data/block-2 (9.0GiB) write-thru activated] | | o- block-3 ................................................... [/dev/data/block-3 (9.0GiB) write-thru activated] .. .. | | o- block-28 ................................................. [/dev/data/block-28 (9.0GiB) write-thru activated] | | o- block-29 ................................................. [/dev/data/block-29 (9.0GiB) write-thru activated] | | o- block-30 ................................................. [/dev/data/block-30 (9.0GiB) write-thru activated] | o- fileio ................................................................................... [Storage Objects: 0] | o- pscsi .................................................................................... [Storage Objects: 0] | o- ramdisk .................................................................................. [Storage Objects: 0] o- iscsi .............................................................................................. [Targets: 0] o- loopback ........................................................................................... [Targets: 0] o- qla2xxx ............................................................................................ [Targets: 0] o- srpt ............................................................................................... [Targets: 2] | o- ib.fe800000000000007cfe900300726e4e ............................................................. [no-gen-acls] | | o- acls .............................................................................................. [ACLs: 2] | | | o- ib.4e6e72000390fe7c7cfe900300726ed2 ..................................................... [Mapped LUNs: 30] | | | | o- mapped_lun0 ................................................................... [lun0 block/block-1 (rw)] | | | | o- mapped_lun1 ................................................................... [lun1 block/block-2 (rw)] | | | | o- mapped_lun2 ................................................................... [lun2 block/block-3 (rw)] .. .. | | | | o- mapped_lun26 ................................................................ [lun26 block/block-27 (rw)] | | | | o- mapped_lun27 ................................................................ [lun27 block/block-28 (rw)] | | | | o- mapped_lun28 ................................................................ [lun28 block/block-29 (rw)] | | | | o- mapped_lun29 ................................................................ [lun29 block/block-30 (rw)] | | | o- ib.4f6e72000390fe7c7cfe900300726ed3 ..................................................... [Mapped LUNs: 30] | | | o- mapped_lun0 ................................................................... [lun0 block/block-1 (rw)] | | | o- mapped_lun1 ................................................................... [lun1 block/block-2 (rw)] | | | o- mapped_lun2 ................................................................... [lun2 block/block-3 (rw)] | | | o- mapped_lun3 ................................................................... [lun3 block/block-4 (rw)] | | | o- mapped_lun4 ................................................................... [lun4 block/block-5 (rw)] .. ,, Laurence Oberman Principal Software Maintenance Engineer Red Hat Global Support Services ----- Original Message ----- From: "Nicholas A. Bellinger" <nab@xxxxxxxxxxxxxxx> To: "Laurence Oberman" <loberman@xxxxxxxxxx> Cc: "linux-scsi" <linux-scsi@xxxxxxxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx, "target-devel" <target-devel@xxxxxxxxxxxxxxx> Sent: Friday, April 8, 2016 1:30:28 AM Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target Hi Laurence, On Thu, 2016-04-07 at 17:15 -0400, Laurence Oberman wrote: > Hello > > I have been testing the SRP initiator code to an LIO array here and > part of the testing requires me to set the max_sectors_kb size to get > 4k I/O's. > This has been due to me having to debug various sg_map issues. > > Linux srptest 4.5.0 #2 SMP Thu Apr 7 16:14:38 EDT 2016 x86_64 x86_64 > x86_64 GNU/Linux > This kernel has the scan patch from Hannes, as well as the "[PATCH] > IB/mlx5: Expose correct max_sge_rd limit" patch. > However, I also tested with vanilla 4.5.0 as well and its the same > issue. > > For some reason I cannot change the max_sectors_kb size on 4.5.0 here. > > I chatted with Ewan about it as well and he reminded me about Martins > changes so wondering if that's playing into this. > > Take /dev/sdb as an example > > [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb > VPD INQUIRY: Block limits page (SBC) > Maximum compare and write length: 1 blocks > Optimal transfer length granularity: 256 blocks > Maximum transfer length: 256 blocks > Optimal transfer length: 768 blocks > Maximum prefetch, xdread, xdwrite transfer length: 0 blocks > Just curious what target backend this is with..? Specifically the optimal transfer length granularity and optimal transfer length may be reported by underlying backend device (eg: IBLOCK) in spc_emulate_evpd_b0(). What does 'head /sys/kernel/config/target/core/$HBA/$DEV/attrib/*' of the backend device in question look like..? -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
