Hi, lio config (extract - .json file attached)o- iscsi ........................................................................................... [1-way disc auth, Targets: 1] | o- iqn.2017-01.at.cf-it.at-storage-01 ................................................................................ [TPGs: 1] | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.2017-01.at.cf-it.kubernetes.db-1 ..................................................... [1-way auth, Mapped LUNs: 1] | | o- mapped_lun0 ...................................................................... [lun0 fileio/kubernetes_db_1 (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 .......................................... [fileio/kubernetes_db_1 (/data/kubernetes_storage/kubernetes_db_1.img)] | o- portals .................................................................................................... [Portals: 1] | o- 0.0.0.0:3260 ................................................................................................... [iser]
at-storage-01 modules: ib_isert 49152 4 iscsi_target_mod 294912 8 ib_isert ib_iser 49152 0 libiscsi 57344 1 ib_iser scsi_transport_iscsi 98304 2 ib_iser,libiscsitarget_core_mod 352256 43 iscsi_target_mod,ib_isert,target_core_iblock,ib_srpt,target_core_user,target_core_file,target_core_pscsi
rdma_cm 53248 4 ib_iser,ib_isert,rpcrdma,rdma_ucmib_core 208896 15 ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
at-host-18 modules: ib_isert 49152 0 iscsi_target_mod 299008 1 ib_isert ib_iser 49152 0 libiscsi 57344 1 ib_iser scsi_transport_iscsi 98304 3 ib_iser,libiscsi target_core_mod 352256 3 iscsi_target_mod,ib_isert,ib_srpt rdma_cm 53248 4 ib_iser,ib_isert,rpcrdma,rdma_ucmib_core 208896 15 ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdmavt,ib_qib,rdma_ucm
Commands to initialize iscsiadm: iscsiadm -m discovery -t st -p 10.0.13.3:3260 -I iser iscsiadm -m node -P 1 Target: iqn.2017-01.at.cf-it.at-storage-01 Portal: 10.0.13.3:3260,1 Iface Name: iser iscsiadm -m node -l ->> creates the logs @deleted and recreated, multiple times ib_isert and ib_iser are loaded on both servers BR Thomas On 7 Jun 2017, at 20:35, Robert LeBlanc wrote:
Can you post the config for lio and the iscsiadm commands you are using? Have you deleted the record from the discoverydb and rediscovered? You need to be sure the enable_iser boolean is set on the target and that the modules (ib_isert and ib_iser) are loaded. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jun 7, 2017 at 12:15 PM, Thomas Rosenstein <thomas.rosenstein@xxxxxxxxxxxxxxxx> wrote:Hello,i have setup a lio with 1 LUN on kernel 4.9.11, works fine via iSCSI, butwhen switching to iSER the message logs are spammed with: Jun 7 17:59:57 at-host-18 kernel: iser: iser_fast_reg_fmr: ib_fmr_pool_map_phys failed: -22Jun 7 17:59:57 at-host-18 kernel: iser: iser_prepare_read_cmd: Failed toset up Data-IN RDMA Jun 7 17:59:57 at-host-18 kernel: iser: iser_send_command: conn ffff88239eaf3b30 failed task->itt 127 err -22Based on the error I found that there was a bug in 4.5.x rc Kernels and therefore tried 4.4.71, 4.9.31 and 4.9.27 but all of them have exactly thesame behaviour.I got HP DL160 (SE316) with QLogic Infiniband Adapters connected via a HPVoltair switch. ibnetdiag says everything is fine. Network via ib0 works without issues (can mount the iSCSI via that). rdma_server and rdma_client both are successful (end 0) ib_read_bw 10.0.13.3 --------------------------------------------------------------------------------------- Device not recognized to implement inline feature. Disabling it --------------------------------------------------------------------------------------- RDMA_Read BW Test Dual-port : OFF Device : qib0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF TX depth : 128 CQ Moderation : 100 Mtu : 4096[B] Link type : IB Outstand reads : 16 rdma_cm QPs : OFF Data ex. method : Ethernet ---------------------------------------------------------------------------------------local address: LID 0x04 QPN 0x0017 PSN 0xd20eae OUT 0x10 RKey 0x2d2d300VAddr 0x007f892ea78000remote address: LID 0x13 QPN 0x0d7d PSN 0x66ee4e OUT 0x10 RKey 0x070800VAddr 0x007fcf485ee000 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]65536 1000 3256.60 3156.37 0.050502--------------------------------------------------------------------------------------- ib_send_bw 10.0.13.3 --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF Device : qib0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF TX depth : 128 CQ Moderation : 100 Mtu : 4096[B] Link type : IB Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0x04 QPN 0x0019 PSN 0xae4e4c remote address: LID 0x13 QPN 0x0d89 PSN 0xd25041 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]65536 1000 3232.96 3232.92 0.051727---------------------------------------------------------------------------------------ib_read and ib_send also look fine. (Throughput as expected with 3.2 GB/s -40 Gbit/s QDR) Also srpt is working, but I need iSCSI.Does anyone know what the issue could be, or how I can better analyze what'shappening there? Thanks BR Thomas --To unsubscribe from this list: send the line "unsubscribe linux-rdma" inthe body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
Attachment:
saveconfig.json
Description: application/json
