On Thu, Jan 17, 2008 at 01:34:46PM +0300, Vladislav Bolkhovitin wrote: >Hmm, I can't find which IB hardware did he use and it's declared Gbps >speed. He declared only "Mellanox 4X SDR, switch". What does it mean? SDR is 10Gbit carrier, at most about ~900MB/s data rate. DDR is 20Gbit carrier, at most about ~1400MB/s data rate. On Thu, 17 Jan 2008 10:27:08 +0100 "Bart Van Assche" <bart.vanassche@xxxxxxxxx> wrote: > Test performed: read 2 GB of data in blocks of 1 MB from a target (hot > cache -- no disk reads were performed, all reads were from the cache). > Test command: time dd if=/dev/sde of=/dev/null bs=1M count=2000 > > STGT read SCST read > performance (MB/s) performance (MB/s) > Ethernet (1 Gb/s network) 77 89 > IPoIB (8 Gb/s network) 82 229 > SRP (8 Gb/s network) N/A 600 > iSER (8 Gb/s network) 80 N/A it kinda looks to me like the tgt iSER tests were waaay too slow to be using RDMA :-/ I use tgt to get 500MB/s writes over iSER DDR IB to real files (not ramdisk). Reads are a little slower, but that changes a bit with distro vs. mainline kernels. was iscsiadm was pointed at the IP of the IPoIB interface on the target? I think tgtd requires that. how about setting the transport to be iser with eg. iscsiadm --mode node --targetname <something> --portal <ipoib>:3260 --op update -n node.transport_name -v iser iscsiadm --mode node --targetname <something> --portal <ipoib>:3260 --op update -n "node.conn[0].iscsi.HeaderDigest" -v None does the initiator side kernel report that it's using iSER? it should look roughly like the below. Jan 14 14:37:21 x2 kernel: iscsi: registered transport (iser) Jan 14 14:37:21 x2 iscsid: iSCSI logger with pid=5617 started! Jan 14 14:37:22 x2 iscsid: transport class version 2.0-724. iscsid version 2.0-865 Jan 14 14:37:22 x2 iscsid: iSCSI daemon with pid=5618 started! Jan 14 14:37:22 x2 kernel: iser: iser_connect:connecting to: 192.168.1.8, port 0xbc0c Jan 14 14:37:23 x2 kernel: iser: iser_cma_handler:event 0 conn ffff8102523c4c80 id ffff81025df68e00 Jan 14 14:37:23 x2 kernel: iser: iser_cma_handler:event 2 conn ffff8102523c4c80 id ffff81025df68e00 Jan 14 14:37:24 x2 kernel: iser: iser_create_ib_conn_res:setting conn ffff8102523c4c80 cma_id ffff81025df68e00: fmr_pool ffff81025341b5c0 qp ffff810252109200 Jan 14 14:37:24 x2 kernel: iser: iser_cma_handler:event 9 conn ffff8102523c4c80 id ffff81025df68e00 Jan 14 14:37:24 x2 kernel: iser: iscsi_iser_ep_poll:ib conn ffff8102523c4c80 rc = 1 Jan 14 14:37:24 x2 kernel: scsi6 : iSCSI Initiator over iSER, v.0.1 Jan 14 14:37:24 x2 kernel: iser: iscsi_iser_conn_bind:binding iscsi conn ffff810251a94290 to iser_conn ffff8102523c4c80 Jan 14 14:37:24 x2 kernel: Vendor: IET Model: Controller Rev: 0001 Jan 14 14:37:24 x2 kernel: Type: RAID ANSI SCSI revision: 05 Jan 14 14:37:24 x2 kernel: scsi 6:0:0:0: Attached scsi generic sg2 type 12 Jan 14 14:37:25 x2 kernel: Vendor: IET Model: VIRTUAL-DISK Rev: 0001 Jan 14 14:37:25 x2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 14 14:37:25 x2 kernel: SCSI device sdc: 20971520 512-byte hdwr sectors (10737 MB) Jan 14 14:37:25 x2 kernel: sdc: Write Protect is off Jan 14 14:37:25 x2 kernel: SCSI device sdc: drive cache: write back Jan 14 14:37:25 x2 kernel: SCSI device sdc: 20971520 512-byte hdwr sectors (10737 MB) Jan 14 14:37:25 x2 kernel: sdc: Write Protect is off Jan 14 14:37:25 x2 kernel: SCSI device sdc: drive cache: write back Jan 14 14:37:25 x2 kernel: sdc: unknown partition table Jan 14 14:37:25 x2 kernel: sd 6:0:0:1: Attached scsi disk sdc Jan 14 14:37:26 x2 kernel: sd 6:0:0:1: Attached scsi generic sg3 type 0 Jan 14 14:37:26 x2 iscsid: connection1:0 is operational now cheers, robin - To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
