That's interesting. Although I'm running 3.16.7 and that I'd expect the patch to be in already, I'll downgrade to the "working" 3.16.0 kernel and report back if this fixes the issue. Thanks for the pointer. -- David Moreau Simard > On Nov 13, 2014, at 1:15 PM, German Anders <ganders@xxxxxxxxxxxx> wrote: > > Is possible that you hit bug #8818 ? > > > German Anders > > > > > > > > > > > > > > > > >> --- Original message --- >> Asunto: Re: Poor RBD performance as LIO iSCSI target >> De: David Moreau Simard <dmsimard@xxxxxxxx> >> Para: Christopher Spearman <neromaverick@xxxxxxxxx> >> Cc: ceph-users@xxxxxxxxxxxxxx <ceph-users@xxxxxxxxxxxxxx> >> Fecha: Thursday, 13/11/2014 13:17 >> >> Running into weird issues here as well in a test environment. I don't have a solution either but perhaps we can find some things in common.. >> >> Setup in a nutshell: >> - Ceph cluster: Ubuntu 14.04, Kernel 3.16.7, Ceph 0.87-1 (OSDs with separate public/cluster network in 10 Gbps) >> - iSCSI Proxy node (targetcli/LIO): Ubuntu 14.04, Kernel 3.16.7, Ceph 0.87-1 (10 Gbps) >> - Client node: Ubuntu 12.04, Kernel 3.11 (10 Gbps) >> >> Relevant cluster config: Writeback cache tiering with NVME PCI-E cards (2 replica) in front of a erasure coded pool (k=3,m=2) backed by spindles. >> >> I'm following the instructions here: http://www.hastexo.com/resources/hints-and-kinks/turning-ceph-rbd-images-san-storage-devices >> No issues with creating and mapping a 100GB RBD image and then creating the target. >> >> I'm interested in finding out the overhead/performance impact of re-exporting through iSCSI so the idea is to run benchmarks. >> Here's a fio test I'm trying to run on the client node on the mounted iscsi device: >> fio --name=writefile --size=100G --filesize=100G --filename=/dev/sdu --bs=1M --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 --iodepth=200 --ioengine=libaio >> >> The benchmark will eventually hang towards the end of the test for some long seconds before completing. >> On the proxy node, the kernel complains with iscsi portal login timeout: http://pastebin.com/Q49UnTPr and I also see irqbalance errors in syslog: http://pastebin.com/AiRTWDwR >> >> Doing the same test on the machines directly (raw, rbd, on the osd filesystem) doesn't yield any issues. >> >> I've tried a couple things to see if I could get things to work... >> - Set irqbalance --hintpolicy=ignore (http://sourceforge.net/p/e1000/bugs/394/ & https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1321425) >> - Changed size on cache pool to 1 (for the sake of testing, improved performance but still hangs) >> - Set crush tunables to legacy (and back to optimal) >> - Various package and kernel versions and putting the proxy node on Ubuntu precise >> - Formatting and mounting the iscsi block device and running the test on the formatted filesystem >> >> I don't think it's related .. but I don't remember running into issues before I've swapped out SSDs for the NVME cards for the cache pool. >> I don't have time *right now* but I definitely want to test if I am able to reproduce the issue on the SSDs.. >> >> Let me know if this gives you any ideas, I'm all ears. >> -- >> David Moreau Simard >> >>> On Oct 28, 2014, at 4:07 PM, Christopher Spearman <neromaverick@xxxxxxxxx> wrote: >>> >>> Sage: >>> >>> That'd be my assumption, performance looked pretty fantastic over loop until it started being used it heavily >>> >>> Mike: >>> >>> The configs you asked for are at the end of this message I've subtracted & changed some info, iqn/wwn/portal, for security purposes. The raw & loop target configs are all in one since I'm running both types of configs currently. I also included the running config (ls /) of targetcli for anyone interested in what it looks like from the console. >>> >>> The tool I used was dd, I ran through various options using dd but didn't really see much difference. The one on top is my go to command for my first test >>> >>> time dd if=/dev/zero of=test bs=32M count=32 oflag=direct,sync >>> time dd if=/dev/zero of=test bs=32M count=128 oflag=direct,sync >>> time dd if=/dev/zero of=test bs=8M count=512 oflag=direct,sync >>> time dd if=/dev/zero of=test bs=4M count=1024 oflag=direct,sync >>> >>> >>> ---ls / from current targetcli (no mounted ext4 -> image file config)--- >>> >>> /iscsi> ls / >>> o- / ......................................................................................................................... [...] >>> o- backstores .............................................................................................................. [...] >>> | o- block .................................................................................................. [Storage Objects: 2] >>> | | o- ceph_lun0 ...................................................................... [/dev/loop0 (2.0TiB) write-thru activated] >>> | | o- ceph_noloop00 .............................................. [/dev/rbd/vmiscsi/noloop00 (1.0TiB) write-thru activated] >>> | o- fileio ................................................................................................. [Storage Objects: 0] >>> | o- pscsi .................................................................................................. [Storage Objects: 0] >>> | o- ramdisk ................................................................................................ [Storage Objects: 0] >>> o- iscsi ............................................................................................................ [Targets: 2] >>> | o- iqn.gateway2_01 ..................................................... [TPGs: 1] >>> | | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] >>> | | o- acls .......................................................................................................... [ACLs: 2] >>> | | | o- iqn.esxhost01 ............................................................ [Mapped LUNs: 1] >>> | | | | o- mapped_lun0 ......................................................................... [lun0 block/ceph_noloop00 (rw)] >>> | | | o- iqn.esxhost02 ....................................................... [Mapped LUNs: 1] >>> | | | o- mapped_lun0 ......................................................................... [lun0 block/ceph_noloop00 (rw)] >>> | | o- luns .......................................................................................................... [LUNs: 1] >>> | | | o- lun0 ........................................................... [block/ceph_noloop00 (/dev/rbd/vmiscsi/noloop00)] >>> | | o- portals .................................................................................................... [Portals: 1] >>> | | o- xxx.xxx.xxx.xxx:3260 ............................................................................................... [OK] >>> | o- iqn.gateway2_02 ..................................................... [TPGs: 1] >>> | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] >>> | o- acls .......................................................................................................... [ACLs: 2] >>> | | o- iqn.esxhost01 ............................................................ [Mapped LUNs: 1] >>> | | | o- mapped_lun0 ............................................................................. [lun0 block/ceph_lun0 (rw)] >>> | | o- iqn.esxhost02 ............................................................ [Mapped LUNs: 1] >>> | | | o- mapped_lun0 ............................................................................. [lun0 block/ceph_lun0 (rw)] >>> | o- luns .......................................................................................................... [LUNs: 1] >>> | | o- lun0 ................................................................................... [block/ceph_lun0 (/dev/loop0)] >>> | o- portals .................................................................................................... [Portals: 1] >>> | o- xxx.xxx.xxx.xxx:3260 ............................................................................................... [OK] >>> o- loopback ......................................................................................................... [Targets: 0] >>> >>> ---saveconfig.json for mounted ext4 config--- >>> >>> { >>> "fabric_modules": [], >>> "storage_objects": [ >>> { >>> "attributes": { >>> "block_size": 512, >>> "emulate_dpo": 0, >>> "emulate_fua_read": 0, >>> "emulate_fua_write": 1, >>> "emulate_model_alias": 1, >>> "emulate_rest_reord": 0, >>> "emulate_tas": 1, >>> "emulate_tpu": 0, >>> "emulate_tpws": 0, >>> "emulate_ua_intlck_ctrl": 0, >>> "emulate_write_cache": 1, >>> "enforce_pr_isids": 1, >>> "fabric_max_sectors": 8192, >>> "is_nonrot": 0, >>> "max_unmap_block_desc_count": 1, >>> "max_unmap_lba_count": 8192, >>> "max_write_same_len": 4096, >>> "optimal_sectors": 8192, >>> "queue_depth": 128, >>> "unmap_granularity": 1, >>> "unmap_granularity_alignment": 0 >>> }, >>> "dev": "/mnt/ceph_perf_test/mounted_rbd_img_test.img", >>> "name": "mounted_rbd_img_test", >>> "plugin": "fileio", >>> "size": 8589934592, >>> "write_back": true, >>> "wwn": "xxxx-xxxx-xxxx-xxxx" >>> } >>> ], >>> "targets": [ >>> { >>> "fabric": "iscsi", >>> "tpgs": [ >>> { >>> "attributes": { >>> "authentication": 0, >>> "cache_dynamic_acls": 0, >>> "default_cmdsn_depth": 16, >>> "demo_mode_write_protect": 1, >>> "generate_node_acls": 0, >>> "login_timeout": 15, >>> "netif_timeout": 2, >>> "prod_mode_write_protect": 0 >>> }, >>> "enable": true, >>> "luns": [ >>> { >>> "index": 0, >>> "storage_object": "/backstores/fileio/mounted_rbd_img_test" >>> } >>> ], >>> "node_acls": [ >>> { >>> "attributes": { >>> "dataout_timeout": 3, >>> "dataout_timeout_retries": 5, >>> "default_erl": 0, >>> "nopin_response_timeout": 30, >>> "nopin_timeout": 15, >>> "random_datain_pdu_offsets": 0, >>> "random_datain_seq_offsets": 0, >>> "random_r2t_offsets": 0 >>> }, >>> "mapped_luns": [ >>> { >>> "index": 0, >>> "tpg_lun": 0, >>> "write_protect": false >>> } >>> ], >>> "node_wwn": "iqn.centoshost01" >>> } >>> ], >>> "parameters": { >>> "AuthMethod": "CHAP,None", >>> "DataDigest": "CRC32C,None", >>> "DataPDUInOrder": "Yes", >>> "DataSequenceInOrder": "Yes", >>> "DefaultTime2Retain": "20", >>> "DefaultTime2Wait": "2", >>> "ErrorRecoveryLevel": "0", >>> "FirstBurstLength": "65536", >>> "HeaderDigest": "CRC32C,None", >>> "IFMarkInt": "2048~65535", >>> "IFMarker": "No", >>> "ImmediateData": "Yes", >>> "InitialR2T": "Yes", >>> "MaxBurstLength": "262144", >>> "MaxConnections": "1", >>> "MaxOutstandingR2T": "1", >>> "MaxRecvDataSegmentLength": "8192", >>> "MaxXmitDataSegmentLength": "262144", >>> "OFMarkInt": "2048~65535", >>> "OFMarker": "No", >>> "TargetAlias": "LIO Target" >>> }, >>> "portals": [ >>> { >>> "ip_address": "xxx.xxx.xxx.xxx", >>> "iser": false, >>> "port": 3260 >>> } >>> ], >>> "tag": 1 >>> } >>> ], >>> "wwn": "iqn.gateway1_01" >>> }, >>> ] >>> } >>> >>> ---saveconfig.json for raw & loop targets--- >>> >>> { >>> "fabric_modules": [], >>> "storage_objects": [ >>> { >>> "attributes": { >>> "block_size": 512, >>> "emulate_dpo": 0, >>> "emulate_fua_read": 0, >>> "emulate_fua_write": 1, >>> "emulate_model_alias": 1, >>> "emulate_rest_reord": 0, >>> "emulate_tas": 1, >>> "emulate_tpu": 0, >>> "emulate_tpws": 0, >>> "emulate_ua_intlck_ctrl": 0, >>> "emulate_write_cache": 0, >>> "enforce_pr_isids": 1, >>> "fabric_max_sectors": 8192, >>> "is_nonrot": 0, >>> "max_unmap_block_desc_count": 0, >>> "max_unmap_lba_count": 0, >>> "max_write_same_len": 65535, >>> "optimal_sectors": 8192, >>> "queue_depth": 128, >>> "unmap_granularity": 0, >>> "unmap_granularity_alignment": 0 >>> }, >>> "dev": "/dev/rbd/vmiscsi/noloop00", >>> "name": "ceph_noloop00", >>> "plugin": "block", >>> "readonly": false, >>> "write_back": false, >>> "wwn": "xxxx-xxxx-xxxx" >>> }, >>> { >>> "attributes": { >>> "block_size": 512, >>> "emulate_dpo": 0, >>> "emulate_fua_read": 0, >>> "emulate_fua_write": 1, >>> "emulate_model_alias": 1, >>> "emulate_rest_reord": 0, >>> "emulate_tas": 1, >>> "emulate_tpu": 0, >>> "emulate_tpws": 0, >>> "emulate_ua_intlck_ctrl": 0, >>> "emulate_write_cache": 0, >>> "enforce_pr_isids": 1, >>> "fabric_max_sectors": 8192, >>> "is_nonrot": 0, >>> "max_unmap_block_desc_count": 0, >>> "max_unmap_lba_count": 0, >>> "max_write_same_len": 65535, >>> "optimal_sectors": 8192, >>> "queue_depth": 128, >>> "unmap_granularity": 0, >>> "unmap_granularity_alignment": 0 >>> }, >>> "dev": "/dev/loop0", >>> "name": "ceph_lun0", >>> "plugin": "block", >>> "readonly": false, >>> "write_back": false, >>> "wwn": "yyyy-yyyy-yyyy" >>> }, >>> ], >>> "targets": [ >>> { >>> "fabric": "iscsi", >>> "tpgs": [ >>> { >>> "attributes": { >>> "authentication": 0, >>> "cache_dynamic_acls": 0, >>> "default_cmdsn_depth": 16, >>> "demo_mode_write_protect": 1, >>> "generate_node_acls": 0, >>> "login_timeout": 15, >>> "netif_timeout": 2, >>> "prod_mode_write_protect": 0 >>> }, >>> "enable": true, >>> "luns": [ >>> { >>> "index": 0, >>> "storage_object": "/backstores/block/ceph_noloop00" >>> } >>> ], >>> "node_acls": [ >>> { >>> "attributes": { >>> "dataout_timeout": 3, >>> "dataout_timeout_retries": 5, >>> "default_erl": 0, >>> "nopin_response_timeout": 30, >>> "nopin_timeout": 15, >>> "random_datain_pdu_offsets": 0, >>> "random_datain_seq_offsets": 0, >>> "random_r2t_offsets": 0 >>> }, >>> "mapped_luns": [ >>> { >>> "index": 0, >>> "tpg_lun": 0, >>> "write_protect": false >>> } >>> ], >>> "node_wwn": "iqn.esxhost01" >>> }, >>> { >>> "attributes": { >>> "dataout_timeout": 3, >>> "dataout_timeout_retries": 5, >>> "default_erl": 0, >>> "nopin_response_timeout": 30, >>> "nopin_timeout": 15, >>> "random_datain_pdu_offsets": 0, >>> "random_datain_seq_offsets": 0, >>> "random_r2t_offsets": 0 >>> }, >>> "mapped_luns": [ >>> { >>> "index": 0, >>> "tpg_lun": 0, >>> "write_protect": false >>> } >>> ], >>> "node_wwn": "iqn.esxhost02" >>> } >>> ], >>> "parameters": { >>> "AuthMethod": "CHAP,None", >>> "DataDigest": "CRC32C,None", >>> "DataPDUInOrder": "Yes", >>> "DataSequenceInOrder": "Yes", >>> "DefaultTime2Retain": "20", >>> "DefaultTime2Wait": "2", >>> "ErrorRecoveryLevel": "0", >>> "FirstBurstLength": "65536", >>> "HeaderDigest": "CRC32C,None", >>> "IFMarkInt": "2048~65535", >>> "IFMarker": "No", >>> "ImmediateData": "Yes", >>> "InitialR2T": "Yes", >>> "MaxBurstLength": "262144", >>> "MaxConnections": "1", >>> "MaxOutstandingR2T": "1", >>> "MaxRecvDataSegmentLength": "8192", >>> "MaxXmitDataSegmentLength": "262144", >>> "OFMarkInt": "2048~65535", >>> "OFMarker": "No", >>> "TargetAlias": "LIO Target" >>> }, >>> "portals": [ >>> { >>> "ip_address": "xxx.xxx.xxx.xxx", >>> "iser": false, >>> "port": 3260 >>> } >>> ], >>> "tag": 1 >>> } >>> ], >>> "wwn": "iqn.gateway2_01" >>> }, >>> { >>> "fabric": "iscsi", >>> "tpgs": [ >>> { >>> "attributes": { >>> "authentication": 0, >>> "cache_dynamic_acls": 0, >>> "default_cmdsn_depth": 16, >>> "demo_mode_write_protect": 1, >>> "generate_node_acls": 0, >>> "login_timeout": 15, >>> "netif_timeout": 2, >>> "prod_mode_write_protect": 0 >>> }, >>> "enable": true, >>> "luns": [ >>> { >>> "index": 0, >>> "storage_object": "/backstores/block/ceph_lun0" >>> } >>> ], >>> "node_acls": [ >>> { >>> "attributes": { >>> "dataout_timeout": 3, >>> "dataout_timeout_retries": 5, >>> "default_erl": 0, >>> "nopin_response_timeout": 30, >>> "nopin_timeout": 15, >>> "random_datain_pdu_offsets": 0, >>> "random_datain_seq_offsets": 0, >>> "random_r2t_offsets": 0 >>> }, >>> "mapped_luns": [ >>> { >>> "index": 0, >>> "tpg_lun": 0, >>> "write_protect": false >>> } >>> ], >>> "node_wwn": "iqn.esxhost01" >>> }, >>> { >>> "attributes": { >>> "dataout_timeout": 3, >>> "dataout_timeout_retries": 5, >>> "default_erl": 0, >>> "nopin_response_timeout": 30, >>> "nopin_timeout": 15, >>> "random_datain_pdu_offsets": 0, >>> "random_datain_seq_offsets": 0, >>> "random_r2t_offsets": 0 >>> }, >>> "mapped_luns": [ >>> { >>> "index": 0, >>> "tpg_lun": 0, >>> "write_protect": false >>> } >>> ], >>> "node_wwn": "iqn.esxhost02" >>> }, >>> ], >>> "parameters": { >>> "AuthMethod": "CHAP,None", >>> "DataDigest": "CRC32C,None", >>> "DataPDUInOrder": "Yes", >>> "DataSequenceInOrder": "Yes", >>> "DefaultTime2Retain": "20", >>> "DefaultTime2Wait": "2", >>> "ErrorRecoveryLevel": "0", >>> "FirstBurstLength": "65536", >>> "HeaderDigest": "CRC32C,None", >>> "IFMarkInt": "2048~65535", >>> "IFMarker": "No", >>> "ImmediateData": "Yes", >>> "InitialR2T": "Yes", >>> "MaxBurstLength": "262144", >>> "MaxConnections": "1", >>> "MaxOutstandingR2T": "1", >>> "MaxRecvDataSegmentLength": "8192", >>> "MaxXmitDataSegmentLength": "262144", >>> "OFMarkInt": "2048~65535", >>> "OFMarker": "No", >>> "TargetAlias": "LIO Target" >>> }, >>> "portals": [ >>> { >>> "ip_address": "xxx.xxx.xxx.xxx", >>> "iser": false, >>> "port": 3260 >>> } >>> ], >>> "tag": 1 >>> } >>> ], >>> "wwn": "iqn.gateway2_02" >>> } >>> ] >>> } >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |