Re: iSCSI write performance

Ryan <rswagoner@xxxxxxxxx> · Fri, 25 Oct 2019 10:31:31 -0400

I'm not seeing the 

emulate_3pc setting under disks/rbd/diskname when calling info. A google search shows that SUSE Enterprise Storage has it available. I thought I had the latest packages, but maybe not. I'm using tcmu-runner 1.5.2 and ceph-iscsi 3.3. Almost all of my VMs are currently on Nimble iSCSI storage. I've actually tested from both and performance is the same. Doing the math off the ceph status does show it using 64K blocks in both cases.

Control Values
- hw_max_sectors .. 1024
- max_data_area_mb .. 256 (override)
- osd_op_timeout .. 30
- qfull_timeout .. 5

On Fri, Oct 25, 2019 at 4:46 AM Maged Mokhtar <mmokhtar@xxxxxxxxxxx> wrote:

    Actually this may not work if moving from a local datastore to
      Ceph. For iSCSI xcopy, both the source and destination need to be
      accessible by the target such as in moving vms across Ceph
      datastores. So in your case, vmotion will be handled by VMWare
      data mover which uses 64K block sizes. 

    On 25/10/2019 10:28, Maged Mokhtar
      wrote:

      For vmotion speed, check "emulate_3pc" attribute on the LIO
        target. If 0 (default), VMWare will issue io in 64KB blocks
        which gives low speed. if set to 1  this will trigger VMWare to
        use vaai extended copy, which activates LIO's xcopy
        functionality which uses 512KB block sizes by default. We also
        bumped the xcopy block size to 4M (rbd object size) which gives
        around 400 MB/s vmotion speed, the same speed can also be
        achieved via Veeam backups.
      /Maged

      On 25/10/2019 06:47, Ryan wrote:

          I'm using CentOS 7.7.1908 with kernel
            3.10.0-1062.1.2.el7.x86_64. The workload was a VMware
            Storage Motion from a local SSD backed datastore to the ceph
            backed datastore. Performance was measured using dstat on
            the iscsi gateway for network traffic and ceph status as
            this cluster is basically idle.  I changed max_data_area_mb
            to 256 and cmdsn_depth to 128. This appears to have given a
            slight improvement of maybe 10MB/s. 

          Moving VM to the ceph backed datastore
          io:

                client:   124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k
            op/s wr

          Moving VM off the ceph backed datastore
            io:

                client:   344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62
            op/s wr

          I'm going to test bonnie++ with an rbd volume mounted
            directly on the iscsi gateway. Also will test bonnie++
            inside a VM on a ceph backed datastore.

            On Thu, Oct 24, 2019 at
              7:15 PM Mike Christie <mchristi@xxxxxxxxxx>
              wrote:

            On 10/24/2019 12:22 PM, Ryan
              wrote:

              > I'm in the process of testing the iscsi target
              feature of ceph. The

              > cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It
              consists of 5

              What kernel are you using?

              > hosts with 12 SSD OSDs per host. Some basic testing
              moving VMs to a ceph

              > backed datastore is only showing 60MB/s transfers.
              However moving these

              > back off the datastore is fast at 200-300MB/s.

              What is the workload and what are you using to measure the
              throughput?

              If you are using fio, what arguments are you using? And,
              could you

              change the ioengine to rbd and re-run the test from the
              target system so

              we can check if rbd is slow or iscsi?

              For small IOs, 60 is about right.

              For 128-512K IOs you should be able to get around 300 MB/s
              for writes

              and 600 for reads.

              1. Increase max_data_area_mb. This is a kernel buffer
              lio/tcmu uses to

              pass data between the kernel and tcmu-runner. The default
              is only 8MB.

              In gwcli cd to your disk and do:

              # reconfigure max_data_area_mb %N

              where N is between 8 and 2048 MBs.

              2. The Linux kernel target only allows 64 commands per
              iscsi session by

              default. We increase that to 128, but you can increase
              this to 512.

              In gwcli cd to the target dir and do

              reconfigure cmdsn_depth 512

              3. I think ceph-iscsi and lio work better with higher
              queue depths so if

              you are using fio you want higher numjobs and/or iodepths.

              > 

              > What should I be looking at to track down the write
              performance issue?

              > In comparison with the Nimble Storage arrays I can
              see 200-300MB/s in

              > both directions.

              > 

              > Thanks,

              > Ryan

              > 

              > 

              > _______________________________________________

              > ceph-users mailing list -- ceph-users@xxxxxxx

              > To unsubscribe send an email to ceph-users-leave@xxxxxxx

              > 

        _______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

      _______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx