I'll refer you to the original thread about this [1] that was awaiting an answer. I would recommend dropping the "-t none" option since that might severely slow down sequential write operations if "qemu-img convert" is performing 512 byte IO operations. You might also want to consider adding the "-S 4k' option to potentially re-sparsify a non-sparse image (i.e. so it doesn't waste time writing zeroes to the RBD image). [1] https://www.spinics.net/lists/ceph-users/msg37064.html On Thu, Jul 13, 2017 at 8:29 AM, Mahesh Jambhulkar <mahesh.jambhulkar@xxxxxxxxx> wrote: > Seeing some performance issues on my ceph cluster with qemu-img convert > directly writing to ceph against normal rbd import command. > > Direct data copy (without qemu-img convert) took 5 hours 43 minutes for > 465GB data. > > > [root@cephlarge vm_res_id_24291e4b-93d2-47ad-80a8-bf3c395319b9_vdb]# time > rbd import 66582225-6539-4e5e-9b7a-59aa16739df1 -p volumes > 66582225-6539-4e5e-9b7a-59aa16739df1_directCopy --image-format 2 > rbd: --pool is deprecated for import, use --dest-pool > Importing image: 100% complete...done. > > real 343m38.028s > user 4m40.779s > sys 7m18.916s > [root@cephlarge vm_res_id_24291e4b-93d2-47ad-80a8-bf3c395319b9_vdb]# rbd > info volumes/66582225-6539-4e5e-9b7a-59aa16739df1_directCopy > rbd image '66582225-6539-4e5e-9b7a-59aa16739df1_directCopy': > size 465 GB in 119081 objects > order 22 (4096 kB objects) > block_name_prefix: rbd_data.373174b0dc51 > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, > deep-flatten > flags: > [root@cephlarge vm_res_id_24291e4b-93d2-47ad-80a8-bf3c395319b9_vdb]# > > > Qemu-img convert is still in progress and completed merely 10% in more than > 40 hours. (for 465GB data) > > [root@cephlarge mnt]# time qemu-img convert -p -t none -O raw > /mnt/data/workload_326e8a43-a90a-4fe9-8aab-6d33bcdf5a05/snapshot_9f0cee13-8200-4562-82ec-1fb9f234bcd8/vm_id_05e9534e-5c84-4487-9613-1e0e227e4c1a/vm_res_id_24291e4b-93d2-47ad-80a8-bf3c395319b9_vdb/66582225-6539-4e5e-9b7a-59aa16739df1 > rbd:volumes/24291e4b-93d2-47ad-80a8-bf3c395319b9 > (0.00/100%) > > > (10.00/100%) > > > Rbd bench-write shows speed of ~21MB/s. > > [root@cephlarge ~]# rbd bench-write image01 --pool=rbdbench > bench-write io_size 4096 io_threads 16 bytes 1073741824 pattern sequential > SEC OPS OPS/SEC BYTES/SEC > 2 6780 3133.53 12834946.35 > 3 6831 1920.65 7866998.17 > 4 8896 2040.50 8357871.83 > 5 13058 2562.61 10496432.34 > 6 17225 2836.78 11619432.99 > 7 20345 2736.84 11210076.25 > 8 23534 3761.57 15407392.94 > 9 25689 3601.35 14751109.98 > 10 29670 3391.53 13891695.57 > 11 33169 3218.29 13182107.64 > 12 36356 3135.34 12842344.21 > 13 38431 2972.62 12175863.99 > 14 47780 4389.77 17980497.11 > 15 55452 5156.40 21120627.26 > 16 59298 4772.32 19547440.33 > 17 61437 5151.20 21099315.94 > 18 67702 5861.64 24009295.97 > 19 77086 5895.03 24146032.34 > 20 85474 5936.09 24314243.88 > 21 93848 7499.73 30718898.25 > 22 100115 7783.39 31880760.34 > 23 105405 7524.76 30821410.70 > 24 111677 6797.12 27841003.78 > 25 116971 6274.51 25700386.48 > 26 121156 5468.77 22400087.81 > 27 126484 5345.83 21896515.02 > 28 137937 6412.41 26265239.30 > 29 143229 6347.28 25998461.13 > 30 149505 6548.76 26823729.97 > 31 159978 7815.37 32011752.09 > 32 171431 8821.65 36133479.15 > 33 181084 8795.28 36025472.27 > 35 182856 6322.41 25896605.75 > 36 186891 5592.25 22905872.73 > 37 190906 4876.30 19973339.07 > 38 190943 3076.87 12602853.89 > 39 190974 1536.79 6294701.64 > 40 195323 2344.75 9604081.07 > 41 198479 2703.00 11071492.89 > 42 208893 3918.55 16050365.70 > 43 214172 4702.42 19261091.89 > 44 215263 5167.53 21166212.98 > 45 219435 5392.57 22087961.94 > 46 225731 5242.85 21474728.85 > 47 234101 5009.43 20518607.70 > 48 243529 6326.00 25911280.08 > 49 254058 7944.90 32542315.10 > elapsed: 50 ops: 262144 ops/sec: 5215.19 bytes/sec: 21361431.86 > [root@cephlarge ~]# > > This CEPH deployment has 2 OSDs. > > It would be of great help if anyone can give me pointers. > > -- > Regards, > mahesh j > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Jason _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |