Thanks Mark! Yes, we are hitting probably some other bottleneck, will try to find that out... -----Original Message----- From: Mark Nelson [mailto:mnelson@xxxxxxxxxx] Sent: Thursday, December 22, 2016 2:23 PM To: Somnath Roy; Sage Weil Cc: ceph-devel Subject: Re: Bluestore performance bottleneck Hi Somnath, Based on your testing, I went through and did some single OSD tests with master (pre-extent patch) with different sharding target/max settings on one of our NVMe nodes: https://drive.google.com/uc?export=download&id=0B2gTBZrkrnpZb2ZWcVZHbzJRVHc What I saw is that for 4k min_alloc/max_alloc/max_blob sizes, decreasing the sharding target/max helped to a point where it started hurting more than it helped. The peak is probably somewhere between 100/200 and 200/400, though we may want to error on higher values rather than lower. RSS memory usage of the OSD increased dramatically as the target/max sizes shrunk. CPU usage didn't change dramatically, though was a little lower at the extremes where performance was lowest. For reference, 16k min_alloc pegs at around 20K IOPS in this test as well, meaning that I think we may be hitting a common bottleneck holding us to 20K write IOPS per OSD. I noticed that as the target/max size shrunk, certain code paths became more heavily worked however. RocksDB generally took about a 2x larger percentage of the used CPU, with a lot of it going toward CRC calculations. We also spent a lot more time in BlueStore::ExtentMap::init_shards doing key appends, and triming the TwoQCache. Given that the IOPS dropped precipitously, while overall CPU usage remained high and memory usage increased dramatically, there may be some opportunities to tune these areas of the code. One example might be to avoid doing string appends in the key encoding by switching to a different data structure. FWIW, I did not notice any resharding during the steady state for any of these tests. Mark On 12/21/2016 08:25 PM, Somnath Roy wrote: > << How many blobs are in each shard, and how many shards are there? > Is there any easy way to find out these other than adding some log ? > > > -----Original Message----- > From: Sage Weil [mailto:sweil@xxxxxxxxxx] > Sent: Wednesday, December 21, 2016 5:30 PM > To: Somnath Roy > Cc: ceph-devel > Subject: RE: Bluestore performance bottleneck > > How many blobs are in each shard, and how many shards are there? > > If we go this route, I think we'll want a larger threshold for the inline blobs (stored in the onode key) so that "normal" objects without a zillion blobs still fit in one key... > > sage > > On Thu, 22 Dec 2016, Somnath Roy wrote: > >> Ok, *205 bytes* reduction per IO by removing extents.. Thanks ! >> >> 2016-12-21 20:00:07.701845 7fcff8412700 30 submit_transaction Rocksdb transaction: >> Put( Prefix = M key = >> 0x00000000000006fc'.0000000009.00000000000000001051' Value size = >> 182) Put( Prefix = M key = 0x00000000000006fc'._fastinfo' Value size >> = 186) Put( Prefix = O key = >> 0x7f8000000000000001b56f21ae217262'd_data.10046b8b4567.00000000000028 >> c 6!='0xfffffffffffffffeffffffffffffffff6f0005f000'x' Value size = >> 45) Put( Prefix = O key = >> 0x7f8000000000000001b56f21ae217262'd_data.10046b8b4567.00000000000028 >> c 6!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1300) Merge( >> Prefix = b key = 0x0000000c8cd00000 Value size = 16) Merge( Prefix = >> b key = 0x0000001067700000 Value size = 16) >> >> >> >> -----Original Message----- >> From: ceph-devel-owner@xxxxxxxxxxxxxxx >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Somnath Roy >> Sent: Wednesday, December 21, 2016 4:39 PM >> To: Sage Weil >> Cc: ceph-devel >> Subject: RE: Bluestore performance bottleneck >> >> Yeah, make sense I missed it..I will remove extents and see how much we can save. >> But, why a 4K length/offset is started touching 2 shards now if shard is smaller is still unclear to me ? >> >> Thanks & Regards >> Somnath >> >> -----Original Message----- >> From: Sage Weil [mailto:sweil@xxxxxxxxxx] >> Sent: Wednesday, December 21, 2016 4:21 PM >> To: Somnath Roy >> Cc: ceph-devel >> Subject: RE: Bluestore performance bottleneck >> >> On Thu, 22 Dec 2016, Somnath Roy wrote: >>> Sage, >>> By reducing shard size I am able to improve bluestore +rocks performance by 80% for 60G image. Will do detailed analysis on bigger images. >>> >>> Here is what I changed to reduce decode_some() overhead. It is now looping 5 times instead of default 33. >>> >>> bluestore_extent_map_shard_max_size = 50 >>> bluestore_extent_map_shard_target_size = 45 >>> >>> Fio output : >>> ------------- >>> Default: >>> ---------- >>> >>> rbd_iodepth32: (groupid=0, jobs=5): err= 0: pid=34420: Wed Dec 21 17:30:50 2016 >>> write: io=114208MB, bw=63615KB/s, iops=15903, runt=1838384msec >>> slat (usec): min=3, max=3878, avg=15.63, stdev= 8.78 >>> clat (usec): min=618, max=176057, avg=20092.56, stdev=21017.34 >>> lat (usec): min=642, max=176063, avg=20108.20, stdev=21017.29 >>> clat percentiles (usec): >>> | 1.00th=[ 1416], 5.00th=[ 2928], 10.00th=[ 4320], 20.00th=[ 6112], >>> | 30.00th=[ 7648], 40.00th=[ 9152], 50.00th=[10944], 60.00th=[13760], >>> | 70.00th=[18816], 80.00th=[32384], 90.00th=[55552], 95.00th=[68096], >>> | 99.00th=[87552], 99.50th=[94720], 99.90th=[121344], 99.95th=[129536], >>> | 99.99th=[142336] >>> >>> Small shards: >>> ---------------- >>> >>> rbd_iodepth32: (groupid=0, jobs=5): err= 0: pid=34767: Wed Dec 21 18:50:30 2016 >>> write: io=186917MB, bw=110585KB/s, iops=27646, runt=1730819msec >>> slat (usec): min=2, max=1447, avg=14.95, stdev= 7.64 >>> clat (usec): min=531, max=541140, avg=11547.88, stdev=8131.72 >>> lat (usec): min=544, max=541156, avg=11562.83, stdev=8131.73 >>> clat percentiles (msec): >>> | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], >>> | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], >>> | 70.00th=[ 14], 80.00th=[ 17], 90.00th=[ 22], 95.00th=[ 27], >>> | 99.00th=[ 37], 99.50th=[ 40], 99.90th=[ 51], 99.95th=[ 75], >>> | 99.99th=[ 208] >>> >>> >>> >>> *But* here is the overhead I am seeing which I don't quite understand. See the per io metadata overhead for onode/shard is with smaller shard is ~30% more. >>> >>> Default: >>> ---------- >>> >>> 2016-12-21 17:22:31.693503 7f18aa7c1700 30 submit_transaction Rocksdb transaction: >>> Put( Prefix = M key = >>> 0x000000000000048a'.0000000009.00000000000000019093' Value size = >>> 182) Put( Prefix = M key = 0x000000000000048a'._fastinfo' Value size >>> = 186) Put( Prefix = O key = >>> 0x7f8000000000000001dc3f23'`!rbd_data.10046b8b4567.00000000000024b1!=' >>> 0xfffffffffffffffeffffffffffffffff6f0014c000'x' Value size = 669) >>> Put( Prefix = O key = >>> 0x7f8000000000000001dc3f23'`!rbd_data.10046b8b4567.00000000000024b1!=' >>> 0xfffffffffffffffeffffffffffffffff'o' Value size = 462) Merge( >>> Prefix = b key = 0x000000069ae00000 Value size = 16) Merge( Prefix = >>> b key = >>> 0x0000001330d00000 Value size = 16) >>> >>> >>> Smaller shard: >>> ----------------- >>> >>> 2016-12-21 18:52:18.564423 7f3e5d167700 30 submit_transaction Rocksdb transaction: >>> Put( Prefix = M key = >>> 0x0000000000000691'.0000000009.00000000000000057248' Value size = >>> 182) Put( Prefix = M key = 0x0000000000000691'._fastinfo' Value size >>> = 186) Put( Prefix = O key = >>> 0x7f8000000000000001427b9fe9217262'd_data.10046b8b4567.0000000000003 >>> 48 f!='0xfffffffffffffffeffffffffffffffff6f00195000'x' Value size = >>> 45) Put( Prefix = O key = >>> 0x7f8000000000000001427b9fe9217262'd_data.10046b8b4567.0000000000003 >>> 48 f!='0xfffffffffffffffeffffffffffffffff6f0019a000'x' Value size = >>> 45) Put( Prefix = O key = >>> 0x7f8000000000000001427b9fe9217262'd_data.10046b8b4567.0000000000003 >>> 48 f!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1505) >>> Merge( Prefix = b key = 0x00000006b9780000 Value size = 16) Merge( >>> Prefix = b key = 0x0000000c64f00000 Value size = 16) >>> >>> >>> *And* lot of times I am seeing 2 shards are written compare to default. This will be a problem for ZS , may not be for Rocks. >>> >>> Initially, I thought blobs are spanning, but, it seems not the case. See the below log snippet , it seems onode itself is bigger now. >>> >>> 2016-12-21 18:40:27.734044 7f3e43934700 20 bluestore(/var/lib/ceph/osd/ceph-0) onode #1:d1bc6a86:::rbd_data.10046b8b4567.00000000000036a2:head# is 1505 (1503 bytes onode + 2 bytes spanning blobs + 0 bytes inline extents) >>> >>> Any idea what's going on ? >> >> The onode has a list of the shards. Since there are more, the onode is bigger. I wasn't really expecting the shard count to be that high. The structure is: >> >> struct shard_info { >> uint32_t offset = 0; ///< logical offset for start of shard >> uint32_t bytes = 0; ///< encoded bytes >> uint32_t extents = 0; ///< extents >> DENC(shard_info, v, p) { >> denc_varint(v.offset, p); >> denc_varint(v.bytes, p); >> denc_varint(v.extents, p); >> } >> void dump(Formatter *f) const; >> }; >> vector<shard_info> extent_map_shards; ///< extent map shards (if >> any) >> >> The offset is the important piece. The byte and extent counts aren't that important... they're mostly there so that a future reshard operation can be more clever (merging or splitting adjacent shards instead of resharding everything). Well, the bytes field is currently used, but extents is not at all. We could just drop that field now and add it (or something else) back in later if/when we need it... >> >> sage >> >> >>> >>> Thanks & Regards >>> Somnath >>> >>> >>> -----Original Message----- >>> From: Somnath Roy >>> Sent: Friday, December 16, 2016 7:23 PM >>> To: Sage Weil (sweil@xxxxxxxxxx) >>> Cc: 'ceph-devel' >>> Subject: RE: Bluestore performance bottleneck >>> >>> Sage, >>> Some update on this. Without decode_some within fault_range() I am able to drive Bluestore + rocksdb close to ~38K iops compare to ~20K iops with decode_some. I had to disable data write because I am skipping decode but in this device data write is not a bottleneck. I have seen enabling/disabling data write is giving similar result. So, in NVME device if we can optimize decode_some() for performance Bluestore performance should bump up by ~2X. >>> I did some print around decode_some() and it seems it is taking ~60-121 micro sec to finish depending on bytes to decode. >>> >>> Thanks & Regards >>> Somnath >>> >>> -----Original Message----- >>> From: Somnath Roy >>> Sent: Thursday, December 15, 2016 7:30 PM >>> To: Sage Weil (sweil@xxxxxxxxxx) >>> Cc: ceph-devel >>> Subject: Bluestore performance bottleneck >>> >>> Sage, >>> Today morning I was talking about 2x performance drop for Bluestore without data/db writes for 1G vs 60G volumes and it turn out the decode_some() is the culprit for that. Presently, I am drilling down that function to identify what exactly causing this issue, but, most probably it is blob decode and le->blob->get_ref() combination. Will confirm that soon. If we can fix that we should be able to considerably bump up end-to-end pick performance with rocks/ZS on faster NVME. Slower devices most likely we will not be able to see any benefits other than saving some cpu cost. >>> >>> Thanks & Regards >>> Somnath >>> >>> >>> >>> ________________________________ >>> >>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo >>> info at http://vger.kernel.org/majordomo-info.html >>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
