What's the assumption that suggests a limit of 64 blobs / 4MB ? Are you assuming a 64K blobsize?? That certainly won't be the case for flash. Allen Samuels SanDisk |a Western Digital brand 2880 Junction Avenue, San Jose, CA 95134 T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > -----Original Message----- > From: Igor Fedotov [mailto:ifedotov@xxxxxxxxxxxx] > Sent: Friday, June 10, 2016 9:51 AM > To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Sage Weil > <sweil@xxxxxxxxxx>; Somnath Roy <Somnath.Roy@xxxxxxxxxxx> > Cc: Mark Nelson <mnelson@xxxxxxxxxx>; Manavalan Krishnan > <Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph- > devel@xxxxxxxxxxxxxxx> > Subject: Re: RocksDB tuning > > An update: > > I found that my previous results were invalid - SyntheticWorkloadState had > an odd swap for offset > len case... Made a brief fix. > > Now onode size with csum raises up to 38K, without csum - 28K. > > For csum case there is 350 lextents and about 170 blobs > > For no csum - 343 lextents and about 170 blobs. > > (blobs counting is very inaccurate!) > > Potentially we shouldn't have >64 blobs per 4M thus looks like some issues in > the write path... > > And CSum vs. NoCsum differenct looks pretty consistent - 170 blobs * 4 byte > * 16 values = 10880 > > Branch's @github been updated with corresponding fixes. > > Thanks, > Igor. > > On 10.06.2016 19:06, Allen Samuels wrote: > > Let's see, 4MB is 2^22 bytes. If we storage a checksum for each 2^12 bytes > that's 2^10 checksums at 2^2 bytes each is 2^12 bytes. > > > > So with optimal encoding, the checksum baggage shouldn't be more than > 4KB per oNode. > > > > But you're seeing 13K as the upper bound on the onode size. > > > > In the worst case, you'll need at least another block address (8 bytes > currently) and length (another 8 bytes) [though as I point out, the length is > something that can be optimized out] So worst case, this encoding would be > an addition 16KB per onode. > > > > I suspect you're not at the worst-case yet :) > > > > Allen Samuels > > SanDisk |a Western Digital brand > > 2880 Junction Avenue, Milpitas, CA 95134 > > T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > > > > > >> -----Original Message----- > >> From: Igor Fedotov [mailto:ifedotov@xxxxxxxxxxxx] > >> Sent: Friday, June 10, 2016 8:58 AM > >> To: Sage Weil <sweil@xxxxxxxxxx>; Somnath Roy > >> <Somnath.Roy@xxxxxxxxxxx> > >> Cc: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Mark Nelson > >> <mnelson@xxxxxxxxxx>; Manavalan Krishnan > >> <Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph- > >> devel@xxxxxxxxxxxxxxx> > >> Subject: Re: RocksDB tuning > >> > >> Just modified store_test synthetic test case to simulate many random 4K > >> writes to 4M object. > >> > >> With default settings ( crc32c + 4K block) onode size varies from 2K to > ~13K > >> > >> with disabled crc it's ~500 - 1300 bytes. > >> > >> > >> Hence the root cause seems to be in csum array. > >> > >> > >> Here is the updated branch: > >> > >> https://github.com/ifed01/ceph/tree/wip-bluestore-test-size > >> > >> > >> Thanks, > >> > >> Igor > >> > >> > >> On 10.06.2016 18:40, Sage Weil wrote: > >>> On Fri, 10 Jun 2016, Somnath Roy wrote: > >>>> Just turning off checksum with the below param is not helping, I > >>>> still need to see the onode size though by enabling debug..Do I need > >>>> to mkfs > >>>> (Sage?) as it is still holding checksum of old data I wrote ? > >>> Yeah.. you'll need to mkfs to blow away the old onodes and blobs with > >>> csum data. > >>> > >>> As Allen pointed out, this is only part of the problem.. but I'm > >>> curious how much! > >>> > >>>> bluestore_csum = false > >>>> bluestore_csum_type = none > >>>> > >>>> Here is the snippet of 'dstat'.. > >>>> > >>>> ----total-cpu-usage---- -dsk/total- -net/total- ---paging--> > >>>> usr sys idl wai hiq siq| read writ| recv send| in out > > >>>> 41 14 36 5 0 4| 138M 841M| 212M 145M| 0 0 > > >>>> 42 14 35 5 0 4| 137M 855M| 213M 147M| 0 0 > > >>>> 40 14 38 5 0 3| 143M 815M| 209M 144M| 0 0 > > >>>> 40 14 38 5 0 3| 137M 933M| 194M 134M| 0 0 > > >>>> 42 15 34 5 0 4| 133M 918M| 220M 151M| 0 0 > > >>>> 35 13 43 6 0 3| 147M 788M| 194M 134M| 0 0 > > >>>> 31 11 49 6 0 3| 157M 713M| 151M 104M| 0 0 > > >>>> 39 14 38 5 0 4| 139M 836M| 246M 169M| 0 0 > > >>>> 40 14 38 5 0 3| 139M 845M| 204M 140M| 0 0 > > >>>> 40 14 37 5 0 4| 149M 743M| 210M 144M| 0 0 > > >>>> 42 14 35 5 0 4| 143M 852M| 216M 150M| 0 0 > > >>>> For example, what last entry is saying that system (with 8 osds) is > >> receiving 216M of data over network and in response to that it is writing > total > >> of 852M of data and reading 143M of data. At this time FIO on client side is > >> reporting ~35K 4K RW iops. > >>>> Now, after a min or so, the throughput goes down to barely 1K from > FIO > >> (and very bumpy) and here is the 'dstat' snippet at that time.. > >>>> ----total-cpu-usage---- -dsk/total- -net/total- ---paging--> > >>>> usr sys idl wai hiq siq| read writ| recv send| in out > > >>>> 2 1 83 14 0 0| 220M 58M|4346k 3002k| 0 0 > > >>>> 2 1 82 14 0 0| 223M 60M|4050k 2919k| 0 0 > > >>>> 3 1 82 13 0 0| 217M 63M|6403k 4306k| 0 0 > > >>>> 2 1 83 14 0 0| 226M 54M|2126k 1497k| 0 0 > > >>>> > >>>> So, system is barely receiving anything (~2M) but still writing ~54M of > data > >> and reading 226M of data from disk. > >>>> After killing fio script , here is the 'dstat' output.. > >>>> > >>>> ----total-cpu-usage---- -dsk/total- -net/total- ---paging--> > >>>> usr sys idl wai hiq siq| read writ| recv send| in out > > >>>> 2 1 86 12 0 0| 186M 66M| 28k 26k| 0 0 > > >>>> 2 1 86 12 0 0| 201M 78M| 20k 21k| 0 0 > > >>>> 2 1 85 12 0 0| 230M 100M| 24k 24k| 0 0 > > >>>> 2 1 85 12 0 0| 206M 78M| 21k 20k| 0 0 > > >>>> > >>>> Not receiving anything from client but still writing 78M of data and > 206M > >> of read. > >>>> Clearly, it is an effect of rocksdb compaction that stalling IO and even if > we > >> increased compaction thread (and other tuning), compaction is not able to > >> keep up with incoming IO. > >>>> Thanks & Regards > >>>> Somnath > >>>> > >>>> -----Original Message----- > >>>> From: Allen Samuels > >>>> Sent: Friday, June 10, 2016 8:06 AM > >>>> To: Sage Weil > >>>> Cc: Somnath Roy; Mark Nelson; Manavalan Krishnan; Ceph > Development > >>>> Subject: RE: RocksDB tuning > >>>> > >>>>> -----Original Message----- > >>>>> From: Sage Weil [mailto:sweil@xxxxxxxxxx] > >>>>> Sent: Friday, June 10, 2016 7:55 AM > >>>>> To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx> > >>>>> Cc: Somnath Roy <Somnath.Roy@xxxxxxxxxxx>; Mark Nelson > >>>>> <mnelson@xxxxxxxxxx>; Manavalan Krishnan > >>>>> <Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph- > >>>>> devel@xxxxxxxxxxxxxxx> > >>>>> Subject: RE: RocksDB tuning > >>>>> > >>>>> On Fri, 10 Jun 2016, Allen Samuels wrote: > >>>>>> Checksums are definitely a part of the problem, but I suspect the > >>>>>> smaller part of the problem. This particular use-case (random 4K > >>>>>> overwrites without the WAL stuff) is the worst-case from an > >>>>>> encoding perspective and highlights the inefficiency in the current > >> code. > >>>>>> As has been discussed earlier, a specialized encode/decode > >>>>>> implementation for these data structures is clearly called for. > >>>>>> > >>>>>> IMO, you'll be able to cut the size of this by AT LEAST a factor of > >>>>>> 3 or > >>>>>> 4 without a lot of effort. The price will be somewhat increase CPU > >>>>>> cost for the serialize/deserialize operation. > >>>>>> > >>>>>> If you think of this as an application-specific data compression > >>>>>> problem, here is a short list of potential compression opportunities. > >>>>>> > >>>>>> (1) Encoded sizes and offsets are 8-byte byte values, converting > >>>>>> these too > >>>>> block values will drop 9 or 12 bits from each value. Also, the > >>>>> ranges for these values is usually only 2^22 -- often much less. > >>>>> Meaning that there's 3-5 bytes of zeros at the top of each word that > can > >> be dropped. > >>>>>> (2) Encoded device addresses are often less than 2^32, meaning > >>>>>> there's 3-4 > >>>>> bytes of zeros at the top of each word that can be dropped. > >>>>>> (3) Encoded offsets and sizes are often exactly "1" block, clever > >>>>>> choices of > >>>>> formatting can eliminate these entirely. > >>>>>> IMO, an optimized encoded form of the extent table will be around > >>>>>> 1/4 of the current encoding (for this use-case) and will likely > >>>>>> result in an Onode that's only 1/3 of the size that Somnath is seeing. > >>>>> That will be true for the lextent and blob extent maps. I'm > >>>>> guessing this is a small part of the ~5K somnath saw. If his > >>>>> objects are 4MB then 4KB of it > >>>>> (80%) is the csum_data vector, which is a flat vector of > >>>>> u32 values that are presumably not very compressible. > >>>> I don't think that's what Somnath is seeing (obviously some data here > will > >> sharpen up our speculations). But in his use case, I believe that he has a > >> separate blob and pextent for each 4K write (since it's been subjected to > >> random 4K overwrites), that means somewhere in the data structures at > >> least one address and one length for each of the 4K blocks (and likely > much > >> more in the lextent and blob maps as you alluded to above). The encoding > of > >> just this information alone is larger than the checksum data. > >>>>> We could perhaps break these into a separate key or keyspace.. > >>>>> That'll give rocksdb a bit more computation work to do (for a custom > >>>>> merge operator, probably, to update just a piece of the value) but > >>>>> for a 4KB value I'm not sure it's big enough to really help. Also > >>>>> we'd lose locality, would need a second get to load csum metadata on > >> read, etc. > >>>>> :/ I don't really have any good ideas here. > >>>>> > >>>>> sage > >>>>> > >>>>> > >>>>>> Allen Samuels > >>>>>> SanDisk |a Western Digital brand > >>>>>> 2880 Junction Avenue, Milpitas, CA 95134 > >>>>>> T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > >>>>>> > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Sage Weil [mailto:sweil@xxxxxxxxxx] > >>>>>>> Sent: Friday, June 10, 2016 2:35 AM > >>>>>>> To: Somnath Roy <Somnath.Roy@xxxxxxxxxxx> > >>>>>>> Cc: Mark Nelson <mnelson@xxxxxxxxxx>; Allen Samuels > >>>>>>> <Allen.Samuels@xxxxxxxxxxx>; Manavalan Krishnan > >>>>>>> <Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph- > >>>>>>> devel@xxxxxxxxxxxxxxx> > >>>>>>> Subject: RE: RocksDB tuning > >>>>>>> > >>>>>>> On Fri, 10 Jun 2016, Somnath Roy wrote: > >>>>>>>> Sage/Mark, > >>>>>>>> I debugged the code and it seems there is no WAL write going on > >>>>>>>> and > >>>>>>> working as expected. But, in the process, I found that onode size > >>>>>>> it is > >>>>> writing > >>>>>>> to my environment ~7K !! See this debug print. > >>>>>>>> 2016-06-09 15:49:24.710149 7f7732fe3700 20 > >>>>>>> bluestore(/var/lib/ceph/osd/ceph-0) onode > >>>>>>> #1:7d3c6423:::rbd_data.10186b8b4567.0000000000070cd4:head# is > >> 7518 > >>>>>>>> This explains why so much data going to rocksdb I guess. Once > >>>>>>>> compaction kicks in iops I am getting is *30 times* slower. > >>>>>>>> > >>>>>>>> I have 15 osds on 8TB drives and I have created 4TB rbd image > >>>>>>>> preconditioned with 1M. I was running 4K RW test. > >>>>>>> The onode is big because of the csum metdata. Try setting > >>>>>>> 'bluestore > >>>>> csum > >>>>>>> type = none' and see if that is the entire reason or if something > >>>>>>> else is > >>>>> going > >>>>>>> on. > >>>>>>> > >>>>>>> We may need to reconsider the way this is stored. > >>>>>>> > >>>>>>> s > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Thanks & Regards > >>>>>>>> Somnath > >>>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: ceph-devel-owner@xxxxxxxxxxxxxxx > >>>>>>>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of > Somnath > >>>>> Roy > >>>>>>>> Sent: Thursday, June 09, 2016 8:23 AM > >>>>>>>> To: Mark Nelson; Allen Samuels; Manavalan Krishnan; Ceph > >>>>> Development > >>>>>>>> Subject: RE: RocksDB tuning > >>>>>>>> > >>>>>>>> Mark, > >>>>>>>> As we discussed, it seems there is ~5X write amp on the system > >>>>>>>> with 4K > >>>>>>> RW. Considering the amount of data going into rocksdb (and thus > >>>>>>> kicking > >>>>> of > >>>>>>> compaction so fast and degrading performance drastically) , it > >>>>>>> seems it is > >>>>> still > >>>>>>> writing WAL (?)..I used the following rocksdb option for faster > >>>>> background > >>>>>>> compaction as well hoping it can keep up with upcoming writes and > >>>>> writes > >>>>>>> won't be stalling. But, eventually, after a min or so, it is stalling io.. > >>>>>>>> bluestore_rocksdb_options = > >> > "compression=kNoCompression,max_write_buffer_number=16,min_write_ > >> > buffer_number_to_merge=3,recycle_log_file_num=16,compaction_style=k > >>>>> > CompactionStyleLevel,write_buffer_size=67108864,target_file_size_bas > >>>>> e=6 > >>>>> > >> > 7108864,max_background_compactions=31,level0_file_num_compaction_tri > >> gger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger= > >>>>> 64, > >>>>> > >> > num_levels=4,max_bytes_for_level_base=536870912,max_bytes_for_level > >>>>>>> _multiplier=8,compaction_threads=32,flusher_threads=8" > >>>>>>>> I will try to debug what is going on there.. > >>>>>>>> > >>>>>>>> Thanks & Regards > >>>>>>>> Somnath > >>>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: ceph-devel-owner@xxxxxxxxxxxxxxx > >>>>>>>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Mark > >>>>>>>> Nelson > >>>>>>>> Sent: Thursday, June 09, 2016 6:46 AM > >>>>>>>> To: Allen Samuels; Manavalan Krishnan; Ceph Development > >>>>>>>> Subject: Re: RocksDB tuning > >>>>>>>> > >>>>>>>> On 06/09/2016 08:37 AM, Mark Nelson wrote: > >>>>>>>>> Hi Allen, > >>>>>>>>> > >>>>>>>>> On a somewhat related note, I wanted to mention that I had > >>>>> forgotten > >>>>>>>>> that chhabaremesh's min_alloc_size commit for different media > >>>>>>>>> types was committed into master: > >>>>>>>>> > >>>>>>>>> > >> > https://github.com/ceph/ceph/commit/8185f2d356911274ca679614611dc335 > >>>>>>>>> e3 > >>>>>>>>> efd187 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> IE those tests appear to already have been using a 4K min alloc > >>>>>>>>> size due to non-rotational NVMe media. I went back and > verified > >>>>>>>>> that explicitly changing the min_alloc size (in fact all of them > >>>>>>>>> to be > >>>>>>>>> sure) to 4k does not change the behavior from graphs I showed > >>>>>>>>> yesterday. The rocksdb compaction stalls due to excessive reads > >>>>>>>>> appear (at least on the > >>>>>>>>> surface) to be due to metadata traffic during heavy small > random > >>>>> writes. > >>>>>>>> Sorry, this was worded poorly. Traffic due to compaction of > >>>>>>>> metadata > >>>>> (ie > >>>>>>> not leaked WAL data) during small random writes. > >>>>>>>> Mark > >>>>>>>> > >>>>>>>>> Mark > >>>>>>>>> > >>>>>>>>> On 06/08/2016 06:52 PM, Allen Samuels wrote: > >>>>>>>>>> Let's make a patch that creates actual Ceph parameters for > >>>>>>>>>> these things so that we don't have to edit the source code in > the > >> future. > >>>>>>>>>> > >>>>>>>>>> Allen Samuels > >>>>>>>>>> SanDisk |a Western Digital brand > >>>>>>>>>> 2880 Junction Avenue, San Jose, CA 95134 > >>>>>>>>>> T: +1 408 801 7030| M: +1 408 780 6416 > >>>>>>>>>> allen.samuels@xxxxxxxxxxx > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph- > devel- > >>>>>>>>>>> owner@xxxxxxxxxxxxxxx] On Behalf Of Manavalan Krishnan > >>>>>>>>>>> Sent: Wednesday, June 08, 2016 3:10 PM > >>>>>>>>>>> To: Mark Nelson <mnelson@xxxxxxxxxx>; Ceph > Development > >>>>> <ceph- > >>>>>>>>>>> devel@xxxxxxxxxxxxxxx> > >>>>>>>>>>> Subject: RocksDB tuning > >>>>>>>>>>> > >>>>>>>>>>> Hi Mark > >>>>>>>>>>> > >>>>>>>>>>> Here are the tunings that we used to avoid the IOPs > choppiness > >>>>>>>>>>> caused by rocksdb compaction. > >>>>>>>>>>> > >>>>>>>>>>> We need to add the following options in > src/kv/RocksDBStore.cc > >>>>>>>>>>> before rocksdb::DB::Open in RocksDBStore::do_open > >>>>>>> opt.IncreaseParallelism(16); > >>>>>>>>>>> opt.OptimizeLevelStyleCompaction(512 * 1024 * 1024); > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Thanks > >>>>>>>>>>> Mana > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> PLEASE NOTE: The information contained in this electronic > mail > >>>>>>>>>>> message is intended only for the use of the designated > >>>>>>>>>>> recipient(s) named above. > >>>>>>>>>>> If the > >>>>>>>>>>> reader of this message is not the intended recipient, you are > >>>>>>>>>>> hereby notified that you have received this message in error > >>>>>>>>>>> and that any review, dissemination, distribution, or copying > >>>>>>>>>>> of this message is strictly prohibited. If you have received > >>>>>>>>>>> this communication in error, please notify the sender by > >>>>>>>>>>> telephone or e-mail (as shown > >>>>>>>>>>> above) immediately and destroy any and all copies of this > >>>>>>>>>>> message in your possession (whether hard copies or > >>>>>>>>>>> electronically stored copies). > >>>>>>>>>>> -- > >>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe > >>>>>>>>>>> ceph- > >>>>> devel" > >>>>>>>>>>> in the > >>>>>>>>>>> body of a message to majordomo@xxxxxxxxxxxxxxx More > >>>>> majordomo > >>>>>>> info > >>>>>>>>>>> at http://vger.kernel.org/majordomo-info.html > >>>>>>>>>> -- > >>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph- > >> devel" > >>>>>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More > >>>>>>>>>> majordomo info at http://vger.kernel.org/majordomo- > info.html > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph- > >> devel" > >>>>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More > >>>>>>> majordomo > >>>>>>>>> info at http://vger.kernel.org/majordomo-info.html > >>>>>>>> -- > >>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph- > devel" > >>>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More > >>>>>>> majordomo > >>>>>>>> info at http://vger.kernel.org/majordomo-info.html > >>>>>>>> PLEASE NOTE: The information contained in this electronic mail > >>>>>>>> message > >>>>> is > >>>>>>> intended only for the use of the designated recipient(s) named > >>>>>>> above. If > >>>>> the > >>>>>>> reader of this message is not the intended recipient, you are > >>>>>>> hereby > >>>>> notified > >>>>>>> that you have received this message in error and that any review, > >>>>>>> dissemination, distribution, or copying of this message is > >>>>>>> strictly > >>>>> prohibited. If > >>>>>>> you have received this communication in error, please notify the > >>>>>>> sender > >>>>> by > >>>>>>> telephone or e-mail (as shown above) immediately and destroy > any > >>>>>>> and > >>>>> all > >>>>>>> copies of this message in your possession (whether hard copies or > >>>>>>> electronically stored copies). > >>>>>>>> -- > >>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph- > devel" > >>>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More > >>>>>>> majordomo > >>>>>>>> info at http://vger.kernel.org/majordomo-info.html > >>>>>>>> -- > >>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph- > devel" > >>>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More > >>>>>>> majordomo > >>>>>>>> info at http://vger.kernel.org/majordomo-info.html > >>>>>>>> > >>>>>>>> > >>>>>> -- > >>>>>> To unsubscribe from this list: send the line "unsubscribe > >>>>>> ceph-devel" in the body of a message to > majordomo@xxxxxxxxxxxxxxx > >>>>>> More majordomo info at http://vger.kernel.org/majordomo- > info.html > >>>>>> > >>>>>> > >>>> PLEASE NOTE: The information contained in this electronic mail > message is > >> intended only for the use of the designated recipient(s) named above. If > the > >> reader of this message is not the intended recipient, you are hereby > notified > >> that you have received this message in error and that any review, > >> dissemination, distribution, or copying of this message is strictly > prohibited. If > >> you have received this communication in error, please notify the sender > by > >> telephone or e-mail (as shown above) immediately and destroy any and > all > >> copies of this message in your possession (whether hard copies or > >> electronically stored copies). > >>>> > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More > >> majordomo > >>> info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
