Sorry to those that recieve this twice but the initial message bounced from the list because some HTML slipped in (thanks Google!). On Tue, Jun 7, 2016 at 1:32 PM, Mark Nelson <mnelson@xxxxxxxxxx> wrote: > > After running a bisect I'm semi-confident that this was a result of the rocksdb pull up to latest master in #9466: > > https://github.com/ceph/ceph/commit/f485596dd0656b01ba130b4e5040fcd4c02d1a31 I'm hitting something very similar while running unittest_rocksdb_option_static. (gdb) bt #0 __atomic_add_dispatch (__val=1, __mem=0x7ffff5f774d5 <operator new(unsigned long)+37>) at /usr/include/c++/4.8.3/ext/atomicity.h:96 #1 _M_add_ref_copy (this=0x7ffff5f774cd <operator new(unsigned long)+29>) at /usr/include/c++/4.8.3/bits/shared_ptr_base.h:131 #2 __shared_count (__r=..., this=0x7fffffffd180) at /usr/include/c++/4.8.3/bits/shared_ptr_base.h:553 #3 __shared_ptr (this=0x7fffffffd178) at /usr/include/c++/4.8.3/bits/shared_ptr_base.h:779 #4 shared_ptr (this=0x7fffffffd178) at /usr/include/c++/4.8.3/bits/shared_ptr.h:103 #5 rocksdb::ColumnFamilyOptions::ColumnFamilyOptions (this=0x7fffffffd170, options=...) at util/options.cc:201 #6 0x0000555555804c6a in rocksdb::GetOptionsFromString (base_options=..., opts_str="compression=kNoCompression", new_options=new_options@entry=0x7fffffffdad0) at util/options_helper.cc:1327 #7 0x0000555555734b19 in RocksDBStore::ParseOptionsFromString (this=0x55555deefa40, opt_str=..., opt=...) at kv/RocksDBStore.cc:188 #8 0x000055555572b361 in RocksDBOption_simple_Test::TestBody (this=<optimized out>) at test/objectstore/TestRocksdbOptionParse.cc:31 #9 0x0000555555874b14 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x55555deee060, method=&virtual testing::Test::TestBody(), location=0x55555596c7db "the test body") at ./src/gtest.cc:2078 #10 0x000055555586fc16 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x55555deee060, method=&virtual testing::Test::TestBody(), location=0x55555596c7db "the test body") at ./src/gtest.cc:2114 #11 0x000055555585688f in testing::Test::Run (this=0x55555deee060) at ./src/gtest.cc:2151 #12 0x0000555555857088 in testing::TestInfo::Run (this=0x55555de581e0) at ./src/gtest.cc:2326 #13 0x000055555585774c in testing::TestCase::Run (this=0x55555de58600) at ./src/gtest.cc:2444 #14 0x000055555585e214 in testing::internal::UnitTestImpl::RunAllTests (this=0x55555de582a0) at ./src/gtest.cc:4315 #15 0x0000555555875efa in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55555de582a0, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x55555585df7c <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55555596d030 "auxiliary test code (environments or event listeners)") at ./src/gtest.cc:2078 #16 0x0000555555870a70 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55555de582a0, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x55555585df7c <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55555596d030 "auxiliary test code (environments or event listeners)") at ./src/gtest.cc:2114 #17 0x000055555585cde4 in testing::UnitTest::Run (this=0x555555c569a0 <testing::UnitTest::GetInstance()::instance>) at ./src/gtest.cc:3929 #18 0x00005555556e86c6 in RUN_ALL_TESTS () at ../src/gmock/gtest/include/gtest/gtest.h:2288 #19 main (argc=1, argv=0x7fffffffe308) at test/objectstore/TestRocksdbOptionParse.cc:84 (gdb) f 5 #5 rocksdb::ColumnFamilyOptions::ColumnFamilyOptions (this=0x7fffffffd170, options=...) at util/options.cc:201 201 paranoid_file_checks(options.paranoid_file_checks), That line is part of the initialiser list for the ColumnFamilyOptions struct which has changed rescently according to git blame. (gdb) up #6 0x0000555555804c6a in rocksdb::GetOptionsFromString (base_options=..., opts_str="compression=kNoCompression", new_options=new_options@entry=0x7fffffffdad0) at util/options_helper.cc:1327 1327 } (gdb) l 1322 Status s = StringToMap(opts_str, &opts_map); 1323 if (!s.ok()) { 1324 return s; 1325 } 1326 return GetColumnFamilyOptionsFromMap(base_options, opts_map, new_options); 1327 } I believe this is related to base_options and new_options which are the same object of type rocksdb::Options. Valgrind has this to say. ==26974== Conditional jump or move depends on uninitialised value(s) ==26974== at 0x3B09A6: rocksdb::ColumnFamilyOptions::ColumnFamilyOptions(rocksdb::Options const&) (shared_ptr_base.h:552) ==26974== by 0x3B8C69: rocksdb::GetOptionsFromString(rocksdb::Options const&, std::string const&, rocksdb::Options*) (options_helper.cc:1327) ==26974== by 0x2E8B18: RocksDBStore::ParseOptionsFromString(std::string, rocksdb::Options&) (RocksDBStore.cc:188) ==26974== by 0x2DF360: RocksDBOption_simple_Test::TestBody() (TestRocksdbOptionParse.cc:31) ==26974== by 0x428B13: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078) ==26974== by 0x423C15: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114) ==26974== by 0x40A88E: testing::Test::Run() (gtest.cc:2151) ==26974== by 0x40B087: testing::TestInfo::Run() (gtest.cc:2326) ==26974== by 0x40B74B: testing::TestCase::Run() (gtest.cc:2444) ==26974== by 0x412213: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4315) ==26974== by 0x429EF9: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2078) ==26974== by 0x424A6F: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2114) ==26974== by 0x410DE3: testing::UnitTest::Run() (gtest.cc:3929) ==26974== by 0x29C6C5: main (gtest.h:2288) ==26974== Uninitialised value was created by a stack allocation ==26974== at 0x2DF1AC: RocksDBOption_simple_Test::TestBody() (options.h:1350) That's this line. 1347 // Options to control the behavior of a database (passed to DB::Open) 1348 struct Options : public DBOptions, public ColumnFamilyOptions { 1349 // Create an Options object with default values for all fields. 1350 Options() : DBOptions(), ColumnFamilyOptions() {} <----------- HERE It then gives two more errors before the crash. ==26974== Invalid read of size 8 ==26974== at 0x94162B2: memcpy@GLIBC_2.2.5 (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==26974== by 0x3B0A99: rocksdb::ColumnFamilyOptions::ColumnFamilyOptions(rocksdb::Options const&) (stl_algobase.h:372) ==26974== by 0x3B8C69: rocksdb::GetOptionsFromString(rocksdb::Options const&, std::string const&, rocksdb::Options*) (options_helper.cc:1327) ==26974== by 0x2E8B18: RocksDBStore::ParseOptionsFromString(std::string, rocksdb::Options&) (RocksDBStore.cc:188) ==26974== by 0x2DF360: RocksDBOption_simple_Test::TestBody() (TestRocksdbOptionParse.cc:31) ==26974== by 0x428B13: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078) ==26974== by 0x423C15: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114) ==26974== by 0x40A88E: testing::Test::Run() (gtest.cc:2151) ==26974== by 0x40B087: testing::TestInfo::Run() (gtest.cc:2326) ==26974== by 0x40B74B: testing::TestCase::Run() (gtest.cc:2444) ==26974== by 0x412213: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4315) ==26974== by 0x429EF9: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2078) ==26974== by 0x424A6F: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2114) ==26974== by 0x410DE3: testing::UnitTest::Run() (gtest.cc:3929) ==26974== by 0x29C6C5: main (gtest.h:2288) ==26974== Address 0x39fe1038 is 8 bytes before a block of size 4,290,772,994 alloc'd ==26974== at 0x9412965: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==26974== by 0x3B0FD7: rocksdb::ColumnFamilyOptions::ColumnFamilyOptions(rocksdb::Options const&) (new_allocator.h:104) ==26974== by 0x3B8C69: rocksdb::GetOptionsFromString(rocksdb::Options const&, std::string const&, rocksdb::Options*) (options_helper.cc:1327) ==26974== by 0x2E8B18: RocksDBStore::ParseOptionsFromString(std::string, rocksdb::Options&) (RocksDBStore.cc:188) ==26974== by 0x2DF360: RocksDBOption_simple_Test::TestBody() (TestRocksdbOptionParse.cc:31) ==26974== by 0x428B13: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078) ==26974== by 0x423C15: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114) ==26974== by 0x40A88E: testing::Test::Run() (gtest.cc:2151) ==26974== by 0x40B087: testing::TestInfo::Run() (gtest.cc:2326) ==26974== by 0x40B74B: testing::TestCase::Run() (gtest.cc:2444) ==26974== by 0x412213: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4315) ==26974== by 0x429EF9: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2078) ==26974== by 0x424A6F: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2114) ==26974== by 0x410DE3: testing::UnitTest::Run() (gtest.cc:3929) ==26974== by 0x29C6C5: main (gtest.h:2288) ==26974== Invalid read of size 8 ==26974== at 0x94162A0: memcpy@GLIBC_2.2.5 (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==26974== by 0x3B0A99: rocksdb::ColumnFamilyOptions::ColumnFamilyOptions(rocksdb::Options const&) (stl_algobase.h:372) ==26974== by 0x3B8C69: rocksdb::GetOptionsFromString(rocksdb::Options const&, std::string const&, rocksdb::Options*) (options_helper.cc:1327) ==26974== by 0x2E8B18: RocksDBStore::ParseOptionsFromString(std::string, rocksdb::Options&) (RocksDBStore.cc:188) ==26974== by 0x2DF360: RocksDBOption_simple_Test::TestBody() (TestRocksdbOptionParse.cc:31) ==26974== by 0x428B13: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078) ==26974== by 0x423C15: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114) ==26974== by 0x40A88E: testing::Test::Run() (gtest.cc:2151) ==26974== by 0x40B087: testing::TestInfo::Run() (gtest.cc:2326) ==26974== by 0x40B74B: testing::TestCase::Run() (gtest.cc:2444) ==26974== by 0x412213: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4315) ==26974== by 0x429EF9: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2078) ==26974== by 0x424A6F: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2114) ==26974== by 0x410DE3: testing::UnitTest::Run() (gtest.cc:3929) ==26974== by 0x29C6C5: main (gtest.h:2288) ==26974== Address 0x39fe1030 is 16 bytes before a block of size 4,290,772,994 alloc'd ==26974== at 0x9412965: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==26974== by 0x3B0FD7: rocksdb::ColumnFamilyOptions::ColumnFamilyOptions(rocksdb::Options const&) (new_allocator.h:104) ==26974== by 0x3B8C69: rocksdb::GetOptionsFromString(rocksdb::Options const&, std::string const&, rocksdb::Options*) (options_helper.cc:1327) ==26974== by 0x2E8B18: RocksDBStore::ParseOptionsFromString(std::string, rocksdb::Options&) (RocksDBStore.cc:188) ==26974== by 0x2DF360: RocksDBOption_simple_Test::TestBody() (TestRocksdbOptionParse.cc:31) ==26974== by 0x428B13: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078) ==26974== by 0x423C15: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114) ==26974== by 0x40A88E: testing::Test::Run() (gtest.cc:2151) ==26974== by 0x40B087: testing::TestInfo::Run() (gtest.cc:2326) ==26974== by 0x40B74B: testing::TestCase::Run() (gtest.cc:2444) ==26974== by 0x412213: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4315) ==26974== by 0x429EF9: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2078) ==26974== by 0x424A6F: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2114) ==26974== by 0x410DE3: testing::UnitTest::Run() (gtest.cc:3929) ==26974== by 0x29C6C5: main (gtest.h:2288) Note, for what it's worth, that we've done two reads of size 8 at 8 and 16 bytes before a valid block respectively. The last line that is not in teh STL is the line we eventually crash on. 1317 Status GetColumnFamilyOptionsFromString( 1318 const ColumnFamilyOptions& base_options, 1319 const std::string& opts_str, 1320 ColumnFamilyOptions* new_options) { 1321 std::unordered_map<std::string, std::string> opts_map; 1322 Status s = StringToMap(opts_str, &opts_map); 1323 if (!s.ok()) { 1324 return s; 1325 } 1326 return GetColumnFamilyOptionsFromMap(base_options, opts_map, new_options); 1327 } I think this has something to do with the changes in the ColumnFamilyOptions struct and the fact that there is a conversion happening from an Options object to a ColumnFamilyOptions object. That's the feeling I get from poking around in this for a while but I can't nail exactly what's happening here and, of course, I could very well be wrong. HTH, Brad > > > > Next need to track down the commit in rocksdb. > > Mark > > > On 06/06/2016 06:53 PM, Mark Nelson wrote: >> >> Looks like memory is being corrupted. One example: >> >> (gdb) info locals >> >> <snip> >> max_bytes_for_level_multiplier_additional = std::vector of length >> 35181114486204, capacity -33 = {<error reading variable> >> >> Running through valgrind now to see if I can track it down. >> >> Mark >> >> On 06/06/2016 11:38 AM, Somnath Roy wrote: >>> >>> Thanks Mark for confirming this ! >>> May be we pulled latest rocksdb that are causing some incompatibility >>> on option parsing ? >>> >>> Regards >>> Somnath >>> >>> -----Original Message----- >>> From: Mark Nelson [mailto:mnelson@xxxxxxxxxx] >>> Sent: Monday, June 06, 2016 9:35 AM >>> To: Somnath Roy; Sage Weil >>> Cc: Jianjian Huo; ceph-devel >>> Subject: Re: Bluestore crash in mkfs >>> >>> I'm hitting this after upgrading to master. Probably something that >>> got merged on the 2nd or 3rd. I'll see if I can track it down. >>> >>> Mark >>> >>> On 06/04/2016 09:41 PM, Somnath Roy wrote: >>>> >>>> Here is the steps I did.. >>>> 1. Git pull >>>> 2. Make clean >>>> 3. Do_autogen -j o3 >>>> 4. Make install >>>> 5. Mkcephfs to install, I still prefer it >>>> >>>> I will debug more and update >>>> >>>> Thanks & Regards >>>> Somnath >>>> >>>> Sent from my iPhone >>>> >>>>> On Jun 4, 2016, at 7:35 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote: >>>>> >>>>>> On Sat, 4 Jun 2016, Somnath Roy wrote: >>>>>> Same conf file worked for me sometimes back. I think db path/wal >>>>>> path is optional and if you don't mention anything, it will coexist >>>>>> on the block partition. >>>>>> Not sure if this is something changed recently though.. >>>>>> The trace is also not complaining about db path , it is getting a >>>>>> seg fault while parsing rocksdb option. >>>>> >>>>> >>>>> Hmm, I haven't seen this. >>>>> >>>>> Note that you don't need 'devs'. And you're right that just >>>>> specifying the block path is enough. If you're doing a manual build, >>>>> is it possible you're dynamically linking rocksdb or something? >>>>> >>>>> sage >>>>> >>>>>> >>>>>> Thanks & Regards >>>>>> Somnath >>>>>> >>>>>> -----Original Message----- >>>>>> From: Jianjian Huo [mailto:samuel.huo@xxxxxxxxx] >>>>>> Sent: Friday, June 03, 2016 9:29 PM >>>>>> To: Somnath Roy >>>>>> Cc: Sage Weil (sage@xxxxxxxxxxxx); ceph-devel >>>>>> Subject: Re: Bluestore crash in mkfs >>>>>> >>>>>>> On Fri, Jun 3, 2016 at 7:16 PM, Somnath Roy >>>>>>> <Somnath.Roy@xxxxxxxxxxx> wrote: >>>>>>> Sage, >>>>>>> Latest code in Bluestore is crashing on me during mkfs. Here is >>>>>>> the dump. >>>>>>> >>>>>>> ceph version 10.2.0-1809-gac8c40f >>>>>>> (ac8c40f3e3257517f71d2af8bd26dcd264887b7f) >>>>>>> 1: (()+0x99e65e) [0x55cf5263565e] >>>>>>> 2: (()+0x113d0) [0x7f323526f3d0] >>>>>>> 3: >>>>>>> (rocksdb::ColumnFamilyOptions::ColumnFamilyOptions(rocksdb::Options >>>>>>> const&)+0x59) [0x55cf525d4ce9] >>>>>>> 4: (rocksdb::GetOptionsFromString(rocksdb::Options const&, >>>>>>> std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>> std::allocator<char> > const&, rocksdb::Options*)+0x15a) >>>>>>> [0x55cf525dc74a] >>>>>>> 5: >>>>>>> (RocksDBStore::ParseOptionsFromString(std::__cxx11::basic_string<ch >>>>>>> ar, std::char_traits<char>, std::allocator<char> >, >>>>>>> rocksdb::Options&)+0x22e) [0x55cf524f58be] >>>>>>> 6: (RocksDBStore::init(std::__cxx11::basic_string<char, >>>>>>> std::char_traits<char>, std::allocator<char> >)+0x77) >>>>>>> [0x55cf524f5f17] >>>>>>> 7: (BlueStore::_open_db(bool)+0x274) [0x55cf5234b9e4] >>>>>>> 8: (BlueStore::mkfs()+0x899) [0x55cf52361ad9] >>>>>>> 9: (OSD::mkfs(CephContext*, ObjectStore*, >>>>>>> std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>> std::allocator<char> > const&, uuid_d, int)+0x117) [0x55cf52001427] >>>>>>> 10: (main()+0x1003) [0x55cf51f9f2e3] >>>>>>> 11: (__libc_start_main()+0xf0) [0x7f32331fa830] >>>>>>> 12: (_start()+0x29) [0x55cf51fe2719] >>>>>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >>>>>>> needed to interpret this. >>>>>>> >>>>>>> >>>>>>> I am using the following option : >>>>>>> >>>>>>> devs = /dev/sdb1 >>>>>>> bluestore_block_path = /dev/sdb2 >>>>>>> >>>>>>> Am I missing anything ? >>>>>> >>>>>> >>>>>> This is what Mark posted last time, you forgot db path? >>>>>> https://github.com/ceph/cbt/blob/master/example/bluestore/ceph.conf. >>>>>> bluestore_example >>>>>>> >>>>>>> >>>>>>> Thanks & Regards >>>>>>> Somnath >>>>>>> PLEASE NOTE: The information contained in this electronic mail >>>>>>> message is intended only for the use of the designated >>>>>>> recipient(s) named above. If the reader of this message is not the >>>>>>> intended recipient, you are hereby notified that you have received >>>>>>> this message in error and that any review, dissemination, >>>>>>> distribution, or copying of this message is strictly prohibited. >>>>>>> If you have received this communication in error, please notify >>>>>>> the sender by telephone or e-mail (as shown above) immediately and >>>>>>> destroy any and all copies of this message in your possession >>>>>>> (whether hard copies or electronically stored copies). >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More >>>>>>> majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> N?????r??y??????X??ǧv???){.n?????z?]z????ay? ʇڙ??j ??f???h????? >>>>>> ?w??? ???j:+v???w???????? ????zZ+???????j"????i >>>> >>>> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w >>> >>> j:+v w j m zZ+ ݢj" !tml= >>>> >>>> >>> N�����r��y���b�X��ǧv�^�){.n�+���z�]z���{ay� ʇڙ�,j ��f���h���z� �w��� ���j:+v���w�j�m���� ����zZ+�����ݢj"��!tml= >>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Cheers, Brad -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
