The valgrind output I posted seems to indicate an issue in the stl, which is unlikely and probably a false positive. Strange coincidence though... Another option for attacking this might be sanitizers? On Fri, Dec 23, 2016 at 11:01 AM, Willem Jan Withagen <wjw@xxxxxxxxxxx> wrote: > On 23-12-2016 01:54, Brad Hubbard wrote: >> Here's what I ran. >> >> $ valgrind --trace-children=yes --show-reachable=yes >> --track-origins=yes --read-var-info=yes --tool=memcheck >> --leak-check=full --num-callers=50 -v --log-file=unittest_denc.log >> bin/unittest_denc > > Errgh, > > Installing valgrind package also wants to install gcc. > Something I desperately want to avoid, because I otherwise import even > more diversity. > So I'll have to see if I can build it myself with clang. > > --WjW > >> Your stack shows /usr/srcs/Ceph/work/ceph/src/test/test_denc.cc:206 >> which seems to match up with this. >> >> ==10312== Mismatched free() / delete / delete [] >> ==10312== at 0x4C2ED4A: free (vg_replace_malloc.c:530) >> ==10312== by 0x162A6A: deallocate (new_allocator.h:110) >> ==10312== by 0x162A6A: deallocate (alloc_traits.h:442) >> ==10312== by 0x162A6A: _M_deallocate (stl_vector.h:178) >> ==10312== by 0x162A6A: void >> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >, >> std::allocator<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > > >:: ⤷ >> _M_emplace_back_aux<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > >>> (std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >&&) (vector.tcc:438) >> ==10312== by 0x15E0B5: push_back (stl_vector.h:933) >> ==10312== by 0x15E0B5: denc_vector_Test::TestBody() >> (test_denc.cc:207) >> ==10312== by 0x19D203: >> HandleSehExceptionsInMethodIfSupported<testing::Test, void> >> (gtest.cc:2402) >> ==10312== by 0x19D203: void >> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, >> void>(testing::Test*, void (testing::Test::*)(), char const*) >> (gtest.cc:2438) >> ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) >> ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) >> ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) >> ==10312== by 0x194536: >> testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) >> ==10312== by 0x19D6B3: >> HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool> (gtest.cc:2402) >> ==10312== by 0x19D6B3: bool >> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool>(testing::internal::UnitTestImpl*, bool >> (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) >> ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) >> ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) >> ==10312== by 0x15A7D8: main (gmock_main.cc:53) >> ==10312== Address 0x8acba20 is 0 bytes inside a block of size 32 alloc'd >> ==10312== at 0x4C2E8E9: operator new[](unsigned long) >> (vg_replace_malloc.c:423) >> ==10312== by 0x16294D: allocate (new_allocator.h:104) >> ==10312== by 0x16294D: allocate (alloc_traits.h:416) >> ==10312== by 0x16294D: _M_allocate (stl_vector.h:170) >> ==10312== by 0x16294D: void >> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >, >> std::allocator<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > > >:: ⤷ >> _M_emplace_back_aux<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > >>> (std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >&&) (vector.tcc:412) >> ==10312== by 0x15E07D: push_back (stl_vector.h:933) >> ==10312== by 0x15E07D: denc_vector_Test::TestBody() (test_denc.cc:206) >> ==10312== by 0x19D203: >> HandleSehExceptionsInMethodIfSupported<testing::Test, void> >> (gtest.cc:2402) >> ==10312== by 0x19D203: void >> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, >> void>(testing::Test*, void (testing::Test::*)(), char const*) >> (gtest.cc:2438) >> ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) >> ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) >> ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) >> ==10312== by 0x194536: >> testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) >> ==10312== by 0x19D6B3: >> HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool> (gtest.cc:2402) >> ==10312== by 0x19D6B3: bool >> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool>(testing::internal::UnitTestImpl*, bool >> (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) >> ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) >> ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) >> ==10312== by 0x15A7D8: main (gmock_main.cc:53) >> >> HTH. >> >> >> On Fri, Dec 23, 2016 at 10:51 AM, Willem Jan Withagen <wjw@xxxxxxxxxxx> wrote: >>> On 23-12-2016 01:32, Brad Hubbard wrote: >>>> Checked this myself and valgrind shows numerous problems of this type. >>> >>> I know of valgrind, and what it is suppossed to do, but have not really >>> used it in a real problem. So that is going to be a first. >>> Any chance of a simple commandline to run this on the minimized test I >>> now have.... >>> >>> --WjW >>> >>> >>>> ==10312== Mismatched free() / delete / delete [] >>>> ==10312== at 0x4C2ED4A: free (vg_replace_malloc.c:530) >>>> ==10312== by 0x160117: deallocate (new_allocator.h:110) >>>> ==10312== by 0x160117: deallocate (alloc_traits.h:442) >>>> ==10312== by 0x160117: _M_put_node (stl_list.h:387) >>>> ==10312== by 0x160117: _M_clear (list.tcc:80) >>>> ==10312== by 0x160117: ~_List_base (stl_list.h:442) >>>> ==10312== by 0x160117: ~list (stl_list.h:503) >>>> ==10312== by 0x160117: denc_list_Test::TestBody() (test_denc.cc:282) >>>> ==10312== by 0x19D203: >>>> HandleSehExceptionsInMethodIfSupported<testing::Test, void> >>>> (gtest.cc:2402) >>>> ==10312== by 0x19D203: void >>>> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, >>>> void>(testing::Test*, void (testing::Test::*)(), char const*) >>>> (gtest.cc:2438) >>>> ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) >>>> ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) >>>> ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) >>>> ==10312== by 0x194536: >>>> testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) >>>> ==10312== by 0x19D6B3: >>>> HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >>>> bool> (gtest.cc:2402) >>>> ==10312== by 0x19D6B3: bool >>>> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >>>> bool>(testing::internal::UnitTestImpl*, bool >>>> (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) >>>> ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) >>>> ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) >>>> ==10312== by 0x15A7D8: main (gmock_main.cc:53) >>>> ==10312== Address 0x8ad31a0 is 0 bytes inside a block of size 24 alloc'd >>>> ==10312== at 0x4C2E8E9: operator new[](unsigned long) >>>> (vg_replace_malloc.c:423) >>>> ==10312== by 0x15FDF1: allocate (new_allocator.h:104) >>>> ==10312== by 0x15FDF1: allocate (alloc_traits.h:416) >>>> ==10312== by 0x15FDF1: _M_get_node (stl_list.h:383) >>>> ==10312== by 0x15FDF1: _M_create_node<denc_counter_bounded_t> >>>> (stl_list.h:568) >>>> ==10312== by 0x15FDF1: _M_insert<denc_counter_bounded_t> (stl_list.h:1770) >>>> ==10312== by 0x15FDF1: emplace_back<denc_counter_bounded_t> (stl_list.h:1108) >>>> ==10312== by 0x15FDF1: decode (denc.h:716) >>>> ==10312== by 0x15FDF1: >>>> decode<std::__cxx11::list<denc_counter_bounded_t> > (denc.h:1289) >>>> ==10312== by 0x15FDF1: >>>> decode<std::__cxx11::list<denc_counter_bounded_t> > (encoding.h:289) >>>> ==10312== by 0x15FDF1: denc_list_Test::TestBody() (test_denc.cc:289) >>>> ==10312== by 0x19D203: >>>> HandleSehExceptionsInMethodIfSupported<testing::Test, void> >>>> (gtest.cc:2402) >>>> ==10312== by 0x19D203: void >>>> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, >>>> void>(testing::Test*, void (testing::Test::*)(), char const*) >>>> (gtest.cc:2438) >>>> ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) >>>> ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) >>>> ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) >>>> ==10312== by 0x194536: >>>> testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) >>>> ==10312== by 0x19D6B3: >>>> HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >>>> bool> (gtest.cc:2402) >>>> ==10312== by 0x19D6B3: bool >>>> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >>>> bool>(testing::internal::UnitTestImpl*, bool >>>> (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) >>>> ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) >>>> ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) >>>> ==10312== by 0x15A7D8: main (gmock_main.cc:53) >>>> >>>> On Fri, Dec 23, 2016 at 10:19 AM, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote: >>>>> Any clue from Valgrind? >>>>> >>>>> Did you say this only happens with clang or doesn't happen with clang? >>>>> >>>>> On Fri, Dec 23, 2016 at 8:01 AM, Allen Samuels >>>>> <Allen.Samuels@xxxxxxxxxxx> wrote: >>>>>> I believe I mis-read the data. What I've seen before doesn't fit this data. >>>>>> >>>>>> If it fails in unit test, it shouldn't be hard to just set a HW breakpoint on the vptr and see who the culprit is. >>>>>> >>>>>> >>>>>> Allen Samuels >>>>>> SanDisk |a Western Digital brand >>>>>> 2880 Junction Avenue, San Jose, CA 95134 >>>>>> T: +1 408 801 7030| M: +1 408 780 6416 >>>>>> allen.samuels@xxxxxxxxxxx >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Willem Jan Withagen [mailto:wjw@xxxxxxxxxxx] >>>>>>> Sent: Thursday, December 22, 2016 10:37 AM >>>>>>> To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Ceph Development >>>>>>> <ceph-devel@xxxxxxxxxxxxxxx> >>>>>>> Subject: Re: An empty vptr in an raw object >>>>>>> >>>>>>> On 22-12-2016 19:02, Allen Samuels wrote: >>>>>>>> I have seen cases of null vptr due to an incompletely constructed object, >>>>>>> i.e., an object that's in the middle of being constructed. >>>>>>> >>>>>>> I going to believe you right away. >>>>>>> But I'm having a hard time imagining such a case. >>>>>>> >>>>>>> Are you suggesting a object is referenced, whilest it is not yet complete. who >>>>>>> does the referencing then? due to threading? >>>>>>> That would be even harder to find. >>>>>>> >>>>>>> --WjW >>>>>>> >>>>>>> >>>>>>>> Allen Samuels >>>>>>>> SanDisk |a Western Digital brand >>>>>>>> 951 SanDisk Drive, Milpitas, CA 95035 >>>>>>>> T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel- >>>>>>>>> owner@xxxxxxxxxxxxxxx] On Behalf Of Willem Jan Withagen >>>>>>>>> Sent: Thursday, December 22, 2016 9:41 AM >>>>>>>>> To: Ceph Development <ceph-devel@xxxxxxxxxxxxxxx> >>>>>>>>> Subject: An empty vptr in an raw object >>>>>>>>> >>>>>>>>>> I have a piece of code that actually seem to crash because the vptr >>>>>>>>>> is not set: >>>>>>>>>> (gdb) p *_raw >>>>>>>>>> $2 = {_vptr$raw = 0x0, data = 0x10cc000 "\003", len = 72, nref = >>>>>>>>>> {val = 1}, crc_spinlock = 0, crc_map = {__tree_ = { >>>>>>>>>> __begin_node_ = 0x10cc078, >>>>>>>>>> __pair1_ = >>>>>>>>>> >>>>>>>>> >>>>>>> {<std::__1::__libcpp_compressed_pair_imp<std::__1::__tree_end_node<st >>>>>>>>> d::__1::__tree_node_base<void*>*>, >>>>>>>>>> >>>>>>>>> std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<std: >>>>>>>>> :__1 >>>>>>>>> ::pair<unsigned >>>>>>>>>> long, unsigned long>, std::__1::pair<unsigned int, unsigned int> >, >>>>>>>>>> void*> >, 2>> = >>>>>>>>>> >>>>>>>>> {<std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<st >>>>>>>>> d::_ >>>>>>>>> _1::pair<unsigned >>>>>>>>>> long, unsigned long>, std::__1::pair<unsigned int, unsigned int> >, >>>>>>>>>> void*> >> = {<No data fields>}, __first_ = { >>>>>>>>>> __left_ = 0x0}}, <No data fields>}, >>>>>>>>>> >>>>>>>>>> The function that crashes: >>>>>>>>>> char *buffer::ptr::c_str() { >>>>>>>>>> assert(_raw); >>>>>>>>>> if (buffer_track_c_str) >>>>>>>>>> buffer_c_str_accesses.inc(); >>>>>>>>>> char *p = _raw->get_data(); >>>>>>>>>> return p + _off; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> And crash is actually on the return line. >>>>>>>>>> >>>>>>>>>> Any ideas as how the vptr can be empty? >>>>>>>>> >>>>>>>>> Now the _vptr$raw point is part of the internal code of the clang >>>>>>>>> class function table/constructor. Overwriting that means that >>>>>>>>> class-function references are problematic to say the least. (in this >>>>>>> example get_data()). >>>>>>>>> >>>>>>>>> The major reason why this occurs is because an object is being zeroed >>>>>>>>> in C-style way: memset( &obj, 0, sizeof(obj)) And thus overwriting >>>>>>>>> the _vptr. >>>>>>>>> >>>>>>>>> Note that this does not bite the FreeBSD compilation, but also any >>>>>>>>> other attempts to build Ceph with clang. >>>>>>>>> >>>>>>>>> Now the strange thing is that this does not bite Clang compilation >>>>>>>>> much more. But the only test that fails is unittest_denc. So I guess >>>>>>>>> that most of the code is rather well behaved. >>>>>>>>> >>>>>>>>> And I'm off on a search to find the culprit. >>>>>>>>> >>>>>>>>> --WjW >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>>>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More >>>>>>> majordomo >>>>>>>>> info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> Brad >>>> >>>> >>>> >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> > -- Cheers, Brad -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
