Here's what I ran. $ valgrind --trace-children=yes --show-reachable=yes --track-origins=yes --read-var-info=yes --tool=memcheck --leak-check=full --num-callers=50 -v --log-file=unittest_denc.log bin/unittest_denc Your stack shows /usr/srcs/Ceph/work/ceph/src/test/test_denc.cc:206 which seems to match up with this. ==10312== Mismatched free() / delete / delete [] ==10312== at 0x4C2ED4A: free (vg_replace_malloc.c:530) ==10312== by 0x162A6A: deallocate (new_allocator.h:110) ==10312== by 0x162A6A: deallocate (alloc_traits.h:442) ==10312== by 0x162A6A: _M_deallocate (stl_vector.h:178) ==10312== by 0x162A6A: void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >:: ⤷ _M_emplace_back_aux<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (vector.tcc:438) ==10312== by 0x15E0B5: push_back (stl_vector.h:933) ==10312== by 0x15E0B5: denc_vector_Test::TestBody() (test_denc.cc:207) ==10312== by 0x19D203: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2402) ==10312== by 0x19D203: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2438) ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) ==10312== by 0x194536: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) ==10312== by 0x19D6B3: HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (gtest.cc:2402) ==10312== by 0x19D6B3: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) ==10312== by 0x15A7D8: main (gmock_main.cc:53) ==10312== Address 0x8acba20 is 0 bytes inside a block of size 32 alloc'd ==10312== at 0x4C2E8E9: operator new[](unsigned long) (vg_replace_malloc.c:423) ==10312== by 0x16294D: allocate (new_allocator.h:104) ==10312== by 0x16294D: allocate (alloc_traits.h:416) ==10312== by 0x16294D: _M_allocate (stl_vector.h:170) ==10312== by 0x16294D: void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >:: ⤷ _M_emplace_back_aux<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (vector.tcc:412) ==10312== by 0x15E07D: push_back (stl_vector.h:933) ==10312== by 0x15E07D: denc_vector_Test::TestBody() (test_denc.cc:206) ==10312== by 0x19D203: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2402) ==10312== by 0x19D203: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2438) ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) ==10312== by 0x194536: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) ==10312== by 0x19D6B3: HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (gtest.cc:2402) ==10312== by 0x19D6B3: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) ==10312== by 0x15A7D8: main (gmock_main.cc:53) HTH. On Fri, Dec 23, 2016 at 10:51 AM, Willem Jan Withagen <wjw@xxxxxxxxxxx> wrote: > On 23-12-2016 01:32, Brad Hubbard wrote: >> Checked this myself and valgrind shows numerous problems of this type. > > I know of valgrind, and what it is suppossed to do, but have not really > used it in a real problem. So that is going to be a first. > Any chance of a simple commandline to run this on the minimized test I > now have.... > > --WjW > > >> ==10312== Mismatched free() / delete / delete [] >> ==10312== at 0x4C2ED4A: free (vg_replace_malloc.c:530) >> ==10312== by 0x160117: deallocate (new_allocator.h:110) >> ==10312== by 0x160117: deallocate (alloc_traits.h:442) >> ==10312== by 0x160117: _M_put_node (stl_list.h:387) >> ==10312== by 0x160117: _M_clear (list.tcc:80) >> ==10312== by 0x160117: ~_List_base (stl_list.h:442) >> ==10312== by 0x160117: ~list (stl_list.h:503) >> ==10312== by 0x160117: denc_list_Test::TestBody() (test_denc.cc:282) >> ==10312== by 0x19D203: >> HandleSehExceptionsInMethodIfSupported<testing::Test, void> >> (gtest.cc:2402) >> ==10312== by 0x19D203: void >> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, >> void>(testing::Test*, void (testing::Test::*)(), char const*) >> (gtest.cc:2438) >> ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) >> ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) >> ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) >> ==10312== by 0x194536: >> testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) >> ==10312== by 0x19D6B3: >> HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool> (gtest.cc:2402) >> ==10312== by 0x19D6B3: bool >> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool>(testing::internal::UnitTestImpl*, bool >> (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) >> ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) >> ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) >> ==10312== by 0x15A7D8: main (gmock_main.cc:53) >> ==10312== Address 0x8ad31a0 is 0 bytes inside a block of size 24 alloc'd >> ==10312== at 0x4C2E8E9: operator new[](unsigned long) >> (vg_replace_malloc.c:423) >> ==10312== by 0x15FDF1: allocate (new_allocator.h:104) >> ==10312== by 0x15FDF1: allocate (alloc_traits.h:416) >> ==10312== by 0x15FDF1: _M_get_node (stl_list.h:383) >> ==10312== by 0x15FDF1: _M_create_node<denc_counter_bounded_t> >> (stl_list.h:568) >> ==10312== by 0x15FDF1: _M_insert<denc_counter_bounded_t> (stl_list.h:1770) >> ==10312== by 0x15FDF1: emplace_back<denc_counter_bounded_t> (stl_list.h:1108) >> ==10312== by 0x15FDF1: decode (denc.h:716) >> ==10312== by 0x15FDF1: >> decode<std::__cxx11::list<denc_counter_bounded_t> > (denc.h:1289) >> ==10312== by 0x15FDF1: >> decode<std::__cxx11::list<denc_counter_bounded_t> > (encoding.h:289) >> ==10312== by 0x15FDF1: denc_list_Test::TestBody() (test_denc.cc:289) >> ==10312== by 0x19D203: >> HandleSehExceptionsInMethodIfSupported<testing::Test, void> >> (gtest.cc:2402) >> ==10312== by 0x19D203: void >> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, >> void>(testing::Test*, void (testing::Test::*)(), char const*) >> (gtest.cc:2438) >> ==10312== by 0x194029: testing::Test::Run() (gtest.cc:2475) >> ==10312== by 0x194177: testing::TestInfo::Run() (gtest.cc:2656) >> ==10312== by 0x194254: testing::TestCase::Run() (gtest.cc:2774) >> ==10312== by 0x194536: >> testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4649) >> ==10312== by 0x19D6B3: >> HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool> (gtest.cc:2402) >> ==10312== by 0x19D6B3: bool >> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, >> bool>(testing::internal::UnitTestImpl*, bool >> (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2438) >> ==10312== by 0x194853: testing::UnitTest::Run() (gtest.cc:4257) >> ==10312== by 0x15A7D8: RUN_ALL_TESTS (gtest.h:2233) >> ==10312== by 0x15A7D8: main (gmock_main.cc:53) >> >> On Fri, Dec 23, 2016 at 10:19 AM, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote: >>> Any clue from Valgrind? >>> >>> Did you say this only happens with clang or doesn't happen with clang? >>> >>> On Fri, Dec 23, 2016 at 8:01 AM, Allen Samuels >>> <Allen.Samuels@xxxxxxxxxxx> wrote: >>>> I believe I mis-read the data. What I've seen before doesn't fit this data. >>>> >>>> If it fails in unit test, it shouldn't be hard to just set a HW breakpoint on the vptr and see who the culprit is. >>>> >>>> >>>> Allen Samuels >>>> SanDisk |a Western Digital brand >>>> 2880 Junction Avenue, San Jose, CA 95134 >>>> T: +1 408 801 7030| M: +1 408 780 6416 >>>> allen.samuels@xxxxxxxxxxx >>>> >>>> >>>>> -----Original Message----- >>>>> From: Willem Jan Withagen [mailto:wjw@xxxxxxxxxxx] >>>>> Sent: Thursday, December 22, 2016 10:37 AM >>>>> To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Ceph Development >>>>> <ceph-devel@xxxxxxxxxxxxxxx> >>>>> Subject: Re: An empty vptr in an raw object >>>>> >>>>> On 22-12-2016 19:02, Allen Samuels wrote: >>>>>> I have seen cases of null vptr due to an incompletely constructed object, >>>>> i.e., an object that's in the middle of being constructed. >>>>> >>>>> I going to believe you right away. >>>>> But I'm having a hard time imagining such a case. >>>>> >>>>> Are you suggesting a object is referenced, whilest it is not yet complete. who >>>>> does the referencing then? due to threading? >>>>> That would be even harder to find. >>>>> >>>>> --WjW >>>>> >>>>> >>>>>> Allen Samuels >>>>>> SanDisk |a Western Digital brand >>>>>> 951 SanDisk Drive, Milpitas, CA 95035 >>>>>> T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel- >>>>>>> owner@xxxxxxxxxxxxxxx] On Behalf Of Willem Jan Withagen >>>>>>> Sent: Thursday, December 22, 2016 9:41 AM >>>>>>> To: Ceph Development <ceph-devel@xxxxxxxxxxxxxxx> >>>>>>> Subject: An empty vptr in an raw object >>>>>>> >>>>>>>> I have a piece of code that actually seem to crash because the vptr >>>>>>>> is not set: >>>>>>>> (gdb) p *_raw >>>>>>>> $2 = {_vptr$raw = 0x0, data = 0x10cc000 "\003", len = 72, nref = >>>>>>>> {val = 1}, crc_spinlock = 0, crc_map = {__tree_ = { >>>>>>>> __begin_node_ = 0x10cc078, >>>>>>>> __pair1_ = >>>>>>>> >>>>>>> >>>>> {<std::__1::__libcpp_compressed_pair_imp<std::__1::__tree_end_node<st >>>>>>> d::__1::__tree_node_base<void*>*>, >>>>>>>> >>>>>>> std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<std: >>>>>>> :__1 >>>>>>> ::pair<unsigned >>>>>>>> long, unsigned long>, std::__1::pair<unsigned int, unsigned int> >, >>>>>>>> void*> >, 2>> = >>>>>>>> >>>>>>> {<std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<st >>>>>>> d::_ >>>>>>> _1::pair<unsigned >>>>>>>> long, unsigned long>, std::__1::pair<unsigned int, unsigned int> >, >>>>>>>> void*> >> = {<No data fields>}, __first_ = { >>>>>>>> __left_ = 0x0}}, <No data fields>}, >>>>>>>> >>>>>>>> The function that crashes: >>>>>>>> char *buffer::ptr::c_str() { >>>>>>>> assert(_raw); >>>>>>>> if (buffer_track_c_str) >>>>>>>> buffer_c_str_accesses.inc(); >>>>>>>> char *p = _raw->get_data(); >>>>>>>> return p + _off; >>>>>>>> } >>>>>>>> >>>>>>>> And crash is actually on the return line. >>>>>>>> >>>>>>>> Any ideas as how the vptr can be empty? >>>>>>> >>>>>>> Now the _vptr$raw point is part of the internal code of the clang >>>>>>> class function table/constructor. Overwriting that means that >>>>>>> class-function references are problematic to say the least. (in this >>>>> example get_data()). >>>>>>> >>>>>>> The major reason why this occurs is because an object is being zeroed >>>>>>> in C-style way: memset( &obj, 0, sizeof(obj)) And thus overwriting >>>>>>> the _vptr. >>>>>>> >>>>>>> Note that this does not bite the FreeBSD compilation, but also any >>>>>>> other attempts to build Ceph with clang. >>>>>>> >>>>>>> Now the strange thing is that this does not bite Clang compilation >>>>>>> much more. But the only test that fails is unittest_denc. So I guess >>>>>>> that most of the code is rather well behaved. >>>>>>> >>>>>>> And I'm off on a search to find the culprit. >>>>>>> >>>>>>> --WjW >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>>>>>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More >>>>> majordomo >>>>>>> info at http://vger.kernel.org/majordomo-info.html >>>> >>> >>> >>> >>> -- >>> Cheers, >>> Brad >> >> >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Cheers, Brad -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
