One other trick that has been used here: if you look inside the PG directories on the OSDs and find that they are mostly empty then it's possible some of the memory and peering overhead is related to empty and useless PG instances on the wrong OSDs. You can write a script to find empty directories (or ones that only contain the single pgmeta object with a mostly-empty name) and remove them (using ceph-objectstore-tool). (For safety I'd recommend doing ceph-objectstore-tool export first, just in case there is some useful metadata there.) That will only help if most of the pg dirs look empty, though. If so, it's worth a shot! The other thing we once did was use a kludge patch to trim the past_intervals metadata, which was respnosible for most of the memory usage. I can't tell from the profile in this thread if that is the case or not. There is a patch floating around in git somewhere that can be reused if it looks like that is the thing consuming the memory. sage On Sat, 19 Aug 2017, Mustafa Muhammad wrote: > Hi all, > Looks like the memory is consumed in the > "PG::RecoveryState::RecoveryMachine::send_notify", is this related to > messenger? Can we get lower memory usage even if this mean slower > peering (or delayed recovery)? > > Thanks in advance > > Mustafa Muhammad > > > On Thu, Aug 17, 2017 at 9:51 PM, Linux Chips <linux.chips@xxxxxxxxx> wrote: > > > > > > On 08/17/2017 08:53 PM, Gregory Farnum wrote: > >> > >> On Thu, Aug 17, 2017 at 7:13 AM, Linux Chips <linux.chips@xxxxxxxxx> > >> wrote: > >>> > >>> Hello everybody, > >>> I have Kraken cluster with 660 OSD, currently it is down due to not > >>> being able to complete peering, OSDs start consuming lots of memory > >>> draining the system and killing the node, so I set a limit on the OSD > >>> service (on some OSDs 28G and others as high as 35G), so they get > >>> killed before taking down the whole node. > >>> Now I still can't peer, one OSD entering the cluster (with about 300 > >>> already up) makes memory usage of most other OSDs so high (15G+, some as > >>> much as 30G) and > >>> sometimes kills them when they reach the service limit. which cause a > >>> spiral > >>> load and causing all the OSDs to consume all the available. > >>> > >>> I found this thread with similar symptoms: > >>> > >>> > >>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017522.html > >>> > >>> with a request for stack trace, I have a 14G core dump, we generated it > >>> by > >>> running the osd from the terminal, enabling the core dumps, and setting > >>> ulimits to 15G. what kind of a trace would be useful? all thread?! any > >>> better way to debug this? > >>> > >>> What can I do do make it work, is this memory allocation normal? > >>> > >>> some info about the cluster: > >>> 41 hdd nodes with 12 x 4TB osd each, 5 of the nodes have 8TB disks. 324 > >>> GB > >>> RAM and dula socket intel xeon. > >>> 7 nodes with 400GB x 24 ssd and 256GB RAM, and dual socket cpu. > >>> 3 monitors > >>> > >>> all dual 10GB ethernet, except for the monitor with dual 1GB ethers. > >>> > >>> all nodes running centos 7.2 > >>> it is an old cluster that was upgraded continuously for the past 3 years. > >>> the cluster was on jewel when the issue happened due to some accidental > >>> OSD > >>> map changes, causing a heavy recovery operations on the cluster. then we > >>> upgraded to kraken in the hope of less memory foot prints. > >>> > >>> any advice on how to proceed? > >> > >> It's not normal but if something really bad happened to your cluster, > >> it's been known to occur. You should go through the troubleshooting > >> guides at docs.ceph.com, but the general strategy is to set > >> nodown/noout/etc flags, undo whatever horrible thing you tried to make > >> the map do, and then turn all the OSDs back on. > >> -Greg > > > > > > Hi, > > we have been trying this for the past week, it keeps consuming the RAM. > > we got the map back to the original places. marked all the flags, started > > all the OSDs. then "ceph osd unset noup", wait 5 min, and all OSDs are > > killed by the oom. > > we tried one node at a time, let it finish recovering, and start the next. > > we got to a point when we started the next node, every thing got killed. > > we tried one OSD at a time, same result. one OSD up, ~40 killed by oom, then > > it is a snow ball from here until all of the active OSDs get kiiled. > > > > I think all this up/down that we generated has increased the recovery too > > much. btw, we stopped all clients. and also we have some not so friendly > > erasure pools. some OSDs now report loading as much as 800 pg, while we > > originally had about 300-400 (I know too much, but we were trying to fix it > > and.... well we could not). > > > > we did a memory profiling on one of the OSDs. > > here is the results > > > > > > 12878.6 47.6% 47.6% 12878.6 47.6% std::_Rb_tree::_M_create_node > > 12867.6 47.6% 95.2% 25746.2 95.2% std::_Rb_tree::_M_copy > > 532.4 2.0% 97.2% 686.3 2.5% OSD::heartbeat > > 122.8 0.5% 97.7% 122.8 0.5% std::_Rb_tree::_M_emplace_hint_unique > > 121.9 0.5% 98.1% 171.1 0.6% AsyncConnection::send_message > > 104.2 0.4% 98.5% 104.2 0.4% ceph::buffer::list::append@c4a770 > > 99.7 0.4% 98.9% 99.7 0.4% std::vector::_M_default_append > > 99.6 0.4% 99.2% 99.6 0.4% ceph::logging::Log::create_entry > > 72.6 0.3% 99.5% 72.6 0.3% ceph::buffer::create_aligned > > 52.4 0.2% 99.7% 52.5 0.2% std::vector::_M_emplace_back_aux > > 23.9 0.1% 99.8% 57.8 0.2% OSD::do_notifies > > 17.0 0.1% 99.8% 23.1 0.1% OSDService::build_incremental_map_msg > > 9.8 0.0% 99.9% 222.5 0.8% std::enable_if::type decode > > 6.2 0.0% 99.9% 6.3 0.0% std::map::operator[] > > 5.5 0.0% 99.9% 5.5 0.0% std::vector::vector > > 3.5 0.0% 99.9% 3.5 0.0% EventCenter::create_time_event > > 2.5 0.0% 99.9% 2.5 0.0% AsyncConnection::AsyncConnection > > 2.4 0.0% 100.0% 2.4 0.0% std::string::_Rep::_S_create > > 1.5 0.0% 100.0% 1.5 0.0% std::_Rb_tree::_M_insert_unique > > 1.4 0.0% 100.0% 1.4 0.0% std::list::operator= > > 1.3 0.0% 100.0% 1.3 0.0% ceph::buffer::list::list > > 0.9 0.0% 100.0% 204.1 0.8% decode_message > > 0.8 0.0% 100.0% 0.8 0.0% OSD::send_failures > > 0.7 0.0% 100.0% 0.9 0.0% void decode > > 0.6 0.0% 100.0% 0.6 0.0% std::_Rb_tree::_M_insert_equal > > 0.6 0.0% 100.0% 2.5 0.0% PG::queue_null > > 0.6 0.0% 100.0% 1.8 0.0% AsyncMessenger::create_connect > > 0.6 0.0% 100.0% 1.8 0.0% AsyncMessenger::add_accept > > 0.5 0.0% 100.0% 0.5 0.0% boost::statechart::event::clone > > 0.4 0.0% 100.0% 0.4 0.0% PG::queue_peering_event > > 0.3 0.0% 100.0% 0.3 0.0% OSD::PeeringWQ::_enqueue > > 0.3 0.0% 100.0% 148.6 0.5% OSD::_dispatch > > 0.1 0.0% 100.0% 147.9 0.5% OSD::handle_osd_map > > 0.1 0.0% 100.0% 0.1 0.0% std::deque::_M_push_back_aux > > 0.1 0.0% 100.0% 0.2 0.0% SharedLRU::add > > 0.1 0.0% 100.0% 0.1 0.0% OSD::PeeringWQ::_dequeue > > 0.1 0.0% 100.0% 0.1 0.0% ceph::buffer::list::append@c4a9b0 > > 0.1 0.0% 100.0% 0.2 0.0% DispatchQueue::enqueue > > 0.1 0.0% 100.0% 283.5 1.0% EventCenter::process_events > > 0.1 0.0% 100.0% 0.1 0.0% HitSet::Params::create_impl > > 0.1 0.0% 100.0% 0.1 0.0% SimpleLRU::clear_pinned > > 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_ > > 0.0 0.0% 100.0% 0.2 0.0% TrackedOp::mark_event > > 0.0 0.0% 100.0% 0.0 0.0% OSD::create_context > > 0.0 0.0% 100.0% 0.0 0.0% std::_Hashtable::_M_allocate_node > > 0.0 0.0% 100.0% 0.0 0.0% OSDMap::OSDMap > > 0.0 0.0% 100.0% 281.6 1.0% AsyncConnection::process > > 0.0 0.0% 100.0% 25802.4 95.4% > > PG::RecoveryState::RecoveryMachine::send_notify > > 0.0 0.0% 100.0% 0.0 0.0% SharedLRU::lru_add > > 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_unique_ > > 0.0 0.0% 100.0% 0.1 0.0% OpTracker::unregister_inflight_op > > 0.0 0.0% 100.0% 0.0 0.0% OSD::ms_verify_authorizer > > 0.0 0.0% 100.0% 0.0 0.0% OSDService::_add_map > > 0.0 0.0% 100.0% 0.1 0.0% OSD::wait_for_new_map > > 0.0 0.0% 100.0% 0.5 0.0% OSD::handle_pg_notify > > 0.0 0.0% 100.0% 0.0 0.0% std::__shared_count::__shared_count > > 0.0 0.0% 100.0% 0.0 0.0% std::__shared_ptr::reset > > 0.0 0.0% 100.0% 35.1 0.1% OSDMap::decode@b84080 > > 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_emplace_unique > > 0.0 0.0% 100.0% 0.0 0.0% std::vector::operator= > > 0.0 0.0% 100.0% 0.0 0.0% MonClient::_renew_subs > > 0.0 0.0% 100.0% 0.0 0.0% std::_Hashtable::_M_emplace > > 0.0 0.0% 100.0% 0.0 0.0% PORT_Alloc_Util > > 0.0 0.0% 100.0% 0.0 0.0% CryptoAES::get_key_handler > > 0.0 0.0% 100.0% 0.0 0.0% get_auth_session_handler > > 0.0 0.0% 100.0% 0.0 0.0% PosixWorker::connect > > 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::append@c4a440 > > 0.0 0.0% 100.0% 0.0 0.0% std::vector::_M_fill_insert > > 0.0 0.0% 100.0% 4.8 0.0% AsyncConnection::fault > > 0.0 0.0% 100.0% 0.0 0.0% OSD::send_pg_stats > > 0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::accept_conn > > 0.0 0.0% 100.0% 0.0 0.0% PosixServerSocketImpl::accept > > 0.0 0.0% 100.0% 9.3 0.0% AsyncConnection::_process_connection > > 0.0 0.0% 100.0% 0.2 0.0% FileStore::lfn_open > > 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::append@c4a350 > > 0.0 0.0% 100.0% 0.0 0.0% crush_create > > 0.0 0.0% 100.0% 0.1 0.0% MgrClient::send_report > > 0.0 0.0% 100.0% 0.0 0.0% WBThrottle::queue_wb > > 0.0 0.0% 100.0% 0.2 0.0% LogClient::_get_mon_log_message > > 0.0 0.0% 100.0% 0.0 0.0% CryptoKey::_set_secret > > 0.0 0.0% 100.0% 0.0 0.0% std::_Deque_base::_M_initialize_map > > 0.0 0.0% 100.0% 0.1 0.0% > > ThreadPool::BatchWorkQueue::_void_dequeue > > 0.0 0.0% 100.0% 0.0 0.0% ceph::Formatter::create@ba6a50 > > 0.0 0.0% 100.0% 0.0 0.0% MonClient::schedule_tick > > 0.0 0.0% 100.0% 0.1 0.0% OSD::tick > > 0.0 0.0% 100.0% 37.6 0.1% OSD::tick_without_osd_lock > > 0.0 0.0% 100.0% 0.0 0.0% > > boost::spirit::classic::impl::get_definition > > 0.0 0.0% 100.0% 9.4 0.0% MonClient::_send_mon_message > > 0.0 0.0% 100.0% 0.0 0.0% DispatchQueue::queue_refused > > 0.0 0.0% 100.0% 0.0 0.0% OSD::handle_command > > 0.0 0.0% 100.0% 0.0 0.0% DispatchQueue::queue_accept > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::_connect > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::_stop > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::accept > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::handle_connect_msg > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::mark_down > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::prepare_send_message > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::read_bulk > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::read_until > > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::send_keepalive > > 0.0 0.0% 100.0% 3.3 0.0% AsyncConnection::wakeup_from > > 0.0 0.0% 100.0% 1.8 0.0% AsyncMessenger::get_connection > > 0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::reap_dead > > 0.0 0.0% 100.0% 2.5 0.0% C_OnMapCommit::finish > > 0.0 0.0% 100.0% 0.0 0.0% > > CephXTicketHandler::verify_service_ticket_reply > > 0.0 0.0% 100.0% 0.0 0.0% > > CephXTicketManager::verify_service_ticket_reply > > 0.0 0.0% 100.0% 0.0 0.0% > > CephxAuthorizeHandler::verify_authorizer > > 0.0 0.0% 100.0% 0.0 0.0% CephxClientHandler::handle_response > > 0.0 0.0% 100.0% 40.9 0.2% Context::complete > > 0.0 0.0% 100.0% 4.8 0.0% CrushWrapper::encode > > 0.0 0.0% 100.0% 0.0 0.0% CryptoAESKeyHandler::decrypt > > 0.0 0.0% 100.0% 0.0 0.0% CryptoKey::decode > > 0.0 0.0% 100.0% 160.4 0.6% DispatchQueue::DispatchThread::entry > > 0.0 0.0% 100.0% 160.4 0.6% DispatchQueue::entry > > 0.0 0.0% 100.0% 0.4 0.0% DispatchQueue::fast_dispatch > > 0.0 0.0% 100.0% 0.4 0.0% DispatchQueue::pre_dispatch > > 0.0 0.0% 100.0% 0.0 0.0% EntityName::set > > 0.0 0.0% 100.0% 0.0 0.0% EpollDriver::event_wait > > 0.0 0.0% 100.0% 3.0 0.0% EventCenter::dispatch_event_external > > 0.0 0.0% 100.0% 3.3 0.0% EventCenter::process_time_events > > 0.0 0.0% 100.0% 3.0 0.0% EventCenter::wakeup > > 0.0 0.0% 100.0% 0.0 0.0% FileJournal::prepare_entry > > 0.0 0.0% 100.0% 0.2 0.0% FileStore::_do_op > > 0.0 0.0% 100.0% 0.2 0.0% FileStore::_do_transaction > > 0.0 0.0% 100.0% 0.2 0.0% FileStore::_do_transactions > > 0.0 0.0% 100.0% 0.0 0.0% FileStore::_journaled_ahead > > 0.0 0.0% 100.0% 0.2 0.0% FileStore::_write > > 0.0 0.0% 100.0% 0.0 0.0% FileStore::queue_transactions > > 0.0 0.0% 100.0% 2.6 0.0% Finisher::finisher_thread_entry > > 0.0 0.0% 100.0% 0.1 0.0% FunctionContext::finish > > 0.0 0.0% 100.0% 0.1 0.0% HitSet::Params::decode > > 0.0 0.0% 100.0% 0.2 0.0% LogChannel::do_log@a90a00 > > 0.0 0.0% 100.0% 0.3 0.0% LogChannel::do_log@a91030 > > 0.0 0.0% 100.0% 0.2 0.0% LogClient::get_mon_log_message > > 0.0 0.0% 100.0% 0.0 0.0% LogClient::handle_log_ack > > 0.0 0.0% 100.0% 0.1 0.0% LogClient::queue > > 0.0 0.0% 100.0% 0.3 0.0% LogClientTemp::~LogClientTemp > > 0.0 0.0% 100.0% 0.0 0.0% MAuthReply::decode_payload > > 0.0 0.0% 100.0% 0.0 0.0% MCommand::decode_payload > > 0.0 0.0% 100.0% 0.0 0.0% MCommand::print > > 0.0 0.0% 100.0% 0.0 0.0% MMgrMap::decode_payload > > 0.0 0.0% 100.0% 0.0 0.0% MOSDFailure::print > > 0.0 0.0% 100.0% 0.1 0.0% MOSDMap::decode_payload > > 0.0 0.0% 100.0% 203.1 0.8% MOSDPGNotify::decode_payload > > 0.0 0.0% 100.0% 0.0 0.0% MOSDPGNotify::print > > 0.0 0.0% 100.0% 0.0 0.0% MOSDPing::encode_payload > > 0.0 0.0% 100.0% 0.0 0.0% Message::encode > > 0.0 0.0% 100.0% 0.0 0.0% MgrClient::handle_mgr_map > > 0.0 0.0% 100.0% 0.0 0.0% MgrClient::ms_dispatch > > 0.0 0.0% 100.0% 0.0 0.0% MgrMap::decode > > 0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_rotating > > 0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_tickets > > 0.0 0.0% 100.0% 0.0 0.0% MonClient::_finish_hunting > > 0.0 0.0% 100.0% 0.8 0.0% MonClient::_reopen_session@aeab80 > > 0.0 0.0% 100.0% 0.6 0.0% MonClient::_reopen_session@af2ba0 > > 0.0 0.0% 100.0% 9.5 0.0% MonClient::handle_auth > > 0.0 0.0% 100.0% 9.6 0.0% MonClient::ms_dispatch > > 0.0 0.0% 100.0% 0.2 0.0% MonClient::send_log > > 0.0 0.0% 100.0% 0.6 0.0% MonClient::tick > > 0.0 0.0% 100.0% 283.5 1.0% NetworkStack::get_worker > > 0.0 0.0% 100.0% 0.0 0.0% OSD::CommandWQ::_process > > 0.0 0.0% 100.0% 25862.5 95.7% OSD::PeeringWQ::_process > > 0.0 0.0% 100.0% 0.0 0.0% OSD::Session::Session > > 0.0 0.0% 100.0% 686.3 2.5% OSD::T_Heartbeat::entry > > 0.0 0.0% 100.0% 2.5 0.0% OSD::_committed_osd_maps > > 0.0 0.0% 100.0% 25804.6 95.5% OSD::advance_pg > > 0.0 0.0% 100.0% 0.3 0.0% OSD::check_ops_in_flight > > 0.0 0.0% 100.0% 0.0 0.0% OSD::check_osdmap_features > > 0.0 0.0% 100.0% 2.5 0.0% OSD::consume_map > > 0.0 0.0% 100.0% 57.8 0.2% OSD::dispatch_context > > 0.0 0.0% 100.0% 0.5 0.0% OSD::dispatch_op > > 0.0 0.0% 100.0% 0.0 0.0% OSD::do_command > > 0.0 0.0% 100.0% 0.2 0.0% OSD::do_waiters > > 0.0 0.0% 100.0% 0.0 0.0% OSD::get_osdmap_pobject_name > > 0.0 0.0% 100.0% 0.1 0.0% OSD::handle_osd_ping > > 0.0 0.0% 100.0% 0.0 0.0% OSD::handle_pg_peering_evt > > 0.0 0.0% 100.0% 37.2 0.1% OSD::heartbeat_check > > 0.0 0.0% 100.0% 0.1 0.0% OSD::heartbeat_dispatch > > 0.0 0.0% 100.0% 686.3 2.5% OSD::heartbeat_entry > > 0.0 0.0% 100.0% 1.1 0.0% OSD::heartbeat_reset > > 0.0 0.0% 100.0% 148.7 0.6% OSD::ms_dispatch > > 0.0 0.0% 100.0% 0.8 0.0% OSD::ms_handle_connect > > 0.0 0.0% 100.0% 0.0 0.0% OSD::ms_handle_refused > > 0.0 0.0% 100.0% 0.0 0.0% OSD::ms_handle_reset > > 0.0 0.0% 100.0% 25862.5 95.7% OSD::process_peering_events > > 0.0 0.0% 100.0% 0.1 0.0% OSD::require_same_or_newer_map > > 0.0 0.0% 100.0% 0.0 0.0% OSD::write_superblock > > 0.0 0.0% 100.0% 0.0 0.0% OSDCap::parse > > 0.0 0.0% 100.0% 0.0 0.0% OSDMap::Incremental::decode > > 0.0 0.0% 100.0% 35.1 0.1% OSDMap::decode@b85440 > > 0.0 0.0% 100.0% 110.8 0.4% OSDMap::encode > > 0.0 0.0% 100.0% 0.5 0.0% OSDMap::post_decode > > 0.0 0.0% 100.0% 0.1 0.0% OSDService::_get_map_bl > > 0.0 0.0% 100.0% 0.0 0.0% OSDService::check_nearfull_warning > > 0.0 0.0% 100.0% 0.1 0.0% OSDService::clear_map_bl_cache_pins > > 0.0 0.0% 100.0% 1.1 0.0% OSDService::get_con_osd_hb > > 0.0 0.0% 100.0% 1.3 0.0% OSDService::get_inc_map_bl > > 0.0 0.0% 100.0% 1.3 0.0% OSDService::pin_map_bl > > 0.0 0.0% 100.0% 0.0 0.0% OSDService::pin_map_inc_bl > > 0.0 0.0% 100.0% 0.0 0.0% OSDService::publish_superblock > > 0.0 0.0% 100.0% 0.3 0.0% OSDService::queue_for_peering > > 0.0 0.0% 100.0% 27.2 0.1% OSDService::send_incremental_map > > 0.0 0.0% 100.0% 27.2 0.1% OSDService::share_map_peer > > 0.0 0.0% 100.0% 0.0 0.0% OSDService::update_osd_stat > > 0.0 0.0% 100.0% 0.0 0.0% > > ObjectStore::Transaction::_get_coll_id > > 0.0 0.0% 100.0% 0.0 0.0% > > ObjectStore::Transaction::_get_next_op > > 0.0 0.0% 100.0% 0.2 0.0% ObjectStore::Transaction::write > > 0.0 0.0% 100.0% 0.0 0.0% ObjectStore::queue_transaction > > 0.0 0.0% 100.0% 0.0 0.0% Objecter::_maybe_request_map > > 0.0 0.0% 100.0% 0.1 0.0% Objecter::handle_osd_map > > 0.0 0.0% 100.0% 0.1 0.0% OpHistory::insert > > 0.0 0.0% 100.0% 0.0 0.0% OpRequest::OpRequest > > 0.0 0.0% 100.0% 0.1 0.0% OpRequest::mark_flag_point > > 0.0 0.0% 100.0% 0.1 0.0% OpRequest::mark_started > > 0.0 0.0% 100.0% 0.1 0.0% OpTracker::RemoveOnDelete::operator > > 0.0 0.0% 100.0% 0.1 0.0% OpTracker::_mark_event > > 0.0 0.0% 100.0% 0.0 0.0% OpTracker::get_age_ms_histogram > > 0.0 0.0% 100.0% 25802.4 95.4% PG::RecoveryState::Stray::react > > 0.0 0.0% 100.0% 0.0 0.0% PG::_prepare_write_info > > 0.0 0.0% 100.0% 25802.4 95.4% PG::handle_activate_map > > 0.0 0.0% 100.0% 1.6 0.0% PG::handle_advance_map > > 0.0 0.0% 100.0% 0.0 0.0% PG::prepare_write_info > > 0.0 0.0% 100.0% 0.0 0.0% PG::write_if_dirty > > 0.0 0.0% 100.0% 1.6 0.0% PGPool::update > > 0.0 0.0% 100.0% 0.0 0.0% PK11_FreeSymKey > > 0.0 0.0% 100.0% 0.0 0.0% PK11_GetIVLength > > 0.0 0.0% 100.0% 0.0 0.0% PK11_ImportSymKey > > 0.0 0.0% 100.0% 0.0 0.0% PrebufferedStreambuf::overflow > > 0.0 0.0% 100.0% 1.8 0.0% Processor::accept > > 0.0 0.0% 100.0% 0.0 0.0% SECITEM_CopyItem_Util > > 0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_after > > 0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_at > > 0.0 0.0% 100.0% 38.4 0.1% SafeTimer::timer_thread > > 0.0 0.0% 100.0% 38.4 0.1% SafeTimerThread::entry > > 0.0 0.0% 100.0% 25862.7 95.7% ThreadPool::WorkThread::entry > > 0.0 0.0% 100.0% 25862.7 95.7% ThreadPool::worker > > 0.0 0.0% 100.0% 27023.8 100.0% __clone > > 0.0 0.0% 100.0% 0.0 0.0% > > boost::detail::function::void_function_obj_invoker2::invoke > > 0.0 0.0% 100.0% 0.0 0.0% > > boost::proto::detail::default_assign::impl::operator > > 0.0 0.0% 100.0% 0.0 0.0% > > boost::spirit::classic::impl::concrete_parser::do_parse_virtual > > 0.0 0.0% 100.0% 0.0 0.0% boost::spirit::qi::action::parse > > 0.0 0.0% 100.0% 0.3 0.0% > > boost::statechart::event_base::intrusive_from_this > > 0.0 0.0% 100.0% 25802.4 95.4% > > boost::statechart::simple_state::react_impl > > 0.0 0.0% 100.0% 25802.4 95.4% > > boost::statechart::state_machine::send_event > > 0.0 0.0% 100.0% 0.0 0.0% ceph::Formatter::create@48b620 > > 0.0 0.0% 100.0% 0.4 0.0% > > ceph::buffer::list::contiguous_appender::contiguous_appender > > 0.0 0.0% 100.0% 2.4 0.0% ceph::buffer::list::crc32c > > 0.0 0.0% 100.0% 0.1 0.0% > > ceph::buffer::list::iterator_impl::copy > > 0.0 0.0% 100.0% 0.0 0.0% > > ceph::buffer::list::iterator_impl::copy_deep > > 0.0 0.0% 100.0% 5.7 0.0% > > ceph::buffer::list::iterator_impl::copy_shallow > > 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::ptr::ptr > > 0.0 0.0% 100.0% 0.0 0.0% ceph_heap_profiler_handle_command > > 0.0 0.0% 100.0% 0.0 0.0% ceph_os_fremovexattr > > 0.0 0.0% 100.0% 0.0 0.0% cephx_verify_authorizer > > 0.0 0.0% 100.0% 0.0 0.0% cmdmap_from_json > > 0.0 0.0% 100.0% 2.2 0.0% crush_hash_name > > 0.0 0.0% 100.0% 0.1 0.0% decode > > 0.0 0.0% 100.0% 20.1 0.1% entity_addr_t::encode > > 0.0 0.0% 100.0% 0.0 0.0% get_str_vec > > 0.0 0.0% 100.0% 0.0 0.0% int decode_decrypt@c15110 > > 0.0 0.0% 100.0% 0.0 0.0% int decode_decrypt@c15b90 > > 0.0 0.0% 100.0% 0.0 0.0% > > json_spirit::Semantic_actions::new_name > > 0.0 0.0% 100.0% 0.0 0.0% > > json_spirit::Semantic_actions::new_str > > 0.0 0.0% 100.0% 1.1 0.0% json_spirit::Value_impl::get_uint64 > > 0.0 0.0% 100.0% 0.0 0.0% json_spirit::get_str > > 0.0 0.0% 100.0% 0.0 0.0% json_spirit::get_str_ > > 0.0 0.0% 100.0% 0.0 0.0% json_spirit::read > > 0.0 0.0% 100.0% 0.0 0.0% json_spirit::read_range > > 0.0 0.0% 100.0% 0.0 0.0% json_spirit::read_range_or_throw > > 0.0 0.0% 100.0% 0.0 0.0% json_spirit::substitute_esc_chars > > 0.0 0.0% 100.0% 0.0 0.0% operator<<@a91e90 > > 0.0 0.0% 100.0% 3.5 0.0% osd_info_t::encode > > 0.0 0.0% 100.0% 4.4 0.0% osd_xinfo_t::encode > > 0.0 0.0% 100.0% 0.1 0.0% pg_info_t::decode > > 0.0 0.0% 100.0% 0.0 0.0% pg_info_t::operator= > > 0.0 0.0% 100.0% 9.9 0.0% pg_info_t::pg_info_t > > 0.0 0.0% 100.0% 87.5 0.3% pg_interval_t::decode > > 0.0 0.0% 100.0% 1.0 0.0% pg_pool_t::decode > > 0.0 0.0% 100.0% 1.8 0.0% pg_pool_t::encode > > 0.0 0.0% 100.0% 0.0 0.0% pg_stat_t::decode > > 0.0 0.0% 100.0% 27032.1 100.0% start_thread > > 0.0 0.0% 100.0% 1.3 0.0% std::_Rb_tree::operator= > > 0.0 0.0% 100.0% 0.1 0.0% std::_Sp_counted_base::_M_release > > 0.0 0.0% 100.0% 0.0 0.0% std::__detail::_Map_base::operator[] > > 0.0 0.0% 100.0% 0.0 0.0% std::__ostream_insert > > 0.0 0.0% 100.0% 0.1 0.0% std::basic_streambuf::xsputn > > 0.0 0.0% 100.0% 0.1 0.0% std::basic_string::basic_string > > 0.0 0.0% 100.0% 0.0 0.0% std::basic_stringbuf::overflow > > 0.0 0.0% 100.0% 1.0 0.0% std::basic_stringbuf::str > > 0.0 0.0% 100.0% 71.0 0.3% std::enable_if::type encode > > 0.0 0.0% 100.0% 0.1 0.0% std::getline > > 0.0 0.0% 100.0% 0.0 0.0% std::num_put::_M_insert_int > > 0.0 0.0% 100.0% 0.0 0.0% std::num_put::do_put > > 0.0 0.0% 100.0% 0.0 0.0% std::operator<< > > 0.0 0.0% 100.0% 0.0 0.0% std::ostream::_M_insert > > 0.0 0.0% 100.0% 1.2 0.0% std::string::_Rep::_M_clone > > 0.0 0.0% 100.0% 1.2 0.0% std::string::_S_construct > > 0.0 0.0% 100.0% 1.2 0.0% std::string::append > > 0.0 0.0% 100.0% 1.2 0.0% std::string::reserve > > 0.0 0.0% 100.0% 283.5 1.0% std::this_thread::__sleep_for > > 0.0 0.0% 100.0% 0.0 0.0% void decode_decrypt_enc_bl@c12db0 > > 0.0 0.0% 100.0% 0.0 0.0% void decode_decrypt_enc_bl@c14a80 > > 0.0 0.0% 100.0% 0.0 0.0% void decode_decrypt_enc_bl@c15450 > > 0.0 0.0% 100.0% 20.1 0.1% void encode > > > > > > I also generated the PDf with all the charts, but not sure how to share it > > with you guys. > > any Idea what is happening here ? > > > > thanks > > ali > > > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
