do you try to lower osd_map_cache_size = 20 ? On Mon, Aug 21, 2017 at 6:57 PM, Linux Chips <linux.chips@xxxxxxxxx> wrote: > Hi, > I have an idea to move pools out of the "current" directory (like move them > into a directory "current.bak"), and keep only one pool at a time in there > so the OSD would load less PGs. > any one tried to do this before? will we have a data loss? > > > > On 08/17/2017 09:51 PM, Linux Chips wrote: >> >> >> >> On 08/17/2017 08:53 PM, Gregory Farnum wrote: >>> >>> On Thu, Aug 17, 2017 at 7:13 AM, Linux Chips <linux.chips@xxxxxxxxx> >>> wrote: >>>> >>>> Hello everybody, >>>> I have Kraken cluster with 660 OSD, currently it is down due to not >>>> being able to complete peering, OSDs start consuming lots of memory >>>> draining the system and killing the node, so I set a limit on the OSD >>>> service (on some OSDs 28G and others as high as 35G), so they get >>>> killed before taking down the whole node. >>>> Now I still can't peer, one OSD entering the cluster (with about 300 >>>> already up) makes memory usage of most other OSDs so high (15G+, some as >>>> much as 30G) and >>>> sometimes kills them when they reach the service limit. which cause a >>>> spiral >>>> load and causing all the OSDs to consume all the available. >>>> >>>> I found this thread with similar symptoms: >>>> >>>> >>>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017522.html >>>> >>>> with a request for stack trace, I have a 14G core dump, we generated it >>>> by >>>> running the osd from the terminal, enabling the core dumps, and setting >>>> ulimits to 15G. what kind of a trace would be useful? all thread?! any >>>> better way to debug this? >>>> >>>> What can I do do make it work, is this memory allocation normal? >>>> >>>> some info about the cluster: >>>> 41 hdd nodes with 12 x 4TB osd each, 5 of the nodes have 8TB disks. 324 >>>> GB >>>> RAM and dula socket intel xeon. >>>> 7 nodes with 400GB x 24 ssd and 256GB RAM, and dual socket cpu. >>>> 3 monitors >>>> >>>> all dual 10GB ethernet, except for the monitor with dual 1GB ethers. >>>> >>>> all nodes running centos 7.2 >>>> it is an old cluster that was upgraded continuously for the past 3 >>>> years. >>>> the cluster was on jewel when the issue happened due to some accidental >>>> OSD >>>> map changes, causing a heavy recovery operations on the cluster. then we >>>> upgraded to kraken in the hope of less memory foot prints. >>>> >>>> any advice on how to proceed? >>> >>> It's not normal but if something really bad happened to your cluster, >>> it's been known to occur. You should go through the troubleshooting >>> guides at docs.ceph.com, but the general strategy is to set >>> nodown/noout/etc flags, undo whatever horrible thing you tried to make >>> the map do, and then turn all the OSDs back on. >>> -Greg >> >> >> Hi, >> we have been trying this for the past week, it keeps consuming the RAM. >> we got the map back to the original places. marked all the flags, started >> all the OSDs. then "ceph osd unset noup", wait 5 min, and all OSDs are >> killed by the oom. >> we tried one node at a time, let it finish recovering, and start the next. >> we got to a point when we started the next node, every thing got killed. >> we tried one OSD at a time, same result. one OSD up, ~40 killed by oom, >> then it is a snow ball from here until all of the active OSDs get kiiled. >> >> I think all this up/down that we generated has increased the recovery too >> much. btw, we stopped all clients. and also we have some not so friendly >> erasure pools. some OSDs now report loading as much as 800 pg, while we >> originally had about 300-400 (I know too much, but we were trying to fix it >> and.... well we could not). >> >> we did a memory profiling on one of the OSDs. >> here is the results >> >> >> 12878.6 47.6% 47.6% 12878.6 47.6% std::_Rb_tree::_M_create_node >> 12867.6 47.6% 95.2% 25746.2 95.2% std::_Rb_tree::_M_copy >> 532.4 2.0% 97.2% 686.3 2.5% OSD::heartbeat >> 122.8 0.5% 97.7% 122.8 0.5% >> std::_Rb_tree::_M_emplace_hint_unique >> 121.9 0.5% 98.1% 171.1 0.6% AsyncConnection::send_message >> 104.2 0.4% 98.5% 104.2 0.4% ceph::buffer::list::append@c4a770 >> 99.7 0.4% 98.9% 99.7 0.4% std::vector::_M_default_append >> 99.6 0.4% 99.2% 99.6 0.4% ceph::logging::Log::create_entry >> 72.6 0.3% 99.5% 72.6 0.3% ceph::buffer::create_aligned >> 52.4 0.2% 99.7% 52.5 0.2% std::vector::_M_emplace_back_aux >> 23.9 0.1% 99.8% 57.8 0.2% OSD::do_notifies >> 17.0 0.1% 99.8% 23.1 0.1% >> OSDService::build_incremental_map_msg >> 9.8 0.0% 99.9% 222.5 0.8% std::enable_if::type decode >> 6.2 0.0% 99.9% 6.3 0.0% std::map::operator[] >> 5.5 0.0% 99.9% 5.5 0.0% std::vector::vector >> 3.5 0.0% 99.9% 3.5 0.0% EventCenter::create_time_event >> 2.5 0.0% 99.9% 2.5 0.0% AsyncConnection::AsyncConnection >> 2.4 0.0% 100.0% 2.4 0.0% std::string::_Rep::_S_create >> 1.5 0.0% 100.0% 1.5 0.0% std::_Rb_tree::_M_insert_unique >> 1.4 0.0% 100.0% 1.4 0.0% std::list::operator= >> 1.3 0.0% 100.0% 1.3 0.0% ceph::buffer::list::list >> 0.9 0.0% 100.0% 204.1 0.8% decode_message >> 0.8 0.0% 100.0% 0.8 0.0% OSD::send_failures >> 0.7 0.0% 100.0% 0.9 0.0% void decode >> 0.6 0.0% 100.0% 0.6 0.0% std::_Rb_tree::_M_insert_equal >> 0.6 0.0% 100.0% 2.5 0.0% PG::queue_null >> 0.6 0.0% 100.0% 1.8 0.0% AsyncMessenger::create_connect >> 0.6 0.0% 100.0% 1.8 0.0% AsyncMessenger::add_accept >> 0.5 0.0% 100.0% 0.5 0.0% boost::statechart::event::clone >> 0.4 0.0% 100.0% 0.4 0.0% PG::queue_peering_event >> 0.3 0.0% 100.0% 0.3 0.0% OSD::PeeringWQ::_enqueue >> 0.3 0.0% 100.0% 148.6 0.5% OSD::_dispatch >> 0.1 0.0% 100.0% 147.9 0.5% OSD::handle_osd_map >> 0.1 0.0% 100.0% 0.1 0.0% std::deque::_M_push_back_aux >> 0.1 0.0% 100.0% 0.2 0.0% SharedLRU::add >> 0.1 0.0% 100.0% 0.1 0.0% OSD::PeeringWQ::_dequeue >> 0.1 0.0% 100.0% 0.1 0.0% ceph::buffer::list::append@c4a9b0 >> 0.1 0.0% 100.0% 0.2 0.0% DispatchQueue::enqueue >> 0.1 0.0% 100.0% 283.5 1.0% EventCenter::process_events >> 0.1 0.0% 100.0% 0.1 0.0% HitSet::Params::create_impl >> 0.1 0.0% 100.0% 0.1 0.0% SimpleLRU::clear_pinned >> 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_ >> 0.0 0.0% 100.0% 0.2 0.0% TrackedOp::mark_event >> 0.0 0.0% 100.0% 0.0 0.0% OSD::create_context >> 0.0 0.0% 100.0% 0.0 0.0% std::_Hashtable::_M_allocate_node >> 0.0 0.0% 100.0% 0.0 0.0% OSDMap::OSDMap >> 0.0 0.0% 100.0% 281.6 1.0% AsyncConnection::process >> 0.0 0.0% 100.0% 25802.4 95.4% >> PG::RecoveryState::RecoveryMachine::send_notify >> 0.0 0.0% 100.0% 0.0 0.0% SharedLRU::lru_add >> 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_unique_ >> 0.0 0.0% 100.0% 0.1 0.0% OpTracker::unregister_inflight_op >> 0.0 0.0% 100.0% 0.0 0.0% OSD::ms_verify_authorizer >> 0.0 0.0% 100.0% 0.0 0.0% OSDService::_add_map >> 0.0 0.0% 100.0% 0.1 0.0% OSD::wait_for_new_map >> 0.0 0.0% 100.0% 0.5 0.0% OSD::handle_pg_notify >> 0.0 0.0% 100.0% 0.0 0.0% std::__shared_count::__shared_count >> 0.0 0.0% 100.0% 0.0 0.0% std::__shared_ptr::reset >> 0.0 0.0% 100.0% 35.1 0.1% OSDMap::decode@b84080 >> 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_emplace_unique >> 0.0 0.0% 100.0% 0.0 0.0% std::vector::operator= >> 0.0 0.0% 100.0% 0.0 0.0% MonClient::_renew_subs >> 0.0 0.0% 100.0% 0.0 0.0% std::_Hashtable::_M_emplace >> 0.0 0.0% 100.0% 0.0 0.0% PORT_Alloc_Util >> 0.0 0.0% 100.0% 0.0 0.0% CryptoAES::get_key_handler >> 0.0 0.0% 100.0% 0.0 0.0% get_auth_session_handler >> 0.0 0.0% 100.0% 0.0 0.0% PosixWorker::connect >> 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::append@c4a440 >> 0.0 0.0% 100.0% 0.0 0.0% std::vector::_M_fill_insert >> 0.0 0.0% 100.0% 4.8 0.0% AsyncConnection::fault >> 0.0 0.0% 100.0% 0.0 0.0% OSD::send_pg_stats >> 0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::accept_conn >> 0.0 0.0% 100.0% 0.0 0.0% PosixServerSocketImpl::accept >> 0.0 0.0% 100.0% 9.3 0.0% >> AsyncConnection::_process_connection >> 0.0 0.0% 100.0% 0.2 0.0% FileStore::lfn_open >> 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::append@c4a350 >> 0.0 0.0% 100.0% 0.0 0.0% crush_create >> 0.0 0.0% 100.0% 0.1 0.0% MgrClient::send_report >> 0.0 0.0% 100.0% 0.0 0.0% WBThrottle::queue_wb >> 0.0 0.0% 100.0% 0.2 0.0% LogClient::_get_mon_log_message >> 0.0 0.0% 100.0% 0.0 0.0% CryptoKey::_set_secret >> 0.0 0.0% 100.0% 0.0 0.0% std::_Deque_base::_M_initialize_map >> 0.0 0.0% 100.0% 0.1 0.0% >> ThreadPool::BatchWorkQueue::_void_dequeue >> 0.0 0.0% 100.0% 0.0 0.0% ceph::Formatter::create@ba6a50 >> 0.0 0.0% 100.0% 0.0 0.0% MonClient::schedule_tick >> 0.0 0.0% 100.0% 0.1 0.0% OSD::tick >> 0.0 0.0% 100.0% 37.6 0.1% OSD::tick_without_osd_lock >> 0.0 0.0% 100.0% 0.0 0.0% >> boost::spirit::classic::impl::get_definition >> 0.0 0.0% 100.0% 9.4 0.0% MonClient::_send_mon_message >> 0.0 0.0% 100.0% 0.0 0.0% DispatchQueue::queue_refused >> 0.0 0.0% 100.0% 0.0 0.0% OSD::handle_command >> 0.0 0.0% 100.0% 0.0 0.0% DispatchQueue::queue_accept >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::_connect >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::_stop >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::accept >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::handle_connect_msg >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::mark_down >> 0.0 0.0% 100.0% 0.0 0.0% >> AsyncConnection::prepare_send_message >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::read_bulk >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::read_until >> 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::send_keepalive >> 0.0 0.0% 100.0% 3.3 0.0% AsyncConnection::wakeup_from >> 0.0 0.0% 100.0% 1.8 0.0% AsyncMessenger::get_connection >> 0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::reap_dead >> 0.0 0.0% 100.0% 2.5 0.0% C_OnMapCommit::finish >> 0.0 0.0% 100.0% 0.0 0.0% >> CephXTicketHandler::verify_service_ticket_reply >> 0.0 0.0% 100.0% 0.0 0.0% >> CephXTicketManager::verify_service_ticket_reply >> 0.0 0.0% 100.0% 0.0 0.0% >> CephxAuthorizeHandler::verify_authorizer >> 0.0 0.0% 100.0% 0.0 0.0% CephxClientHandler::handle_response >> 0.0 0.0% 100.0% 40.9 0.2% Context::complete >> 0.0 0.0% 100.0% 4.8 0.0% CrushWrapper::encode >> 0.0 0.0% 100.0% 0.0 0.0% CryptoAESKeyHandler::decrypt >> 0.0 0.0% 100.0% 0.0 0.0% CryptoKey::decode >> 0.0 0.0% 100.0% 160.4 0.6% >> DispatchQueue::DispatchThread::entry >> 0.0 0.0% 100.0% 160.4 0.6% DispatchQueue::entry >> 0.0 0.0% 100.0% 0.4 0.0% DispatchQueue::fast_dispatch >> 0.0 0.0% 100.0% 0.4 0.0% DispatchQueue::pre_dispatch >> 0.0 0.0% 100.0% 0.0 0.0% EntityName::set >> 0.0 0.0% 100.0% 0.0 0.0% EpollDriver::event_wait >> 0.0 0.0% 100.0% 3.0 0.0% >> EventCenter::dispatch_event_external >> 0.0 0.0% 100.0% 3.3 0.0% EventCenter::process_time_events >> 0.0 0.0% 100.0% 3.0 0.0% EventCenter::wakeup >> 0.0 0.0% 100.0% 0.0 0.0% FileJournal::prepare_entry >> 0.0 0.0% 100.0% 0.2 0.0% FileStore::_do_op >> 0.0 0.0% 100.0% 0.2 0.0% FileStore::_do_transaction >> 0.0 0.0% 100.0% 0.2 0.0% FileStore::_do_transactions >> 0.0 0.0% 100.0% 0.0 0.0% FileStore::_journaled_ahead >> 0.0 0.0% 100.0% 0.2 0.0% FileStore::_write >> 0.0 0.0% 100.0% 0.0 0.0% FileStore::queue_transactions >> 0.0 0.0% 100.0% 2.6 0.0% Finisher::finisher_thread_entry >> 0.0 0.0% 100.0% 0.1 0.0% FunctionContext::finish >> 0.0 0.0% 100.0% 0.1 0.0% HitSet::Params::decode >> 0.0 0.0% 100.0% 0.2 0.0% LogChannel::do_log@a90a00 >> 0.0 0.0% 100.0% 0.3 0.0% LogChannel::do_log@a91030 >> 0.0 0.0% 100.0% 0.2 0.0% LogClient::get_mon_log_message >> 0.0 0.0% 100.0% 0.0 0.0% LogClient::handle_log_ack >> 0.0 0.0% 100.0% 0.1 0.0% LogClient::queue >> 0.0 0.0% 100.0% 0.3 0.0% LogClientTemp::~LogClientTemp >> 0.0 0.0% 100.0% 0.0 0.0% MAuthReply::decode_payload >> 0.0 0.0% 100.0% 0.0 0.0% MCommand::decode_payload >> 0.0 0.0% 100.0% 0.0 0.0% MCommand::print >> 0.0 0.0% 100.0% 0.0 0.0% MMgrMap::decode_payload >> 0.0 0.0% 100.0% 0.0 0.0% MOSDFailure::print >> 0.0 0.0% 100.0% 0.1 0.0% MOSDMap::decode_payload >> 0.0 0.0% 100.0% 203.1 0.8% MOSDPGNotify::decode_payload >> 0.0 0.0% 100.0% 0.0 0.0% MOSDPGNotify::print >> 0.0 0.0% 100.0% 0.0 0.0% MOSDPing::encode_payload >> 0.0 0.0% 100.0% 0.0 0.0% Message::encode >> 0.0 0.0% 100.0% 0.0 0.0% MgrClient::handle_mgr_map >> 0.0 0.0% 100.0% 0.0 0.0% MgrClient::ms_dispatch >> 0.0 0.0% 100.0% 0.0 0.0% MgrMap::decode >> 0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_rotating >> 0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_tickets >> 0.0 0.0% 100.0% 0.0 0.0% MonClient::_finish_hunting >> 0.0 0.0% 100.0% 0.8 0.0% MonClient::_reopen_session@aeab80 >> 0.0 0.0% 100.0% 0.6 0.0% MonClient::_reopen_session@af2ba0 >> 0.0 0.0% 100.0% 9.5 0.0% MonClient::handle_auth >> 0.0 0.0% 100.0% 9.6 0.0% MonClient::ms_dispatch >> 0.0 0.0% 100.0% 0.2 0.0% MonClient::send_log >> 0.0 0.0% 100.0% 0.6 0.0% MonClient::tick >> 0.0 0.0% 100.0% 283.5 1.0% NetworkStack::get_worker >> 0.0 0.0% 100.0% 0.0 0.0% OSD::CommandWQ::_process >> 0.0 0.0% 100.0% 25862.5 95.7% OSD::PeeringWQ::_process >> 0.0 0.0% 100.0% 0.0 0.0% OSD::Session::Session >> 0.0 0.0% 100.0% 686.3 2.5% OSD::T_Heartbeat::entry >> 0.0 0.0% 100.0% 2.5 0.0% OSD::_committed_osd_maps >> 0.0 0.0% 100.0% 25804.6 95.5% OSD::advance_pg >> 0.0 0.0% 100.0% 0.3 0.0% OSD::check_ops_in_flight >> 0.0 0.0% 100.0% 0.0 0.0% OSD::check_osdmap_features >> 0.0 0.0% 100.0% 2.5 0.0% OSD::consume_map >> 0.0 0.0% 100.0% 57.8 0.2% OSD::dispatch_context >> 0.0 0.0% 100.0% 0.5 0.0% OSD::dispatch_op >> 0.0 0.0% 100.0% 0.0 0.0% OSD::do_command >> 0.0 0.0% 100.0% 0.2 0.0% OSD::do_waiters >> 0.0 0.0% 100.0% 0.0 0.0% OSD::get_osdmap_pobject_name >> 0.0 0.0% 100.0% 0.1 0.0% OSD::handle_osd_ping >> 0.0 0.0% 100.0% 0.0 0.0% OSD::handle_pg_peering_evt >> 0.0 0.0% 100.0% 37.2 0.1% OSD::heartbeat_check >> 0.0 0.0% 100.0% 0.1 0.0% OSD::heartbeat_dispatch >> 0.0 0.0% 100.0% 686.3 2.5% OSD::heartbeat_entry >> 0.0 0.0% 100.0% 1.1 0.0% OSD::heartbeat_reset >> 0.0 0.0% 100.0% 148.7 0.6% OSD::ms_dispatch >> 0.0 0.0% 100.0% 0.8 0.0% OSD::ms_handle_connect >> 0.0 0.0% 100.0% 0.0 0.0% OSD::ms_handle_refused >> 0.0 0.0% 100.0% 0.0 0.0% OSD::ms_handle_reset >> 0.0 0.0% 100.0% 25862.5 95.7% OSD::process_peering_events >> 0.0 0.0% 100.0% 0.1 0.0% OSD::require_same_or_newer_map >> 0.0 0.0% 100.0% 0.0 0.0% OSD::write_superblock >> 0.0 0.0% 100.0% 0.0 0.0% OSDCap::parse >> 0.0 0.0% 100.0% 0.0 0.0% OSDMap::Incremental::decode >> 0.0 0.0% 100.0% 35.1 0.1% OSDMap::decode@b85440 >> 0.0 0.0% 100.0% 110.8 0.4% OSDMap::encode >> 0.0 0.0% 100.0% 0.5 0.0% OSDMap::post_decode >> 0.0 0.0% 100.0% 0.1 0.0% OSDService::_get_map_bl >> 0.0 0.0% 100.0% 0.0 0.0% OSDService::check_nearfull_warning >> 0.0 0.0% 100.0% 0.1 0.0% OSDService::clear_map_bl_cache_pins >> 0.0 0.0% 100.0% 1.1 0.0% OSDService::get_con_osd_hb >> 0.0 0.0% 100.0% 1.3 0.0% OSDService::get_inc_map_bl >> 0.0 0.0% 100.0% 1.3 0.0% OSDService::pin_map_bl >> 0.0 0.0% 100.0% 0.0 0.0% OSDService::pin_map_inc_bl >> 0.0 0.0% 100.0% 0.0 0.0% OSDService::publish_superblock >> 0.0 0.0% 100.0% 0.3 0.0% OSDService::queue_for_peering >> 0.0 0.0% 100.0% 27.2 0.1% OSDService::send_incremental_map >> 0.0 0.0% 100.0% 27.2 0.1% OSDService::share_map_peer >> 0.0 0.0% 100.0% 0.0 0.0% OSDService::update_osd_stat >> 0.0 0.0% 100.0% 0.0 0.0% >> ObjectStore::Transaction::_get_coll_id >> 0.0 0.0% 100.0% 0.0 0.0% >> ObjectStore::Transaction::_get_next_op >> 0.0 0.0% 100.0% 0.2 0.0% ObjectStore::Transaction::write >> 0.0 0.0% 100.0% 0.0 0.0% ObjectStore::queue_transaction >> 0.0 0.0% 100.0% 0.0 0.0% Objecter::_maybe_request_map >> 0.0 0.0% 100.0% 0.1 0.0% Objecter::handle_osd_map >> 0.0 0.0% 100.0% 0.1 0.0% OpHistory::insert >> 0.0 0.0% 100.0% 0.0 0.0% OpRequest::OpRequest >> 0.0 0.0% 100.0% 0.1 0.0% OpRequest::mark_flag_point >> 0.0 0.0% 100.0% 0.1 0.0% OpRequest::mark_started >> 0.0 0.0% 100.0% 0.1 0.0% OpTracker::RemoveOnDelete::operator >> 0.0 0.0% 100.0% 0.1 0.0% OpTracker::_mark_event >> 0.0 0.0% 100.0% 0.0 0.0% OpTracker::get_age_ms_histogram >> 0.0 0.0% 100.0% 25802.4 95.4% PG::RecoveryState::Stray::react >> 0.0 0.0% 100.0% 0.0 0.0% PG::_prepare_write_info >> 0.0 0.0% 100.0% 25802.4 95.4% PG::handle_activate_map >> 0.0 0.0% 100.0% 1.6 0.0% PG::handle_advance_map >> 0.0 0.0% 100.0% 0.0 0.0% PG::prepare_write_info >> 0.0 0.0% 100.0% 0.0 0.0% PG::write_if_dirty >> 0.0 0.0% 100.0% 1.6 0.0% PGPool::update >> 0.0 0.0% 100.0% 0.0 0.0% PK11_FreeSymKey >> 0.0 0.0% 100.0% 0.0 0.0% PK11_GetIVLength >> 0.0 0.0% 100.0% 0.0 0.0% PK11_ImportSymKey >> 0.0 0.0% 100.0% 0.0 0.0% PrebufferedStreambuf::overflow >> 0.0 0.0% 100.0% 1.8 0.0% Processor::accept >> 0.0 0.0% 100.0% 0.0 0.0% SECITEM_CopyItem_Util >> 0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_after >> 0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_at >> 0.0 0.0% 100.0% 38.4 0.1% SafeTimer::timer_thread >> 0.0 0.0% 100.0% 38.4 0.1% SafeTimerThread::entry >> 0.0 0.0% 100.0% 25862.7 95.7% ThreadPool::WorkThread::entry >> 0.0 0.0% 100.0% 25862.7 95.7% ThreadPool::worker >> 0.0 0.0% 100.0% 27023.8 100.0% __clone >> 0.0 0.0% 100.0% 0.0 0.0% >> boost::detail::function::void_function_obj_invoker2::invoke >> 0.0 0.0% 100.0% 0.0 0.0% >> boost::proto::detail::default_assign::impl::operator >> 0.0 0.0% 100.0% 0.0 0.0% >> boost::spirit::classic::impl::concrete_parser::do_parse_virtual >> 0.0 0.0% 100.0% 0.0 0.0% boost::spirit::qi::action::parse >> 0.0 0.0% 100.0% 0.3 0.0% >> boost::statechart::event_base::intrusive_from_this >> 0.0 0.0% 100.0% 25802.4 95.4% >> boost::statechart::simple_state::react_impl >> 0.0 0.0% 100.0% 25802.4 95.4% >> boost::statechart::state_machine::send_event >> 0.0 0.0% 100.0% 0.0 0.0% ceph::Formatter::create@48b620 >> 0.0 0.0% 100.0% 0.4 0.0% >> ceph::buffer::list::contiguous_appender::contiguous_appender >> 0.0 0.0% 100.0% 2.4 0.0% ceph::buffer::list::crc32c >> 0.0 0.0% 100.0% 0.1 0.0% >> ceph::buffer::list::iterator_impl::copy >> 0.0 0.0% 100.0% 0.0 0.0% >> ceph::buffer::list::iterator_impl::copy_deep >> 0.0 0.0% 100.0% 5.7 0.0% >> ceph::buffer::list::iterator_impl::copy_shallow >> 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::ptr::ptr >> 0.0 0.0% 100.0% 0.0 0.0% ceph_heap_profiler_handle_command >> 0.0 0.0% 100.0% 0.0 0.0% ceph_os_fremovexattr >> 0.0 0.0% 100.0% 0.0 0.0% cephx_verify_authorizer >> 0.0 0.0% 100.0% 0.0 0.0% cmdmap_from_json >> 0.0 0.0% 100.0% 2.2 0.0% crush_hash_name >> 0.0 0.0% 100.0% 0.1 0.0% decode >> 0.0 0.0% 100.0% 20.1 0.1% entity_addr_t::encode >> 0.0 0.0% 100.0% 0.0 0.0% get_str_vec >> 0.0 0.0% 100.0% 0.0 0.0% int decode_decrypt@c15110 >> 0.0 0.0% 100.0% 0.0 0.0% int decode_decrypt@c15b90 >> 0.0 0.0% 100.0% 0.0 0.0% >> json_spirit::Semantic_actions::new_name >> 0.0 0.0% 100.0% 0.0 0.0% >> json_spirit::Semantic_actions::new_str >> 0.0 0.0% 100.0% 1.1 0.0% json_spirit::Value_impl::get_uint64 >> 0.0 0.0% 100.0% 0.0 0.0% json_spirit::get_str >> 0.0 0.0% 100.0% 0.0 0.0% json_spirit::get_str_ >> 0.0 0.0% 100.0% 0.0 0.0% json_spirit::read >> 0.0 0.0% 100.0% 0.0 0.0% json_spirit::read_range >> 0.0 0.0% 100.0% 0.0 0.0% json_spirit::read_range_or_throw >> 0.0 0.0% 100.0% 0.0 0.0% json_spirit::substitute_esc_chars >> 0.0 0.0% 100.0% 0.0 0.0% operator<<@a91e90 >> 0.0 0.0% 100.0% 3.5 0.0% osd_info_t::encode >> 0.0 0.0% 100.0% 4.4 0.0% osd_xinfo_t::encode >> 0.0 0.0% 100.0% 0.1 0.0% pg_info_t::decode >> 0.0 0.0% 100.0% 0.0 0.0% pg_info_t::operator= >> 0.0 0.0% 100.0% 9.9 0.0% pg_info_t::pg_info_t >> 0.0 0.0% 100.0% 87.5 0.3% pg_interval_t::decode >> 0.0 0.0% 100.0% 1.0 0.0% pg_pool_t::decode >> 0.0 0.0% 100.0% 1.8 0.0% pg_pool_t::encode >> 0.0 0.0% 100.0% 0.0 0.0% pg_stat_t::decode >> 0.0 0.0% 100.0% 27032.1 100.0% start_thread >> 0.0 0.0% 100.0% 1.3 0.0% std::_Rb_tree::operator= >> 0.0 0.0% 100.0% 0.1 0.0% std::_Sp_counted_base::_M_release >> 0.0 0.0% 100.0% 0.0 0.0% >> std::__detail::_Map_base::operator[] >> 0.0 0.0% 100.0% 0.0 0.0% std::__ostream_insert >> 0.0 0.0% 100.0% 0.1 0.0% std::basic_streambuf::xsputn >> 0.0 0.0% 100.0% 0.1 0.0% std::basic_string::basic_string >> 0.0 0.0% 100.0% 0.0 0.0% std::basic_stringbuf::overflow >> 0.0 0.0% 100.0% 1.0 0.0% std::basic_stringbuf::str >> 0.0 0.0% 100.0% 71.0 0.3% std::enable_if::type encode >> 0.0 0.0% 100.0% 0.1 0.0% std::getline >> 0.0 0.0% 100.0% 0.0 0.0% std::num_put::_M_insert_int >> 0.0 0.0% 100.0% 0.0 0.0% std::num_put::do_put >> 0.0 0.0% 100.0% 0.0 0.0% std::operator<< >> 0.0 0.0% 100.0% 0.0 0.0% std::ostream::_M_insert >> 0.0 0.0% 100.0% 1.2 0.0% std::string::_Rep::_M_clone >> 0.0 0.0% 100.0% 1.2 0.0% std::string::_S_construct >> 0.0 0.0% 100.0% 1.2 0.0% std::string::append >> 0.0 0.0% 100.0% 1.2 0.0% std::string::reserve >> 0.0 0.0% 100.0% 283.5 1.0% std::this_thread::__sleep_for >> 0.0 0.0% 100.0% 0.0 0.0% void decode_decrypt_enc_bl@c12db0 >> 0.0 0.0% 100.0% 0.0 0.0% void decode_decrypt_enc_bl@c14a80 >> 0.0 0.0% 100.0% 0.0 0.0% void decode_decrypt_enc_bl@c15450 >> 0.0 0.0% 100.0% 20.1 0.1% void encode >> >> >> I also generated the PDf with all the charts, but not sure how to share it >> with you guys. >> any Idea what is happening here ? >> >> thanks >> ali >> >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
