| Hi, we recently migrated some of our nodes to Ubuntu 12, which helped everything quite a bit. But we hit a snag where the upstart initscript would not set the file open ulimit correctly and some OSDs ran out of fds. Some roblems manifested since then on the node where this happened such as scrub errors (which were corrected - don't ask me how, I was sleeping :-)) - but not two of the OSDs on this node started failing with SIGABORT: 2015-09-25 10:45:17.860461 361ea17e700 -1 osd.12 pg_epoch: 1209679 pg[4.3d85( v 1209679'13913518 (1209090'13910508,1209679'13913518] local-les=1209679 n=235 ec=3 les/c 1209679/1209679 1209678/1209678/1209678) [12,36,59] r=0 lpr=1209678 mlcod 1209656'13913517 active+clean snaptrimq=[857c0~1,857f4~1,85a43~2]] trim_objectcould not find coid 783dbd85/rbd_data.1a785181f15746a.00000000000238df/857c0//4 2015-09-25 10:45:17.862019 361ea17e700 -1 osd/ReplicatedPG.cc: In function 'ReplicatedPG::RepGather* ReplicatedPG::trim_object(const hobject_t&)' thread 361ea17e700 time 2015-09-25 10:45:17.860501 osd/ReplicatedPG.cc: 1510: FAILED assert(0) ceph version 0.67.11-82-ge5b6eea (e5b6eea91cc37434f78a987d2dd1d3edd4a23f3f) 1: (ReplicatedPG::trim_object(hobject_t const&)+0x150) [0x6e8bd0] 2: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim const&)+0x2e7) [0x6ee0d7] 3: (boost::statechart::detail::reaction_result boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<ReplicatedPG::SnapTrim>, boost::statechart::transition<ReplicatedPG::Reset, ReplicatedPG::NotTrimming, boost::statechart::detail::no_context<ReplicatedPG::Reset>, &(boost::statechart::detail::no_context<ReplicatedPG::Reset>::no_function(ReplicatedPG::Reset const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x96) [0x740fa6] 4: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x137) [0x71bdf7] 5: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x26) [0x71cfe6] 6: (ReplicatedPG::snap_trimmer()+0x4ed) [0x6b59ad] 7: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x790c54] 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68c) [0x9a69cc] 9: (ThreadPool::WorkThread::entry()+0x10) [0x9a7c20] 10: (()+0x7e9a) [0x36258121e9a] 11: (clone()+0x6d) [0x3625669638d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. (full log available on request, file system will be thrashed shortly so tell me if it's helpful to look for something in there) Could this all be caused by the OSD running out of file descriptors? Is it supposed to handle a problem like this (meaning both the assert that happened now and the file descriptor limit) gracefuly? Or is it a known issue that this could happen? The thing about upstart is it apparently keeps restarting the OSD, which makes the problem even worse. Luckily we caught this in time and it only happened on one node, so we are thrashing all the OSDs here. Looks like a problem that could hit anyone, and if it actually damages data then it could be pretty bad and maybe worth looking into - tell me what more is needed. Config: XFS filesystem Ubuntu 12 with 3.14.37 kernel FIEMAP disabled Ceph Dumpling (0.67.11-82-ge5b6eea) Thanks Jan |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
