Re: OSD reaching file open limit - known issues?

Jan Schermer <jan@xxxxxxxxxxx> · Fri, 25 Sep 2015 19:04:29 +0200

I get that, even though I think it should be handled more gracefuly.But is it expected to also lead to consistency issues like this?

I think this is exactly what we're hitting right now http://tracker.ceph.com/issues/6101 except I have no idea why it also happens on a freshly backfilled OSD...

Jan

On 25 Sep 2015, at 19:02, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote:

Yes, known issue, make sure your system open file limit is pretty high..

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Jan Schermer
Sent: Friday, September 25, 2015 4:42 AM
To: ceph-users@xxxxxxxxxxxxxx
Subject:  OSD reaching file open limit - known issues?

Hi,
we recently migrated some of our nodes to Ubuntu 12, which helped everything quite a bit.

But we hit a snag where the upstart initscript would not set the file open ulimit correctly and some OSDs ran out of fds.

Some roblems manifested since then on the node where this happened such as scrub errors (which were corrected - don't ask me how, I was sleeping :-))  - but not two of the OSDs on this node started failing with SIGABORT:

2015-09-25 10:45:17.860461 361ea17e700 -1 osd.12 pg_epoch: 1209679 pg[4.3d85( v 1209679'13913518 (1209090'13910508,1209679'13913518] local-les=1209679 n=235 ec=3 les/c 1209679/1209679 1209678/1209678/1209678) [12,36,59] r=0 lpr=1209678 mlcod 1209656'13913517 active+clean snaptrimq=[857c0~1,857f4~1,85a43~2]] trim_objectcould not find coid 783dbd85/rbd_data.1a785181f15746a.00000000000238df/857c0//4
2015-09-25 10:45:17.862019 361ea17e700 -1 osd/ReplicatedPG.cc: In function 'ReplicatedPG::RepGather* ReplicatedPG::trim_object(const hobject_t&)' thread 361ea17e700 time 2015-09-25 10:45:17.860501
osd/ReplicatedPG.cc: 1510: FAILED assert(0)

 ceph version 0.67.11-82-ge5b6eea (e5b6eea91cc37434f78a987d2dd1d3edd4a23f3f)
 1: (ReplicatedPG::trim_object(hobject_t const&)+0x150) [0x6e8bd0]
 2: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim const&)+0x2e7) [0x6ee0d7]
 3: (boost::statechart::detail::reaction_result boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<ReplicatedPG::SnapTrim>, boost::statechart::transition<ReplicatedPG::Reset, ReplicatedPG::NotTrimming, boost::statechart::detail::no_context<ReplicatedPG::Reset>, &(boost::statechart::detail::no_context<ReplicatedPG::Reset>::no_function(ReplicatedPG::Reset const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x96) [0x740fa6]
 4: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x137) [0x71bdf7]
 5: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x26) [0x71cfe6]
 6: (ReplicatedPG::snap_trimmer()+0x4ed) [0x6b59ad]
 7: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x790c54]
 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68c) [0x9a69cc]
 9: (ThreadPool::WorkThread::entry()+0x10) [0x9a7c20]
 10: (()+0x7e9a) [0x36258121e9a]
 11: (clone()+0x6d) [0x3625669638d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
(full log available on request, file system will be thrashed shortly so tell me if it's helpful to look for something in there)

Could this all be caused by the OSD running out of file descriptors? Is it supposed to handle a problem like this (meaning both the assert that happened now and the file descriptor limit) gracefuly? Or is it a known issue that this could happen?
The thing about upstart is it apparently keeps restarting the OSD, which makes the problem even worse.

Luckily we caught this in time and it only happened on one node, so we are thrashing all the OSDs here.

Looks like a problem that could hit anyone, and if it actually damages data then it could be pretty bad and maybe worth looking into - tell me what more is needed.

Config:
XFS filesystem
Ubuntu 12 with 3.14.37 kernel
FIEMAP disabled
Ceph Dumpling (0.67.11-82-ge5b6eea)

Thanks

Jan

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com