Im pretty certain that is the same bug that we are hitting, unfortunately I don't have a simple script to recreate it. It happens as part of a WSGI application in Apache running with 30 threads and multiple processes. If I tune it down to fewer threads and processes, it becomes harder to trigger. If this fix is accepted and integrated, please backport fix to the next Jewel update (10.2.6). -Wyllys Ingersoll Keeper Technology, LLC On Tue, Jan 3, 2017 at 7:22 PM, Josh Durgin <jdurgin@xxxxxxxxxx> wrote: > On 01/03/2017 01:39 PM, Wyllys Ingersoll wrote: >> >> I have a python based WSGI application running inside Apache that uses the >> ceph python bindings to perform some operations. Recently its been >> throwing exceptions when trying to make ceph connections, but I can't >> figure out what is really causing the issue here. >> >> Running ceph 10.2.5 with latest python-ceph packages. > > > This looks like http://tracker.ceph.com/issues/14115 - in your case > likely due to a bug in librados. > >> Anyone seen this sort of thing before or have any idea how to prevent it? > > > You might be able to avoid it by using a single Rados instance per > process, to avoid re-initializing crypto_context within librados. > > If you have a short script that reproduces it, or can run under > valgrind's memcheck tool we can see if your case would be fixed by > > https://github.com/ceph/ceph/pull/12624 > > or if there are other leaks to fix. > > Josh > > >> The following error appears in the Apache error logs: >> >> common/ceph_crypto.cc: In function 'void ceph::crypto::init(CephContext*)' >> thread 7fb1ca8c4700 time 2017-01-03 16:25:31.334606 >> common/ceph_crypto.cc: 77: FAILED assert(crypto_context != __null) >> ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367) >> 1: (()+0x169b7b) [0x7fb1b1b82b7b] >> 2: (()+0x1b88c0) [0x7fb1b1bd18c0] >> 3: (()+0x184615) [0x7fb1b1b9d615] >> 4: (()+0x181800) [0x7fb1b1b9a800] >> 5: (()+0x95f2d) [0x7fb1b1aaef2d] >> 6: (rados_connect()+0x1c) [0x7fb1b1a7dadc] > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
