On Sep 18, 2013, at 11:50 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: > On Wed, Sep 18, 2013 at 6:33 AM, Dan Van Der Ster > <daniel.vanderster@xxxxxxx> wrote: >> Hi, >> We just finished debugging a problem with RBD-backed Glance image creation failures, and thought our workaround would be useful for others. Basically, we found that during an image upload, librbd on the glance api server was consuming many many processes, eventually hitting the 1024 nproc limit of non-root users in RHEL. The failure occurred when uploading to pools with 2048 PGs, but didn't fail when uploading to pools with 512 PGs (we're guessing that librbd is opening one thread per accessed-PG, and not closing those threads until the whole processes completes.) >> >> If you hit this same problem (and you run RHEL like us), you'll need to modify at least /etc/security/limits.d/90-nproc.conf (adding your non-root user that should be allowed > 1024 procs), and then also possibly run ulimit -u in the init script of your client process. Ubuntu should have some similar limits. > > Did your pools with 2048 PGs have a significantly larger number of > OSDs in them? Or are both pools on a pool with a lot of OSDs relative > to the PG counts? 1056 OSDs at the moment. Uploading a 14GB image we observed up to ~1500 threads. We set the glance client to allow 4096 processes for now. > The PG count shouldn't matter for this directly, but RBD (and other > clients) will create a couple messenger threads for each OSD it talks > to, and while they'll eventually shut down on idle it doesn't > proactively close them. I'd expect this to be a problem around 500 > OSDs. A couple, is that the upper limit? Should we be safe with ulimit -u 2*nOSDs +1 ?? Cheers, Dan > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |