Hi Audrey, I did try your suggestion beforehand and it doesn't appear to fix the issue. [root@KVM04 ~]# cat /proc/sys/kernel/threads-max 2549635 [root@KVM04 ~]# echo 5549635 > /proc/sys/kernel/threads-max [root@KVM04 ~]# virsh start EX03 error: Failed to start domain EX03 error: internal error Process exited while reading console log output: char device redirected to /dev/pts/23 Thread::try_create(): pthread_create failed with error 11common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f5ec9706960 time 2013-01-17 16:46:50.935681 common/Thread.cc: 110: FAILED assert(ret == 0) ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7) 1: (()+0x2aaa8f) [0x7f5ec6a89a8f] 2: (SafeTimer::init()+0x95) [0x7f5ec6973575] 3: (librados::RadosClient::connect()+0x72c) [0x7f5ec69099dc] 4: (()+0xa0290) [0x7f5ec97c8290] 5: (()+0x879dd) [0x7f5ec97af9dd] 6: (()+0x87c1b) [0x7f5ec97afc1b] 7: (()+0x87ae1) [0x7f5ec97afae1] 8: (()+0x87d50) [0x7f5ec97afd50] 9: (()+0xb37b2) [0x7f5ec97db7b2] 10: (()+0x1e83eb) [0x7f5ec99103eb] 11: (()+0x1ab54a) [0x7f5ec98d354a] 12: (main()+0x9da) [0x7f5ec9913a3a] 13: (__libc_start_main()+0xfd) [0x7f5ec5755cdd] 14: (()+0x710b9) [0x7f5ec97990b9] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after -----Original Message----- From: Andrey Korolyov [mailto:andrey@xxxxxxx] Sent: Thursday, 17 January 2013 4:42 PM To: Matthew Anderson Cc: ceph-devel@xxxxxxxxxxxxxxx Subject: Re: Single host VM limit when using RBD Hi Matthew, Seems to a low value in /proc/sys/kernel/threads-max value. On Thu, Jan 17, 2013 at 12:37 PM, Matthew Anderson <matthewa@xxxxxxxxxxxx> wrote: > I've run into a limit on the maximum number of RBD backed VM's that I'm able to run on a single host. I have 20 VM's (21 RBD volumes open) running on a single host and when booting the 21st machine I get the below error from libvirt/QEMU. I'm able to shut down a VM and start another in it's place so there seems to be a hard limit on the amount of volumes I'm able to have open. I did some googling and the error 11 from pthread_create seems to mean 'resource unavailable' so I'm probably running into a thread limit of some sort. I did try increasing the max_thread kernel option but nothing changed. I moved a few VM's to a different empty host and they start with no issues at all. > > This machine has 4 OSD's running on it in addition to the 20 VM's. Kernel 3.7.1. Ceph 0.56.1 and QEMU 1.3.0. There is currently 65GB of 96GB free ram and no swap. > > Can anyone suggest where the limit might be or anything I can do to narrow down the problem? > > Thanks > -Matt > ------------------------- > > Error starting domain: internal error Process exited while reading > console log output: char device redirected to /dev/pts/23 > Thread::try_create(): pthread_create failed with error > 11common/Thread.cc: In function 'void Thread::create(size_t)' thread > 7f4eb5a65960 time 2013-01-17 02:32:58.096437 > common/Thread.cc: 110: FAILED assert(ret == 0) ceph version 0.56.1 > (e4a541624df62ef353e754391cbbb707f54b16f7) > 1: (()+0x2aaa8f) [0x7f4eb2de8a8f] > 2: (SafeTimer::init()+0x95) [0x7f4eb2cd2575] > 3: (librados::RadosClient::connect()+0x72c) [0x7f4eb2c689dc] > 4: (()+0xa0290) [0x7f4eb5b27290] > 5: (()+0x879dd) [0x7f4eb5b0e9dd] > 6: (()+0x87c1b) [0x7f4eb5b0ec1b] > 7: (()+0x87ae1) [0x7f4eb5b0eae1] > 8: (()+0x87d50) [0x7f4eb5b0ed50] > 9: (()+0xb37b2) [0x7f4eb5b3a7b2] > 10: (()+0x1e83eb) [0x7f4eb5c6f3eb] > 11: (()+0x1ab54a) [0x7f4eb5c3254a] > 12: (main()+0x9da) [0x7f4eb5c72a3a] > 13: (__libc_start_main()+0xfd) [0x7f4eb1ab4cdd] > 14: (()+0x710b9) [0x7f4eb5af80b9] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > terminate called after > > Traceback (most recent call last): > File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper > callback(asyncjob, *args, **kwargs) > File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb > callback(*args, **kwargs) > File "/usr/share/virt-manager/virtManager/domain.py", line 1090, in startup > self._backend.create() > File "/usr/lib/python2.7/dist-packages/libvirt.py", line 620, in create > if ret == -1: raise libvirtError ('virDomainCreate() failed', > dom=self) > libvirtError: internal error Process exited while reading console log > output: char device redirected to /dev/pts/23 > Thread::try_create(): pthread_create failed with error > 11common/Thread.cc: In function 'void Thread::create(size_t)' thread > 7f4eb5a65960 time 2013-01-17 02:32:58.096437 > common/Thread.cc: 110: FAILED assert(ret == 0) ceph version 0.56.1 > (e4a541624df62ef353e754391cbbb707f54b16f7) > 1: (()+0x2aaa8f) [0x7f4eb2de8a8f] > 2: (SafeTimer::init()+0x95) [0x7f4eb2cd2575] > 3: (librados::RadosClient::connect()+0x72c) [0x7f4eb2c689dc] > 4: (()+0xa0290) [0x7f4eb5b27290] > 5: (()+0x879dd) [0x7f4eb5b0e9dd] > 6: (()+0x87c1b) [0x7f4eb5b0ec1b] > 7: (()+0x87ae1) [0x7f4eb5b0eae1] > 8: (()+0x87d50) [0x7f4eb5b0ed50] > 9: (()+0xb37b2) [0x7f4eb5b3a7b2] > 10: (()+0x1e83eb) [0x7f4eb5c6f3eb] > 11: (()+0x1ab54a) [0x7f4eb5c3254a] > 12: (main()+0x9da) [0x7f4eb5c72a3a] > 13: (__libc_start_main()+0xfd) [0x7f4eb1ab4cdd] > 14: (()+0x710b9) [0x7f4eb5af80b9] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > terminate called after > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > info at http://vger.kernel.org/majordomo-info.html ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
