But in ceph_user,Mark, and some users are really discussing some supermicro chassis that can have 24 spindles per 2u or 36/48 spindles per 4U even with 20 osds per node,the thread num will more than 5000,and if take internal heartbeat/replication pipe into account, it should be around 10K threads. This is still too high for 8 core or 16 core cpu/cpus and will waste a lot of cycles in context switchinh. 发自我的 iPhone 在 2013-6-7，0:21，"Gregory Farnum" <greg@xxxxxxxxxxx> 写道： > On Thu, Jun 6, 2013 at 12:25 AM, Chen, Xiaoxi <xiaoxi.chen@xxxxxxxxx> wrote: >> >> Hi, >> From the code, each pipe (contains a TCP socket) will fork 2 threads, a reader and a writer. We really observe 100+ threads per OSD daemon with 30 instances of rados bench as clients. >> But this number seems a bit crazy, if I have a 40 disks node, thus I will have 40 OSDs, we plan to have 6 such nodes to serve 120 VMs from OpenStack. Since a RBD is distributed across all the OSDs, we can expect, for every single OSD daemon, we will have 120 TCP socket, that means 240 threads. Thus for 40 OSDs per node, we will have 9600 threads per node. This thread number seems incredible. >> Is there any internal mechanism to track and manage the number of pipes ? and another question may be , why we need such a lot threads ? why not epoll? > Yep, right now the OSD maintains two threads per connection. That > hasn't been a problem so far (they're fairly cheap threads) and people > run into other limits much earlier ― 40 OSDs/node for instance would > require a lot of compute power anyway. > epoll is a good idea and is something we're aware of, but it hasn't > been necessary yet and would involve mucking around with some fairly > sensitive core code so it hasn't risen to the top of anybody's queue. > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |