Hi, One of the osd in my cluster downs w no reason, I saw the error message in the log below, I restarted osd, but after several hours, the problem come back again. Could you help? "Too many open files not handled on operation 24 (541468.0.1, or op 1, counting from 0) -96> 2014-05-14 22:12:24.281185 7f617b33e700 5 -- op tracker -- , seq: 788808, time: 2014-05-14 22:12:24.281164, event: reached_pg, request: osd_op(client.21276.0:3884815 rb.0.31c7.238e1f 29.000000003c15 [write 2273280~65536] 4.110fcf4 e12271) v4 -95> 2014-05-14 22:12:24.281192 7f618556d700 0 filestore(/var/lib/ceph/osd/ceph-3) unexpected error code -94> 2014-05-14 22:12:24.281197 7f6181b4b700 5 -- op tracker -- , seq: 788843, time: 2014-05-14 22:12:24.281011, event: header_read, request: osd_op(client.21276.0:3884929 rb.0.31c7.238e1 f29.000000005614 [write 3137536~65536] 4.63e147e e12271) v4 > 2014-05-14 22:12:24.289987 7f6185d6e700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thre ad 7f6185d6e700 time 2014-05-14 22:12:24.282488 os/FileStore.cc: 2448: FAILED assert(0 == "unexpected error") ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x11c3) [0x723a43] 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x74) [0x72a4d4] 3: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x29a) [0x72a78a] 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x551) [0x988f21] 5: (ThreadPool::WorkThread::entry()+0x10) [0x98bf50] 6: /lib64/libpthread.so.0() [0x3a7ce079d1] 7: (clone()+0x6d) [0x3a7cae8b6d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this."........ #iostat avg-cpu: %user %nice %system %iowait %steal %idle 0.44 0.00 0.14 0.41 0.00 99.01 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdb 1.23 0.10 35.72 12738 4762008 sdc 5.25 214.25 1288.81 28564314 171824232 sdd 4.16 139.98 1021.69 18662490 136211888 sde 4.61 207.50 1039.20 27663258 138545960 sdf 7.94 203.24 2530.63 27095930 337383704 sdg 4.77 0.57 1459.29 75330 194553064 sdh 4.38 0.37 1287.42 48954 171638304 sdi 85.80 132.13 8157.53 17616004 1087562272 sdj 8.77 10.99 1701.90 1465844 226897024 sda 4.55 0.60 1331.50 80010 177516216 osd log attached. Wei Cao (Buddy) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140515/6e417eb6/attachment-0001.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: ceph-osd.21_short.log Type: application/octet-stream Size: 3928021 bytes Desc: ceph-osd.21_short.log URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140515/6e417eb6/attachment-0001.obj>