Which Ceph version are you using?
On Sat, Feb 11, 2017 at 5:02 PM, Chenyehua <chen.yehua@xxxxxxx> wrote:
> Dear Mr Kefu Chai
>
> Sorry to disturb you.
>
> I meet a problem recently. In my ceph cluster ,health status has warning
> “store is getting too big!” for several days; and ceph-mon costs nearly
> 100% cpu;
>
> Have you ever met this situation?
>
> Some detailed information are attached below:
>
>
>
> root@cvknode17:~# ceph -s
>
> cluster 04afba60-3a77-496c-b616-2ecb5e47e141
>
> health HEALTH_WARN
>
> mon.cvknode17 store is getting too big! 34104 MB >= 15360 MB
>
> monmap e1: 3 mons at
> {cvknode15=172.16.51.15:6789/0,cvknode16=172.16.51.16:6789/0,cvknode17=172.16.51.17:6789/0}
>
> election epoch 862, quorum 0,1,2 cvknode15,cvknode16,cvknode17
>
> osdmap e196279: 347 osds: 347 up, 347 in
>
> pgmap v5891025: 33272 pgs, 16 pools, 26944 GB data, 6822 kobjects
>
> 65966 GB used, 579 TB / 644 TB avail
>
> 33270 active+clean
>
> 2 active+clean+scrubbing+deep
>
> client io 840 kB/s rd, 739 kB/s wr, 35 op/s rd, 184 op/s wr
>
>
>
> root@cvknode17:~# top
>
> top - 15:19:28 up 23 days, 23:58, 6 users, load average: 1.08, 1.40, 1.77
>
> Tasks: 346 total, 2 running, 342 sleeping, 0 stopped, 2 zombie
>
> Cpu(s): 8.1%us, 10.8%sy, 0.0%ni, 69.0%id, 9.5%wa, 0.0%hi, 2.5%si,
> 0.0%st
>
> Mem: 65384424k total, 58102880k used, 7281544k free, 240720k buffers
>
> Swap: 29999100k total, 344944k used, 29654156k free, 24274272k cached
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 24407 root 20 0 17.3g 12g 10m S 98 20.2 8420:11 ceph-mon
>
>
>
> root@cvknode17:~# top -Hp 24407
>
> top - 15:19:49 up 23 days, 23:59, 6 users, load average: 1.12, 1.39, 1.76
>
> Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie
>
> Cpu(s): 8.1%us, 10.8%sy, 0.0%ni, 69.0%id, 9.5%wa, 0.0%hi, 2.5%si,
> 0.0%st
>
> Mem: 65384424k total, 58104868k used, 7279556k free, 240744k buffers
>
> Swap: 29999100k total, 344944k used, 29654156k free, 24271188k cached
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 25931 root 20 0 17.3g 12g 9m R 98 20.2 7957:37 ceph-mon
>
> 24514 root 20 0 17.3g 12g 9m S 2 20.2 3:06.75 ceph-mon
>
> 25932 root 20 0 17.3g 12g 9m S 2 20.2 1:07.82 ceph-mon
>
> 24407 root 20 0 17.3g 12g 9m S 0 20.2 0:00.67 ceph-mon
>
> 24508 root 20 0 17.3g 12g 9m S 0 20.2 15:50.24 ceph-mon
>
> 24513 root 20 0 17.3g 12g 9m S 0 20.2 0:07.88 ceph-mon
>
> 24534 root 20 0 17.3g 12g 9m S 0 20.2 196:33.85 ceph-mon
>
> 24535 root 20 0 17.3g 12g 9m S 0 20.2 0:00.01 ceph-mon
>
> 25929 root 20 0 17.3g 12g 9m S 0 20.2 3:06.09 ceph-mon
>
> 25930 root 20 0 17.3g 12g 9m S 0 20.2 8:12.58 ceph-mon
>
> 25933 root 20 0 17.3g 12g 9m S 0 20.2 4:42.22 ceph-mon
>
> 25934 root 20 0 17.3g 12g 9m S 0 20.2 40:53.27 ceph-mon
>
> 25935 root 20 0 17.3g 12g 9m S 0 20.2 0:04.84 ceph-mon
>
> 25936 root 20 0 17.3g 12g 9m S 0 20.2 0:00.01 ceph-mon
>
> 25980 root 20 0 17.3g 12g 9m S 0 20.2 0:06.65 ceph-mon
>
> 25986 root 20 0 17.3g 12g 9m S 0 20.2 48:26.77 ceph-mon
>
> 55738 root 20 0 17.3g 12g 9m S 0 20.2 0:09.06 ceph-mon
>
>
>
>
>
> Thread 20 (Thread 0x7f3e77e80700 (LWP 25931)):
>
> #0 0x00007f3e7e83a653 in pread64 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
>
> #1 0x00000000009286cf in ?? ()
>
> #2 0x000000000092c187 in leveldb::ReadBlock(leveldb::RandomAccessFile*,
> leveldb::ReadOptions const&, leveldb::BlockHandle const&, leveldb::Block**)
> ()
>
> #3 0x0000000000922f41 in leveldb::Table::BlockReader(void*,
> leveldb::ReadOptions const&, leveldb::Slice const&) ()
>
> #4 0x0000000000924840 in ?? ()
>
> #5 0x0000000000924b39 in ?? ()
>
> #6 0x0000000000924a7a in ?? ()
>
> #7 0x00000000009227d0 in ?? ()
>
> #8 0x00000000009140b6 in ?? ()
>
> #9 0x00000000009143dd in ?? ()
>
> #10 0x000000000088d399 in
> LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&,
> std::string const&) ()
>
> #11 0x000000000088bf00 in LevelDBStore::get(std::string const&,
> std::set<std::string, std::less<std::string>, std::allocator<std::string> >
> const&, std::map<std::string, ceph::buffer::list, std::less<std::string>,
> std::allocator<std::pair<std::string const, ceph::buffer::list> > >*) ()
>
> #12 0x000000000056a7a2 in MonitorDBStore::get(std::string const&,
> std::string const&) ()
>
> ---Type <return> to continue, or q <return> to quit---
>
> #13 0x00000000005dcf61 in PaxosService::refresh(bool*) ()
>
> #14 0x000000000058a76b in Monitor::refresh_from_paxos(bool*) ()
>
> #15 0x00000000005c55ac in Paxos::do_refresh() ()
>
> #16 0x00000000005cc093 in Paxos::handle_commit(MMonPaxos*) ()
>
> #17 0x00000000005d4d8b in Paxos::dispatch(PaxosServiceMessage*) ()
>
> #18 0x00000000005ac204 in Monitor::dispatch(MonSession*, Message*, bool) ()
>
> #19 0x00000000005a9b09 in Monitor::_ms_dispatch(Message*) ()
>
> #20 0x00000000005c48a2 in Monitor::ms_dispatch(Message*) ()
>
> #21 0x00000000008b2e67 in Messenger::ms_deliver_dispatch(Message*) ()
>
> #22 0x00000000008b000a in DispatchQueue::entry() ()
>
> #23 0x00000000007a069d in DispatchQueue::DispatchThread::entry() ()
>
> #24 0x00007f3e7e832e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
>
> #25 0x00007f3e7cff638d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> #26 0x0000000000000000 in ?? ()
>
>
>
>
>
> Thanks
>
> Best regards
>
>
>
> -------------------------------------------------------------------------------------------------------------------------------------
> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
> 邮件！
> This e-mail and its attachments contain confidential information from H3C,
> which is
> intended only for the person or entity whose address is listed above. Any
> use of the
> information contained herein in any way (including, but not limited to,
> total or partial
> disclosure, reproduction, or dissemination) by persons other than the
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, please
> notify the sender
> by phone or email immediately and delete it!
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
