I have not configured anything for the msd except this [mds] # 100k+ files in 2 folders mds bal fragment size max = 120000 # maybe for nfs-ganesha problems? # http://docs.ceph.com/docs/master/cephfs/eviction/ mds_session_blacklist_on_timeout = false mds_session_blacklist_on_evict = false mds_cache_memory_limit = 8000000000 # faster fail over? #mds beacon grace = 5 -----Original Message----- From: Eugen Block [mailto:eblock@xxxxxx] Sent: dinsdag 21 mei 2019 10:18 To: ceph-users@xxxxxxxxxxxxxx Subject: Re: Is a not active mds doing something? Hi Marc, have you configured the other MDS to be standby-replay for the active MDS? I have three MDS servers, one is active, the second is active-standby and the third just standby. If the active fails, the second takes over within seconds. This is what I have in my ceph.conf: [mds.<YOUR MDS>] mds_standby_replay = true mds_standby_for_rank = 0 Regards, Eugen Zitat von Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx>: > Should a not active mds be doing something??? When I restarted the not > active mds.c, My client io on the fs_data pool disappeared. > > > services: > mon: 3 daemons, quorum a,b,c > mgr: c(active), standbys: a, b > mds: cephfs-1/1/1 up {0=a=up:active}, 1 up:standby > osd: 32 osds: 32 up, 32 in > rgw: 2 daemons active > > > > -----Original Message----- > From: Marc Roos > Sent: dinsdag 21 mei 2019 10:01 > To: ceph-users@xxxxxxxxxxxxxx; Marc Roos > Subject: RE: cephfs causing high load on vm, taking down > 15 min later another cephfs vm > > > I have evicted all client connections and have still high load on > osd's > > And ceph osd pool stats shows still client activity????? > > pool fs_data id 20 > client io 565KiB/s rd, 120op/s rd, 0op/s wr > > > > > -----Original Message----- > From: Marc Roos > Sent: dinsdag 21 mei 2019 9:51 > To: ceph-users@xxxxxxxxxxxxxx; Marc Roos > Subject: RE: cephfs causing high load on vm, taking down > 15 min later another cephfs vm > > > I have got this today again????? I cannot unmount the filesystem and > looks like some osd's are having 100% cpu utilization? > > > -----Original Message----- > From: Marc Roos > Sent: maandag 20 mei 2019 12:42 > To: ceph-users > Subject: cephfs causing high load on vm, taking down 15 > min later another cephfs vm > > > > I got my first problem with cephfs in a production environment. Is it > possible from these logfiles to deduct what happened? > > svr1 is connected to ceph client network via switch > svr2 vm is collocated on c01 node. > c01 has osd's and the mon.a colocated. > > svr1 was the first to report errors at 03:38:44. I have no error > messages reported of a network connection problem by any of the ceph > nodes. I have nothing in dmesg on c01. > > [@c01 ~]# cat /etc/redhat-release > CentOS Linux release 7.6.1810 (Core) > [@c01 ~]# uname -a > Linux c01 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC > 2019 > > x86_64 x86_64 x86_64 GNU/Linux > [@c01 ~]# ceph versions > { > "mon": { > "ceph version 12.2.12 > (1436006594665279fe734b4c15d7e08c13ebd777) > > luminous (stable)": 3 > }, > "mgr": { > "ceph version 12.2.12 > (1436006594665279fe734b4c15d7e08c13ebd777) > > luminous (stable)": 3 > }, > "osd": { > "ceph version 12.2.12 > (1436006594665279fe734b4c15d7e08c13ebd777) > > luminous (stable)": 32 > }, > "mds": { > "ceph version 12.2.12 > (1436006594665279fe734b4c15d7e08c13ebd777) > > luminous (stable)": 2 > }, > "rgw": { > "ceph version 12.2.12 > (1436006594665279fe734b4c15d7e08c13ebd777) > > luminous (stable)": 2 > }, > "overall": { > "ceph version 12.2.12 > (1436006594665279fe734b4c15d7e08c13ebd777) > > luminous (stable)": 42 > } > } > > > > > [0] svr1 messages > May 20 03:36:01 svr1 systemd: Started Session 308978 of user root. > May 20 03:36:01 svr1 systemd: Started Session 308979 of user root. > May 20 03:36:01 svr1 systemd: Started Session 308979 of user root. > May 20 03:36:01 svr1 systemd: Started Session 308980 of user root. > May 20 03:36:01 svr1 systemd: Started Session 308980 of user root. > May 20 03:38:01 svr1 systemd: Started Session 308981 of user root. > May 20 03:38:01 svr1 systemd: Started Session 308981 of user root. > May 20 03:38:01 svr1 systemd: Started Session 308982 of user root. > May 20 03:38:01 svr1 systemd: Started Session 308982 of user root. > May 20 03:38:01 svr1 systemd: Started Session 308983 of user root. > May 20 03:38:01 svr1 systemd: Started Session 308983 of user root. > May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error > May 20 03:38:44 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error > May 20 03:38:45 svr1 kernel: last message repeated 5 times May 20 > 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error May 20 > 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session lost, > hunting for new mon May 20 03:38:45 svr1 kernel: last message repeated > 5 times May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 > io error May 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 > session lost, hunting for new mon May 20 03:38:45 svr1 kernel: > libceph: mon1 192.168.x.112:6789 session established May 20 03:38:45 > svr1 kernel: libceph: mon1 192.168.x.112:6789 session established May > 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May > 20 03:38:45 svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May > 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 io error May > 20 03:38:45 svr1 kernel: libceph: mon1 192.168.x.112:6789 session > lost, hunting for new mon May 20 03:38:45 svr1 kernel: libceph: mon1 > 192.168.x.112:6789 io error May 20 03:38:45 svr1 kernel: libceph: mon1 > 192.168.x.112:6789 session lost, hunting for new mon May 20 03:38:45 > svr1 kernel: libceph: mon0 192.168.x.111:6789 session established May > 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session > established May 20 03:38:45 svr1 kernel: libceph: mon0 > 192.168.x.111:6789 io error May 20 03:38:45 svr1 kernel: libceph: mon0 > 192.168.x.111:6789 session lost, hunting for new mon May 20 03:38:45 > svr1 kernel: libceph: mon0 192.168.x.111:6789 io error May 20 03:38:45 > svr1 kernel: libceph: mon0 192.168.x.111:6789 session lost, hunting > for new mon May 20 03:38:45 svr1 kernel: libceph: mon2 > 192.168.x.113:6789 session established May 20 03:38:45 svr1 kernel: > libceph: mon2 192.168.x.113:6789 session established May 20 03:38:45 > svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May 20 03:38:45 > svr1 kernel: libceph: osd0 192.168.x.111:6814 io error May 20 03:38:45 > svr1 kernel: libceph: mon2 192.168.x.113:6789 io error May 20 03:38:45 > svr1 kernel: libceph: mon2 192.168.x.113:6789 session lost, hunting > for new mon May 20 03:38:45 svr1 kernel: libceph: mon2 > 192.168.x.113:6789 io error May 20 03:38:45 svr1 kernel: libceph: mon2 > 192.168.x.113:6789 session lost, hunting for new mon May 20 03:38:45 > svr1 kernel: libceph: mon0 192.168.x.111:6789 session established May > 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 io error May > 20 03:38:45 svr1 kernel: libceph: mon0 192.168.x.111:6789 session > lost, hunting for new mon May 20 03:38:45 svr1 kernel: libceph: mon0 > 192.168.x.111:6789 session established > > > [1] svr2 messages > May 20 03:40:01 svr2 systemd: Stopping User Slice of root. > May 20 03:40:01 svr2 systemd: Removed slice User Slice of root. > May 20 03:40:01 svr2 systemd: Stopping User Slice of root. > May 20 03:50:01 svr2 systemd: Created slice User Slice of root. > May 20 03:50:01 svr2 systemd: Created slice User Slice of root. > May 20 03:50:01 svr2 systemd: Starting User Slice of root. > May 20 03:50:01 svr2 systemd: Starting User Slice of root. > May 20 03:50:01 svr2 systemd: Started Session 9442 of user root. > May 20 03:50:01 svr2 systemd: Started Session 9442 of user root. > May 20 03:50:01 svr2 systemd: Starting Session 9442 of user root. > May 20 03:50:01 svr2 systemd: Starting Session 9442 of user root. > May 20 03:50:01 svr2 systemd: Removed slice User Slice of root. > May 20 03:50:01 svr2 systemd: Removed slice User Slice of root. > May 20 03:50:01 svr2 systemd: Stopping User Slice of root. > May 20 03:50:01 svr2 systemd: Stopping User Slice of root. > May 20 03:53:37 svr2 kernel: libceph: osd9 192.168.x.112:6812 io error > May 20 03:53:37 svr2 kernel: libceph: osd9 192.168.x.112:6812 io error > May 20 03:53:37 svr2 kernel: last message repeated 3 times May 20 > 03:53:37 svr2 kernel: libceph: mon1 192.168.x.112:6789 io error May 20 > 03:53:37 svr2 kernel: last message repeated 3 times May 20 03:53:37 > svr2 kernel: libceph: mon1 192.168.x.112:6789 io error May 20 03:53:37 > svr2 kernel: libceph: mon1 192.168.x.112:6789 session lost, hunting > for new mon May 20 03:53:37 svr2 kernel: libceph: mon1 > 192.168.x.112:6789 session lost, hunting for new mon May 20 03:53:37 > svr2 kernel: libceph: mon2 192.168.x.113:6789 session established May > 20 03:53:37 svr2 kernel: libceph: mon2 192.168.x.113:6789 session > established May 20 03:53:37 svr2 kernel: libceph: mon2 > 192.168.x.113:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon2 > 192.168.x.113:6789 session lost, hunting for new mon May 20 03:53:37 > svr2 kernel: libceph: mon2 192.168.x.113:6789 io error May 20 03:53:37 > svr2 kernel: libceph: mon2 192.168.x.113:6789 session lost, hunting > for new mon May 20 03:53:37 svr2 kernel: libceph: osd9 > 192.168.x.112:6812 io error May 20 03:53:37 svr2 kernel: libceph: osd9 > 192.168.x.112:6812 io error May 20 03:53:37 svr2 kernel: libceph: mon1 > 192.168.x.112:6789 session established May 20 03:53:37 svr2 kernel: > libceph: mon1 192.168.x.112:6789 session established May 20 03:53:37 > svr2 kernel: libceph: mon1 192.168.x.112:6789 io error May 20 03:53:37 > svr2 kernel: libceph: mon1 192.168.x.112:6789 session lost, hunting > for new mon May 20 03:53:37 svr2 kernel: libceph: mon1 > 192.168.x.112:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon1 > 192.168.x.112:6789 session lost, hunting for new mon May 20 03:53:37 > svr2 kernel: libceph: mon0 192.168.x.111:6789 session established May > 20 03:53:37 svr2 kernel: libceph: mon0 192.168.x.111:6789 session > established May 20 03:53:37 svr2 kernel: libceph: mon0 > 192.168.x.111:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon0 > 192.168.x.111:6789 session lost, hunting for new mon May 20 03:53:37 > svr2 kernel: libceph: mon0 192.168.x.111:6789 io error May 20 03:53:37 > svr2 kernel: libceph: mon0 192.168.x.111:6789 session lost, hunting > for new mon May 20 03:53:37 svr2 kernel: libceph: mon2 > 192.168.x.113:6789 session established May 20 03:53:37 svr2 kernel: > libceph: mon2 192.168.x.113:6789 session established May 20 03:53:37 > svr2 kernel: libceph: mon2 192.168.x.113:6789 io error May 20 03:53:37 > svr2 kernel: libceph: mon2 192.168.x.113:6789 session lost, hunting > for new mon May 20 03:53:37 svr2 kernel: libceph: mon2 > 192.168.x.113:6789 io error May 20 03:53:37 svr2 kernel: libceph: mon2 > 192.168.x.113:6789 session lost, hunting for new mon > > [2] osd.0 log > 2019-05-20 03:38:46.358270 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55780c19e000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:38:56.155141 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55776afb6000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:38:56.476312 7f9208e6a700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557797300800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:39:35.050674 7f9208e6a700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55784c099000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:41:46.605523 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55778dba8000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:42:05.201417 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.112:0/2834749548 conn(0x5578179cf800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:42:18.275703 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557773ccf800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:42:18.493838 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557898a90000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:42:18.728962 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55776afba800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:42:19.242145 7f9208e6a700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557762c8f000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:43:41.492125 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x5577aa28d800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:45:40.006405 7f9208e6a700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557778d1d800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:45:40.736819 7f9208e6a700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x5577a4224800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:47:08.368138 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557778d1d800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:48:24.848331 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x5577f0819800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:50:44.442386 7f9208e6a700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55777ef3c000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:51:11.352119 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55779e445000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:51:18.615690 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55779e445000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:51:57.887069 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55777ef3c000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:51:58.109173 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557769206000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:51:58.364811 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x5577aa28d800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:51:59.323286 7f9208e6a700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x557773cce000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:57:22.060831 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55786cf37800 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 03:57:48.793125 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.114:0/4006640003 conn(0x5577972ff000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 04:07:07.135252 7f9207e68700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55779b319000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) 2019-05-20 04:07:07.394734 7f9208669700 0 -- > 192.168.x.111:6814/3478915 > >>> 192.168.x.43:0/1827964483 conn(0x55779529e000 :6814 > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg accept replacing existing (lossy) channel (new > one lossy=1) > > > [3] mon.a log > 2019-05-20 03:36:12.418662 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316172418658, "job": 57455, "event": > "table_file_deletion", "file_number": 1788761} 2019-05-20 > 03:36:12.431694 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316172431690, "job": 57455, "event": > "table_file_deletion", "file_number": 1788760} 2019-05-20 > 03:36:12.444811 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316172444808, "job": 57455, "event": > "table_file_deletion", "file_number": 1788759} 2019-05-20 > 03:36:12.458658 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316172458654, "job": 57455, "event": > "table_file_deletion", "file_number": 1788758} 2019-05-20 > 03:36:12.472801 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316172472797, "job": 57455, "event": > "table_file_deletion", "file_number": 1788757} 2019-05-20 > 03:36:12.487007 7f1b08e2b700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316172486995, "job": 57455, "event": > "table_file_deletion", "file_number": 1788756} 2019-05-20 > 03:36:12.487096 7f1b08e2b700 4 rocksdb: (Original Log Time > 2019/05/20-03:36:12.487089) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f > lu > sh.cc:1456] Compaction nothing to do > 2019-05-20 03:36:15.272950 7f1b0a62e700 1 mon.a@0(leader).osd e64430 > e64430: 32 total, 32 up, 32 in > 2019-05-20 03:36:15.298861 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64430: 32 total, 32 up, 32 in 2019-05-20 > 03:36:17.013839 7f1b1163c700 0 > mon.a@0(leader).data_health(7108) update_stats avail 91% total > 50.0GiB, used 4.05GiB, avail 45.9GiB 2019-05-20 03:36:19.738495 > 7f1b0a62e700 1 mon.a@0(leader).osd e64431 > e64431: 32 total, 32 up, 32 in > 2019-05-20 03:36:19.765029 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64431: 32 total, 32 up, 32 in 2019-05-20 > 03:36:25.302195 7f1b0a62e700 1 mon.a@0(leader).osd e64432 > e64432: 32 total, 32 up, 32 in > 2019-05-20 03:36:25.328639 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64432: 32 total, 32 up, 32 in 2019-05-20 > 03:36:29.741140 7f1b0a62e700 1 mon.a@0(leader).osd e64433 > e64433: 32 total, 32 up, 32 in > 2019-05-20 03:36:29.768034 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64433: 32 total, 32 up, 32 in 2019-05-20 > 03:36:35.343530 7f1b0a62e700 1 mon.a@0(leader).osd e64434 > e64434: 32 total, 32 up, 32 in > 2019-05-20 03:36:35.370266 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64434: 32 total, 32 up, 32 in 2019-05-20 > 03:36:39.738502 7f1b0a62e700 1 mon.a@0(leader).osd e64435 > e64435: 32 total, 32 up, 32 in > 2019-05-20 03:36:39.765465 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64435: 32 total, 32 up, 32 in 2019-05-20 > 03:36:45.371359 7f1b0a62e700 1 mon.a@0(leader).osd e64436 > e64436: 32 total, 32 up, 32 in > 2019-05-20 03:36:45.398034 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64436: 32 total, 32 up, 32 in 2019-05-20 > 03:36:49.738038 7f1b0a62e700 1 mon.a@0(leader).osd e64437 > e64437: 32 total, 32 up, 32 in > 2019-05-20 03:36:49.765428 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64437: 32 total, 32 up, 32 in 2019-05-20 > 03:36:55.418010 7f1b0a62e700 1 mon.a@0(leader).osd e64438 > e64438: 32 total, 32 up, 32 in > 2019-05-20 03:36:55.446785 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64438: 32 total, 32 up, 32 in 2019-05-20 > 03:36:59.738000 7f1b0a62e700 1 mon.a@0(leader).osd e64439 > e64439: 32 total, 32 up, 32 in > 2019-05-20 03:36:59.765632 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64439: 32 total, 32 up, 32 in 2019-05-20 > 03:37:05.423528 7f1b0a62e700 1 mon.a@0(leader).osd e64440 > e64440: 32 total, 32 up, 32 in > 2019-05-20 03:37:05.449968 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64440: 32 total, 32 up, 32 in 2019-05-20 > 03:37:06.513317 7f1b0a62e700 1 mon.a@0(leader).osd e64441 > e64441: 32 total, 32 up, 32 in > 2019-05-20 03:37:06.539643 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64441: 32 total, 32 up, 32 in 2019-05-20 > 03:37:09.851351 7f1b0a62e700 1 mon.a@0(leader).osd e64442 > e64442: 32 total, 32 up, 32 in > 2019-05-20 03:37:09.877349 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64442: 32 total, 32 up, 32 in 2019-05-20 > 03:37:15.446739 7f1b0a62e700 1 mon.a@0(leader).osd e64443 > e64443: 32 total, 32 up, 32 in > 2019-05-20 03:37:15.447538 7f1b0a62e700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_write.cc:725 > ] [default] New memtable created with log file: #1788785. Immutable > memtables: 0. > 2019-05-20 03:37:15.470599 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:37:15.447607) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f > lu sh.cc:1158] Calling FlushMemTableToOutputFile with column family > [default], flush slots available 1, compaction slots allowed 1, > compaction slots scheduled 1 2019-05-20 03:37:15.470613 7f1b0962c700 > 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:264] > [default] [JOB 57457] Flushing memtable with next logfile: 1788785 > 2019-05-20 03:37:15.470629 7f1b0962c700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316235470621, "job": 57457, "event": > "flush_started", "num_memtables": 1, "num_entries": 550, "num_deletes": > 0, "memory_usage": 32932624} > 2019-05-20 03:37:15.470632 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:293] > [default] [JOB 57457] Level-0 flush table #1788786: started 2019-05-20 > 03:37:15.474323 7f1b0a62e700 0 log_channel(cluster) log [DBG] : > osdmap e64443: 32 total, 32 up, 32 in 2019-05-20 03:37:15.622750 > 7f1b0962c700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316235622730, "cf_name": "default", "job": 57457, > "event": "table_file_creation", "file_number": 1788786, "file_size": > 29021225, "table_properties": {"data_size": 28956685, "index_size": > 5597, "filter_size": 58018, "raw_key_size": 4239, > "raw_average_key_size": 22, "raw_value_size": 28949846, > "raw_average_value_size": 155644, "num_data_blocks": 158, "num_entries": > > 186, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": > > "0", "kMergeOperands": "0"}} > 2019-05-20 03:37:15.622780 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:319] > [default] [JOB 57457] Level-0 flush table #1788786: 29021225 bytes OK > 2019-05-20 03:37:15.645271 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:37:15.622795) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:360 > ] [default] Level-0 commit table #1788786 started 2019-05-20 > 03:37:15.645286 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:37:15.645176) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:383 > ] [default] Level-0 commit table #1788786: memtable #1 done 2019-05-20 > 03:37:15.645297 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:37:15.645212) EVENT_LOG_v1 {"time_micros": > 1558316235645199, "job": 57457, "event": "flush_finished", "lsm_state": > [1, 0, 0, 0, 0, 0, 19], "immutable_memtables": 0} 2019-05-20 > 03:37:15.645302 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:37:15.645244) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f > lu sh.cc:132] [default] Level summary: base level 5 max bytes base > 69194285 > > files[1 0 0 0 0 0 19] max score 0.25 > 2019-05-20 03:37:15.645318 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_files.cc:388 > ] [JOB 57457] Try to delete WAL files size 28975051, prev total WAL > file size 28975092, number of live WAL files 2. > 2019-05-20 03:37:16.596003 7f1b0a62e700 1 mon.a@0(leader).osd e64444 > e64444: 32 total, 32 up, 32 in > 2019-05-20 03:37:16.621741 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64444: 32 total, 32 up, 32 in 2019-05-20 > 03:37:17.014200 7f1b1163c700 0 > mon.a@0(leader).data_health(7108) update_stats avail 91% total > 50.0GiB, used 4.08GiB, avail 45.9GiB 2019-05-20 03:37:19.822194 > 7f1b0a62e700 1 mon.a@0(leader).osd e64445 > e64445: 32 total, 32 up, 32 in > 2019-05-20 03:37:19.848514 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64445: 32 total, 32 up, 32 in 2019-05-20 > 03:37:25.473280 7f1b0a62e700 1 mon.a@0(leader).osd e64446 > e64446: 32 total, 32 up, 32 in > 2019-05-20 03:37:25.499555 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64446: 32 total, 32 up, 32 in 2019-05-20 > 03:37:26.630462 7f1b0a62e700 1 mon.a@0(leader).osd e64447 > e64447: 32 total, 32 up, 32 in > 2019-05-20 03:37:26.656497 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64447: 32 total, 32 up, 32 in 2019-05-20 > 03:37:29.856077 7f1b0a62e700 1 mon.a@0(leader).osd e64448 > e64448: 32 total, 32 up, 32 in > 2019-05-20 03:37:29.882033 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64448: 32 total, 32 up, 32 in 2019-05-20 > 03:37:35.503666 7f1b0a62e700 1 mon.a@0(leader).osd e64449 > e64449: 32 total, 32 up, 32 in > 2019-05-20 03:37:35.535606 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64449: 32 total, 32 up, 32 in 2019-05-20 > 03:37:36.662389 7f1b0a62e700 1 mon.a@0(leader).osd e64450 > e64450: 32 total, 32 up, 32 in > 2019-05-20 03:37:36.688901 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64450: 32 total, 32 up, 32 in 2019-05-20 > 03:37:39.889782 7f1b0a62e700 1 mon.a@0(leader).osd e64451 > e64451: 32 total, 32 up, 32 in > 2019-05-20 03:37:39.916350 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64451: 32 total, 32 up, 32 in 2019-05-20 > 03:37:45.534370 7f1b0a62e700 1 mon.a@0(leader).osd e64452 > e64452: 32 total, 32 up, 32 in > 2019-05-20 03:37:45.561000 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64452: 32 total, 32 up, 32 in 2019-05-20 > 03:37:49.755239 7f1b0a62e700 1 mon.a@0(leader).osd e64453 > e64453: 32 total, 32 up, 32 in > 2019-05-20 03:37:49.782078 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64453: 32 total, 32 up, 32 in 2019-05-20 > 03:37:50.904926 7f1b0a62e700 1 mon.a@0(leader).osd e64454 > e64454: 32 total, 32 up, 32 in > 2019-05-20 03:37:50.930940 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64454: 32 total, 32 up, 32 in 2019-05-20 > 03:37:55.567119 7f1b0a62e700 1 mon.a@0(leader).osd e64455 > e64455: 32 total, 32 up, 32 in > 2019-05-20 03:37:55.594617 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64455: 32 total, 32 up, 32 in 2019-05-20 > 03:37:59.736903 7f1b0a62e700 1 mon.a@0(leader).osd e64456 > e64456: 32 total, 32 up, 32 in > 2019-05-20 03:37:59.763752 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64456: 32 total, 32 up, 32 in 2019-05-20 > 03:38:05.584960 7f1b0a62e700 1 mon.a@0(leader).osd e64457 > e64457: 32 total, 32 up, 32 in > 2019-05-20 03:38:05.616558 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64457: 32 total, 32 up, 32 in 2019-05-20 > 03:38:09.736670 7f1b0a62e700 1 mon.a@0(leader).osd e64458 > e64458: 32 total, 32 up, 32 in > 2019-05-20 03:38:09.762625 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64458: 32 total, 32 up, 32 in 2019-05-20 > 03:38:15.617648 7f1b0a62e700 1 mon.a@0(leader).osd e64459 > e64459: 32 total, 32 up, 32 in > 2019-05-20 03:38:15.618422 7f1b0a62e700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_write.cc:725 > ] [default] New memtable created with log file: #1788787. Immutable > memtables: 0. > 2019-05-20 03:38:15.641025 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:38:15.618497) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f > lu sh.cc:1158] Calling FlushMemTableToOutputFile with column family > [default], flush slots available 1, compaction slots allowed 1, > compaction slots scheduled 1 2019-05-20 03:38:15.641035 7f1b0962c700 > 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:264] > [default] [JOB 57458] Flushing memtable with next logfile: 1788787 > 2019-05-20 03:38:15.641050 7f1b0962c700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316295641042, "job": 57458, "event": > "flush_started", "num_memtables": 1, "num_entries": 522, "num_deletes": > 0, "memory_usage": 32931384} > 2019-05-20 03:38:15.641054 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:293] > [default] [JOB 57458] Level-0 flush table #1788788: started 2019-05-20 > 03:38:15.644891 7f1b0a62e700 0 log_channel(cluster) log [DBG] : > osdmap e64459: 32 total, 32 up, 32 in 2019-05-20 03:38:15.786077 > 7f1b0962c700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316295786051, "cf_name": "default", "job": 57458, > "event": "table_file_creation", "file_number": 1788788, "file_size": > 28905705, "table_properties": {"data_size": 28841776, "index_size": > 5290, "filter_size": 57714, "raw_key_size": 4057, > "raw_average_key_size": 22, "raw_value_size": 28835274, > "raw_average_value_size": 161995, "num_data_blocks": 149, "num_entries": > > 178, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": > > "0", "kMergeOperands": "0"}} > 2019-05-20 03:38:15.786102 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:319] > [default] [JOB 57458] Level-0 flush table #1788788: 28905705 bytes OK > 2019-05-20 03:38:15.808704 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:38:15.786117) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:360 > ] [default] Level-0 commit table #1788788 started 2019-05-20 > 03:38:15.808721 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:38:15.808602) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:383 > ] [default] Level-0 commit table #1788788: memtable #1 done 2019-05-20 > 03:38:15.808726 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:38:15.808642) EVENT_LOG_v1 {"time_micros": > 1558316295808628, "job": 57458, "event": "flush_finished", "lsm_state": > [2, 0, 0, 0, 0, 0, 19], "immutable_memtables": 0} 2019-05-20 > 03:38:15.808861 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:38:15.808676) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f > lu sh.cc:132] [default] Level summary: base level 5 max bytes base > 69194285 > > files[2 0 0 0 0 0 19] max score 0.50 > 2019-05-20 03:38:15.808894 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_files.cc:388 > ] [JOB 57458] Try to delete WAL files size 28859118, prev total WAL > file size 28859159, number of live WAL files 2. > 2019-05-20 03:38:17.014616 7f1b1163c700 0 > mon.a@0(leader).data_health(7108) update_stats avail 91% total > 50.0GiB, used 4.11GiB, avail 45.9GiB 2019-05-20 03:38:19.736714 > 7f1b0a62e700 1 mon.a@0(leader).osd e64460 > e64460: 32 total, 32 up, 32 in > 2019-05-20 03:38:19.762553 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64460: 32 total, 32 up, 32 in 2019-05-20 > 03:38:25.640142 7f1b0a62e700 1 mon.a@0(leader).osd e64461 > e64461: 32 total, 32 up, 32 in > 2019-05-20 03:38:25.666382 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64461: 32 total, 32 up, 32 in 2019-05-20 > 03:38:26.728128 7f1b0a62e700 1 mon.a@0(leader).osd e64462 > e64462: 32 total, 32 up, 32 in > 2019-05-20 03:38:26.753754 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64462: 32 total, 32 up, 32 in 2019-05-20 > 03:38:27.930914 7f1b0a62e700 1 mon.a@0(leader).osd e64463 > e64463: 32 total, 32 up, 32 in > 2019-05-20 03:38:27.956420 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64463: 32 total, 32 up, 32 in 2019-05-20 > 03:38:30.271955 7f1b0a62e700 1 mon.a@0(leader).osd e64464 > e64464: 32 total, 32 up, 32 in > 2019-05-20 03:38:30.297712 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64464: 32 total, 32 up, 32 in 2019-05-20 > 03:38:35.711055 7f1b0a62e700 1 mon.a@0(leader).osd e64465 > e64465: 32 total, 32 up, 32 in > 2019-05-20 03:38:35.737482 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64465: 32 total, 32 up, 32 in 2019-05-20 > 03:38:36.863809 7f1b0a62e700 1 mon.a@0(leader).osd e64466 > e64466: 32 total, 32 up, 32 in > 2019-05-20 03:38:36.890569 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64466: 32 total, 32 up, 32 in 2019-05-20 > 03:38:40.090679 7f1b0a62e700 1 mon.a@0(leader).osd e64467 > e64467: 32 total, 32 up, 32 in > 2019-05-20 03:38:40.116896 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64467: 32 total, 32 up, 32 in 2019-05-20 > 03:38:45.739907 7f1b0a62e700 1 mon.a@0(leader).osd e64468 > e64468: 32 total, 32 up, 32 in > 2019-05-20 03:38:45.767072 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64468: 32 total, 32 up, 32 in 2019-05-20 > 03:38:46.890384 7f1b0a62e700 1 mon.a@0(leader).osd e64469 > e64469: 32 total, 32 up, 32 in > 2019-05-20 03:38:46.916813 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64469: 32 total, 32 up, 32 in 2019-05-20 > 03:38:50.117718 7f1b0a62e700 1 mon.a@0(leader).osd e64470 > e64470: 32 total, 32 up, 32 in > 2019-05-20 03:38:50.144237 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64470: 32 total, 32 up, 32 in 2019-05-20 > 03:38:55.729962 7f1b0a62e700 1 mon.a@0(leader).osd e64471 > e64471: 32 total, 32 up, 32 in > 2019-05-20 03:38:55.756512 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64471: 32 total, 32 up, 32 in 2019-05-20 > 03:38:56.882171 7f1b0a62e700 1 mon.a@0(leader).osd e64472 > e64472: 32 total, 32 up, 32 in > 2019-05-20 03:38:56.908626 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64472: 32 total, 32 up, 32 in 2019-05-20 > 03:39:00.138309 7f1b0a62e700 1 mon.a@0(leader).osd e64473 > e64473: 32 total, 32 up, 32 in > 2019-05-20 03:39:00.164934 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64473: 32 total, 32 up, 32 in 2019-05-20 > 03:39:04.761609 7f1b0a62e700 1 mon.a@0(leader).osd e64474 > e64474: 32 total, 32 up, 32 in > 2019-05-20 03:39:04.788323 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64474: 32 total, 32 up, 32 in 2019-05-20 > 03:39:09.769143 7f1b0a62e700 1 mon.a@0(leader).osd e64475 > e64475: 32 total, 32 up, 32 in > 2019-05-20 03:39:09.769992 7f1b0a62e700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_write.cc:725 > ] [default] New memtable created with log file: #1788789. Immutable > memtables: 0. > 2019-05-20 03:39:09.792308 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:39:09.770033) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f > lu sh.cc:1158] Calling FlushMemTableToOutputFile with column family > [default], flush slots available 1, compaction slots allowed 1, > compaction slots scheduled 1 2019-05-20 03:39:09.792329 7f1b0962c700 > 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:264] > [default] [JOB 57459] Flushing memtable with next log file: 1788789 > 2019-05-20 03:39:09.792421 7f1b0962c700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316349792351, "job": 57459, "event": > "flush_started", "num_memtables": 1, "num_entries": 501, "num_deletes": > 0, "memory_usage": 32873232} > 2019-05-20 03:39:09.792427 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:293] > [default] [JOB 57459] Level-0 flush table #1788790: started 2019-05-20 > 03:39:09.796258 7f1b0a62e700 0 log_channel(cluster) log [DBG] : > osdmap e64475: 32 total, 32 up, 32 in 2019-05-20 03:39:09.954310 > 7f1b0962c700 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1558316349954290, "cf_name": "default", "job": 57459, > "event": "table_file_creation", "file_number": 1788790, "file_size": > 28814681, "table_properties": {"data_size": 28751196, "index_size": > 5076, "filter_size": 57484, "raw_key_size": 3919, > "raw_average_key_size": 22, "raw_value_size": 28744932, > "raw_average_value_size": 167121, "num_data_blocks": 143, "num_entries": > > 172, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": > > "0", "kMergeOperands": "0"}} > 2019-05-20 03:39:09.954338 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/flush_job.cc:319] > [default] [JOB 57459] Level-0 flush table #1788790: 28814681 bytes OK > 2019-05-20 03:39:09.976850 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:39:09.954351) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:360 > ] [default] Level-0 commit table #1788790 started 2019-05-20 > 03:39:09.976866 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:39:09.976744) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/memtable_list.cc:383 > ] [default] Level-0 commit table #1788790: memtable #1 done 2019-05-20 > 03:39:09.976872 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:39:09.976783) EVENT_LOG_v1 {"time_micros": > 1558316349976770, "job": 57459, "event": "flush_finished", "lsm_state": > [3, 0, 0, 0, 0, 0, 19], "immutable_memtables": 0} 2019-05-20 > 03:39:09.976877 7f1b0962c700 4 rocksdb: (Original Log Time > 2019/05/20-03:39:09.976815) > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_f > lu sh.cc:132] [default] Level summary: base level 5 max bytes base > 69194285 > > files[3 0 0 0 0 0 19] max score 0.75 > 2019-05-20 03:39:09.976894 7f1b0962c700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ > AR > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/releas > e/ > 12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rocksdb/db/db_impl_files.cc:388 > ] [JOB 57459] Try to delete WAL files size 28767628, prev total WAL > file size 28767669, number of live WAL files 2. > 2019-05-20 03:39:10.921941 7f1b0a62e700 1 mon.a@0(leader).osd e64476 > e64476: 32 total, 32 up, 32 in > 2019-05-20 03:39:10.948342 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64476: 32 total, 32 up, 32 in 2019-05-20 > 03:39:14.782433 7f1b0a62e700 1 mon.a@0(leader).osd e64477 > e64477: 32 total, 32 up, 32 in > 2019-05-20 03:39:14.809139 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64477: 32 total, 32 up, 32 in 2019-05-20 > 03:39:17.014951 7f1b1163c700 0 > mon.a@0(leader).data_health(7108) update_stats avail 91% total > 50.0GiB, used 4.13GiB, avail 45.9GiB 2019-05-20 03:39:19.801742 > 7f1b0a62e700 1 mon.a@0(leader).osd e64478 > e64478: 32 total, 32 up, 32 in > 2019-05-20 03:39:19.828970 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64478: 32 total, 32 up, 32 in 2019-05-20 > 03:39:24.747871 7f1b0a62e700 1 mon.a@0(leader).osd e64479 > e64479: 32 total, 32 up, 32 in > 2019-05-20 03:39:24.775203 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64479: 32 total, 32 up, 32 in 2019-05-20 > 03:39:29.831760 7f1b0a62e700 1 mon.a@0(leader).osd e64480 > e64480: 32 total, 32 up, 32 in > 2019-05-20 03:39:29.859184 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64480: 32 total, 32 up, 32 in 2019-05-20 > 03:39:34.747989 7f1b0a62e700 1 mon.a@0(leader).osd e64481 > e64481: 32 total, 32 up, 32 in > 2019-05-20 03:39:34.775166 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64481: 32 total, 32 up, 32 in 2019-05-20 > 03:39:39.863809 7f1b0a62e700 1 mon.a@0(leader).osd e64482 > e64482: 32 total, 32 up, 32 in > 2019-05-20 03:39:39.891227 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64482: 32 total, 32 up, 32 in 2019-05-20 > 03:39:44.750833 7f1b0a62e700 1 mon.a@0(leader).osd e64483 > e64483: 32 total, 32 up, 32 in > 2019-05-20 03:39:44.778742 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64483: 32 total, 32 up, 32 in 2019-05-20 > 03:39:49.900624 7f1b0a62e700 1 mon.a@0(leader).osd e64484 > e64484: 32 total, 32 up, 32 in > 2019-05-20 03:39:49.928412 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64484: 32 total, 32 up, 32 in 2019-05-20 > 03:39:54.743844 7f1b0a62e700 1 mon.a@0(leader).osd e64485 > e64485: 32 total, 32 up, 32 in > 2019-05-20 03:39:54.773021 7f1b0a62e700 0 log_channel(cluster) log > [DBG] : osdmap e64485: 32 total, 32 up, 32 in 2019-05-20 > 03:39:59.937640 7f1b0a62e700 1 mon.a@0(leader).osd e64486 > e64486: 32 total, 32 up, 32 in > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |