Re: Unexplainable high memory usage OSD with BlueStore

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi Kenneth,
mimic 13.2.5 has previous version of bitmap allocator which isn't recommended to use. Please revert. 


New bitmap allocator is will be available since 13.2.6.


Thanks,

Igor

On 5/6/2019 4:19 PM, Kenneth Waegeman wrote:

Hi all,
I am also switching osds to the new bitmap allocater on 13.2.5. That went quite fluently for now, except for one OSD that keeps segfaulting when I enable the bitmap allocator. Each time I disable bitmap allocater on that again, osd is ok again. Segfault error of the OSD: 



--- begin dump of recent events ---
  -319> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command perfcounters_dump hook 0x55b2155b60d0   -318> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command 1 hook 0x55b2155b60d0   -317> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command perf dump hook 0x55b2155b60d0   -316> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command perfcounters_schema hook 0x55b2155b60d0   -315> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command perf histogram dump hook 0x55b2155b60d0   -314> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command 2 hook 0x55b2155b60d0   -313> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command perf schema hook 0x55b2155b60d0   -312> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command perf histogram schema hook 0x55b2155b60d0   -311> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command perf reset hook 0x55b2155b60d0   -310> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command config show hook 0x55b2155b60d0   -309> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command config help hook 0x55b2155b60d0   -308> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command config set hook 0x55b2155b60d0   -307> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command config unset hook 0x55b2155b60d0   -306> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command config get hook 0x55b2155b60d0   -305> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command config diff hook 0x55b2155b60d0   -304> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command config diff get hook 0x55b2155b60d0   -303> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command log flush hook 0x55b2155b60d0   -302> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command log dump hook 0x55b2155b60d0   -301> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command log reopen hook 0x55b2155b60d0   -300> 2019-05-06 15:11:45.172 7f28f4321d80  5 asok(0x55b2155de5a0) register_command dump_mempools hook 0x55b2155ec2c8   -299> 2019-05-06 15:11:45.182 7f28f4321d80 10 monclient: get_monmap_and_config   -298> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient: build_initial_monmap   -297> 2019-05-06 15:11:45.222 7f28e3a19700  2 Event(0x55b215911080 nevent=5000 time_id=1).set_owner idx=1 owner=139813594437376   -296> 2019-05-06 15:11:45.222 7f28e421a700  2 Event(0x55b215910c80 nevent=5000 time_id=1).set_owner idx=0 owner=139813602830080   -295> 2019-05-06 15:11:45.222 7f28e3218700  2 Event(0x55b215911880 nevent=5000 time_id=1).set_owner idx=2 owner=139813586044672 
  -294> 2019-05-06 15:11:45.222 7f28f4321d80  1  Processor -- start
  -293> 2019-05-06 15:11:45.222 7f28f4321d80  1 -- - start start
  -292> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient: init
  -291> 2019-05-06 15:11:45.222 7f28f4321d80  5 adding auth protocol: cephx   -290> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient: auth_supported 2 method cephx   -289> 2019-05-06 15:11:45.222 7f28f4321d80  2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-3/keyring   -288> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient: _reopen_session rank -1   -287> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient(hunting): picked mon.noname-c con 0x55b2159e2600 addr 10.141.16.3:6789/0   -286> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient(hunting): picked mon.noname-b con 0x55b2159e2c00 addr 10.141.16.2:6789/0   -285> 2019-05-06 15:11:45.222 7f28f4321d80  1 -- - --> 10.141.16.2:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55b2155b1200 con 0   -284> 2019-05-06 15:11:45.222 7f28f4321d80  1 -- - --> 10.141.16.3:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55b2155b1440 con 0   -283> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient(hunting): _renew_subs   -282> 2019-05-06 15:11:45.222 7f28f4321d80 10 monclient(hunting): authenticate will time out at 2019-05-06 15:16:45.237660   -281> 2019-05-06 15:11:45.222 7f28e3a19700  1 -- 10.141.16.3:0/3652030958 learned_addr learned my addr 10.141.16.3:0/3652030958   -280> 2019-05-06 15:11:45.222 7f28e3a19700  2 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0   -279> 2019-05-06 15:11:45.222 7f28e3218700  2 -- 10.141.16.3:0/3652030958 >> 10.141.16.2:6789/0 conn(0x55b2159e2c00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0   -278> 2019-05-06 15:11:45.222 7f28e3a19700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7774783 cs=1 l=1). rx mon.2 seq 1 0x55b2155b1440 mon_map magic: 0 v1   -277> 2019-05-06 15:11:45.222 7f28e3218700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.2:6789/0 conn(0x55b2159e2c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7963360 cs=1 l=1). rx mon.1 seq 1 0x55b2155b1200 mon_map magic: 0 v1   -276> 2019-05-06 15:11:45.222 7f28e3218700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.2:6789/0 conn(0x55b2159e2c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7963360 cs=1 l=1). rx mon.1 seq 2 0x55b2155b1d40 auth_reply(proto 2 0 (0) Success) v1   -275> 2019-05-06 15:11:45.222 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 <== mon.2 10.141.16.3:6789/0 1 ==== mon_map magic: 0 v1 ==== 397+0+0 (3021601431 0 0) 0x55b2155b1440 con 0x55b2159e2600   -274> 2019-05-06 15:11:45.222 7f28e3a19700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7774783 cs=1 l=1). rx mon.2 seq 2 0x55b2155b1b00 auth_reply(proto 2 0 (0) Success) v1   -273> 2019-05-06 15:11:45.222 7f28e2a17700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1   -272> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient(hunting):  got monmap 3, mon.noname-c is now rank -1   -271> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient(hunting): dump: 
epoch 3
fsid 92bfcf0a-1d39-43b3-b60f-44f01b630e47
last_changed 2018-08-06 08:11:26.469740
created 0.000000
0: 10.141.16.1:6789/0 mon.mds01
1: 10.141.16.2:6789/0 mon.mds02
2: 10.141.16.3:6789/0 mon.mds03
  -270> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 <== mon.1 10.141.16.2:6789/0 1 ==== mon_map magic: 0 v1 ==== 397+0+0 (3021601431 0 0) 0x55b2155b1200 con 0x55b2159e2c00   -269> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1   -268> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient(hunting):  got monmap 3, mon.mds02 is now rank 1   -267> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient(hunting): dump: 
epoch 3
fsid 92bfcf0a-1d39-43b3-b60f-44f01b630e47
last_changed 2018-08-06 08:11:26.469740
created 0.000000
0: 10.141.16.1:6789/0 mon.mds01
1: 10.141.16.2:6789/0 mon.mds02
2: 10.141.16.3:6789/0 mon.mds03
  -266> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 <== mon.2 10.141.16.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (161163582 0 0) 0x55b2155b1b00 con 0x55b2159e2600   -265> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient(hunting): my global_id is 30077849   -264> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 --> 10.141.16.3:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55b2155b1440 con 0   -263> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 <== mon.1 10.141.16.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3187044275 0 0) 0x55b2155b1d40 con 0x55b2159e2c00   -262> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient(hunting): my global_id is 30067828   -261> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 --> 10.141.16.2:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55b2155b1b00 con 0   -260> 2019-05-06 15:11:45.232 7f28e3a19700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7774783 cs=1 l=1). rx mon.2 seq 3 0x55b2155b1440 auth_reply(proto 2 0 (0) Success) v1   -259> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 <== mon.2 10.141.16.3:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (1601596161 0 0) 0x55b2155b1440 con 0x55b2159e2600   -258> 2019-05-06 15:11:45.232 7f28e3218700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.2:6789/0 conn(0x55b2159e2c00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7963360 cs=1 l=1). rx mon.1 seq 3 0x55b2155b1b00 auth_reply(proto 2 0 (0) Success) v1   -257> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 >> 10.141.16.2:6789/0 conn(0x55b2159e2c00 :-1 s=STATE_OPEN pgs=7963360 cs=1 l=1).mark_down   -256> 2019-05-06 15:11:45.232 7f28e2a17700  2 -- 10.141.16.3:0/3652030958 >> 10.141.16.2:6789/0 conn(0x55b2159e2c00 :-1 s=STATE_OPEN pgs=7963360 cs=1 l=1)._stop   -255> 2019-05-06 15:11:45.232 7f28e2a17700  1 monclient: found mon.mds03   -254> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient: _send_mon_message to mon.mds03 at 10.141.16.3:6789/0   -253> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 --> 10.141.16.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b215911c00 con 0   -252> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-05-06 15:11:15.239399)   -251> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient: _send_mon_message to mon.mds03 at 10.141.16.3:6789/0   -250> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 --> 10.141.16.3:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55b2155b1b00 con 0   -249> 2019-05-06 15:11:45.232 7f28f4321d80  5 monclient: authenticate success, global_id 30077849   -248> 2019-05-06 15:11:45.232 7f28e3a19700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7774783 cs=1 l=1). rx mon.2 seq 4 0x55b2159f4000 config(0 keys) v1   -247> 2019-05-06 15:11:45.232 7f28e3a19700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7774783 cs=1 l=1). rx mon.2 seq 5 0x55b2155b1b00 mon_map magic: 0 v1   -246> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 <== mon.2 10.141.16.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x55b2159f4000 con 0x55b2159e2600   -245> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient: handle_config config(0 keys) v1   -244> 2019-05-06 15:11:45.232 7f28e2a17700  1 -- 10.141.16.3:0/3652030958 <== mon.2 10.141.16.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 397+0+0 (3021601431 0 0) 0x55b2155b1b00 con 0x55b2159e2600   -243> 2019-05-06 15:11:45.232 7f28f4321d80 10 monclient: get_monmap_and_config success 
  -242> 2019-05-06 15:11:45.232 7f28f4321d80 10 monclient: shutdown
  -241> 2019-05-06 15:11:45.232 7f28e1214700  4 set_mon_vals no callback set   -240> 2019-05-06 15:11:45.232 7f28f4321d80  1 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7774783 cs=1 l=1).mark_down   -239> 2019-05-06 15:11:45.232 7f28e3a19700  5 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7774783 cs=1 l=1). rx mon.2 seq 6 0x55b2159fa000 auth_reply(proto 2 0 (0) Success) v1   -238> 2019-05-06 15:11:45.232 7f28f4321d80  2 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:6789/0 conn(0x55b2159e2600 :-1 s=STATE_OPEN pgs=7774783 cs=1 l=1)._stop   -237> 2019-05-06 15:11:45.232 7f28e2a17700 10 monclient: discarding stray monitor message mon_map magic: 0 v1   -236> 2019-05-06 15:11:45.232 7f28f4321d80  1 -- 10.141.16.3:0/3652030958 shutdown_connections   -235> 2019-05-06 15:11:45.232 7f28f4321d80  5 -- 10.141.16.3:0/3652030958 shutdown_connections mark down 10.141.16.2:6789/0 0x55b2159e2c00   -234> 2019-05-06 15:11:45.232 7f28f4321d80  5 -- 10.141.16.3:0/3652030958 shutdown_connections mark down 10.141.16.3:6789/0 0x55b2159e2600   -233> 2019-05-06 15:11:45.232 7f28f4321d80  5 -- 10.141.16.3:0/3652030958 shutdown_connections delete 0x55b2159e2600   -232> 2019-05-06 15:11:45.232 7f28f4321d80  5 -- 10.141.16.3:0/3652030958 shutdown_connections delete 0x55b2159e2c00   -231> 2019-05-06 15:11:45.232 7f28f4321d80  1 -- 10.141.16.3:0/3652030958 shutdown_connections   -230> 2019-05-06 15:11:45.232 7f28f4321d80  1 -- 10.141.16.3:0/3652030958 wait complete.   -229> 2019-05-06 15:11:45.232 7f28f4321d80  1 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:0/3652030958 conn(0x55b2159e2000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down   -228> 2019-05-06 15:11:45.232 7f28f4321d80  2 -- 10.141.16.3:0/3652030958 >> 10.141.16.3:0/3652030958 conn(0x55b2159e2000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop   -227> 2019-05-06 15:11:45.232 7f28f4321d80  0 set uid:gid to 167:167 (ceph:ceph)   -226> 2019-05-06 15:11:45.232 7f28f4321d80  0 ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic (stable), process ceph-osd, pid 142304   -225> 2019-05-06 15:11:45.232 7f28f4321d80  0 pidfile_write: ignore empty --pid-file   -224> 2019-05-06 15:11:45.262 7f28f4321d80  5 asok(0x55b2155de5a0) init /var/run/ceph/ceph-osd.3.asok   -223> 2019-05-06 15:11:45.262 7f28f4321d80  5 asok(0x55b2155de5a0) bind_and_listen /var/run/ceph/ceph-osd.3.asok   -222> 2019-05-06 15:11:45.272 7f28f4321d80  5 asok(0x55b2155de5a0) register_command 0 hook 0x55b2155b5488   -221> 2019-05-06 15:11:45.272 7f28f4321d80  5 asok(0x55b2155de5a0) register_command version hook 0x55b2155b5488   -220> 2019-05-06 15:11:45.272 7f28f4321d80  5 asok(0x55b2155de5a0) register_command git_version hook 0x55b2155b5488   -219> 2019-05-06 15:11:45.272 7f28f4321d80  5 asok(0x55b2155de5a0) register_command help hook 0x55b2155b6320   -218> 2019-05-06 15:11:45.272 7f28f4321d80  5 asok(0x55b2155de5a0) register_command get_command_descriptions hook 0x55b2155b6170   -217> 2019-05-06 15:11:45.272 7f28e2216700  5 asok(0x55b2155de5a0) entry start   -216> 2019-05-06 15:11:45.272 7f28f4321d80  5 object store type is bluestore   -215> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:0/0 learned_addr learned my addr 10.141.16.3:0/0   -214> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:6801/142304 _finish_bind bind my_inst.addr is 10.141.16.3:6801/142304   -213> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:0/0 learned_addr learned my addr 10.141.16.3:0/0   -212> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:6803/142304 _finish_bind bind my_inst.addr is 10.141.16.3:6803/142304   -211> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:0/0 learned_addr learned my addr 10.141.16.3:0/0   -210> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:6804/142304 _finish_bind bind my_inst.addr is 10.141.16.3:6804/142304   -209> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:0/0 learned_addr learned my addr 10.141.16.3:0/0   -208> 2019-05-06 15:11:45.272 7f28f4321d80  1 -- 10.141.16.3:6805/142304 _finish_bind bind my_inst.addr is 10.141.16.3:6805/142304   -207> 2019-05-06 15:11:45.272 7f28f4321d80 10 monclient: build_initial_monmap   -206> 2019-05-06 15:11:45.282 7f28f4321d80  0 load: jerasure load: lrc load: isa   -205> 2019-05-06 15:11:45.282 7f28f4321d80  5 adding auth protocol: cephx   -204> 2019-05-06 15:11:45.282 7f28f4321d80  5 adding auth protocol: cephx   -203> 2019-05-06 15:11:45.282 7f28f4321d80  1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel   -202> 2019-05-06 15:11:45.282 7f28f4321d80  1 bdev(0x55b21594c000 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block   -201> 2019-05-06 15:11:45.282 7f28f4321d80  1 bdev(0x55b21594c000 /var/lib/ceph/osd/ceph-3/block) open size 161860288512 (0x25afa00000, 151 GiB) block_size 4096 (4 KiB) rotational   -200> 2019-05-06 15:11:45.282 7f28f4321d80  1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2   -199> 2019-05-06 15:11:45.282 7f28f4321d80  1 bdev(0x55b21594c000 /var/lib/ceph/osd/ceph-3/block) close   -198> 2019-05-06 15:11:45.652 7f28f4321d80  5 asok(0x55b2155de5a0) register_command objecter_requests hook 0x55b2155b6710   -197> 2019-05-06 15:11:45.652 7f28f4321d80  1 -- 10.141.16.3:6801/142304 start start 
  -196> 2019-05-06 15:11:45.652 7f28f4321d80  1 -- - start start
  -195> 2019-05-06 15:11:45.652 7f28f4321d80  1 -- - start start
  -194> 2019-05-06 15:11:45.652 7f28f4321d80  1 -- 10.141.16.3:6805/142304 start start   -193> 2019-05-06 15:11:45.652 7f28f4321d80  1 -- 10.141.16.3:6804/142304 start start   -192> 2019-05-06 15:11:45.652 7f28f4321d80  1 -- 10.141.16.3:6803/142304 start start 
  -191> 2019-05-06 15:11:45.652 7f28f4321d80  1 -- - start start
  -190> 2019-05-06 15:11:45.652 7f28f4321d80  2 osd.3 0 init /var/lib/ceph/osd/ceph-3 (looks like hdd)   -189> 2019-05-06 15:11:45.652 7f28f4321d80  2 osd.3 0 journal /var/lib/ceph/osd/ceph-3/journal   -188> 2019-05-06 15:11:45.652 7f28f4321d80  1 bluestore(/var/lib/ceph/osd/ceph-3) _mount path /var/lib/ceph/osd/ceph-3   -187> 2019-05-06 15:11:45.652 7f28f4321d80  1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel   -186> 2019-05-06 15:11:45.652 7f28f4321d80  1 bdev(0x55b21594c000 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block   -185> 2019-05-06 15:11:45.652 7f28f4321d80  1 bdev(0x55b21594c000 /var/lib/ceph/osd/ceph-3/block) open size 161860288512 (0x25afa00000, 151 GiB) block_size 4096 (4 KiB) rotational   -184> 2019-05-06 15:11:45.652 7f28f4321d80  1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2   -183> 2019-05-06 15:11:45.652 7f28f4321d80  1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel   -182> 2019-05-06 15:11:45.652 7f28f4321d80  1 bdev(0x55b21594ca80 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block   -181> 2019-05-06 15:11:45.652 7f28f4321d80  1 bdev(0x55b21594ca80 /var/lib/ceph/osd/ceph-3/block) open size 161860288512 (0x25afa00000, 151 GiB) block_size 4096 (4 KiB) rotational   -180> 2019-05-06 15:11:45.652 7f28f4321d80  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 151 GiB 
  -179> 2019-05-06 15:11:45.652 7f28f4321d80  1 bluefs mount
  -178> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option compaction_readahead_size = 2097152   -177> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option compression = kNoCompression   -176> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option max_write_buffer_number = 4   -175> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option min_write_buffer_number_to_merge = 1   -174> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option recycle_log_file_num = 4   -173> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option writable_file_max_buffer_size = 0   -172> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option write_buffer_size = 268435456   -171> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option compaction_readahead_size = 2097152   -170> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option compression = kNoCompression   -169> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option max_write_buffer_number = 4   -168> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option min_write_buffer_number_to_merge = 1   -167> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option recycle_log_file_num = 4   -166> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option writable_file_max_buffer_size = 0   -165> 2019-05-06 15:11:45.672 7f28f4321d80  0  set rocksdb option write_buffer_size = 268435456   -164> 2019-05-06 15:11:45.672 7f28f4321d80  1 rocksdb: do_open column families: [default]   -163> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: RocksDB version: 5.13.0
  -162> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Git sha rocksdb_build_git_sha:@0@   -161> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Compile date Mar 12 2019 
  -160> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: DB SUMMARY
  -159> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: CURRENT file:  CURRENT
  -158> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: IDENTITY file:  IDENTITY
  -157> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: MANIFEST file:  MANIFEST-100046 size: 107514 Bytes
  -156> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: SST files in db dir, Total Num: 997, files: 000239.sst 000531.sst 000680.sst 000709.sst 001058.sst 003160.sst 003420.sst 003587.sst 004003.sst
  -155> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Write Ahead Log file in db: 100047.log size: 0 ;
  -154> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                         Options.error_if_exists: 0   -153> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                       Options.create_if_missing: 0   -152> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                         Options.paranoid_checks: 1   -151> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                                     Options.env: 0x55b215a043a0   -150> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                                Options.info_log: 0x55b215c27020   -149> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.max_file_opening_threads: 16   -148> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                              Options.statistics: (nil)   -147> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                               Options.use_fsync: 0   -146> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                       Options.max_log_file_size: 0   -145> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615   -144> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                   Options.log_file_time_to_roll: 0   -143> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                       Options.keep_log_file_num: 1000   -142> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                    Options.recycle_log_file_num: 4   -141> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                         Options.allow_fallocate: 1   -140> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                        Options.allow_mmap_reads: 0   -139> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                       Options.allow_mmap_writes: 0   -138> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                        Options.use_direct_reads: 0   -137> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0   -136> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:          Options.create_missing_column_families: 0   -135> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                              Options.db_log_dir:   -134> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                                 Options.wal_dir: db   -133> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.table_cache_numshardbits: 6   -132> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                      Options.max_subcompactions: 1   -131> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                  Options.max_background_flushes: -1   -130> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                         Options.WAL_ttl_seconds: 0   -129> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                       Options.WAL_size_limit_MB: 0   -128> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.manifest_preallocation_size: 4194304   -127> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                     Options.is_fd_close_on_exec: 1   -126> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                   Options.advise_random_on_open: 1   -125> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                    Options.db_write_buffer_size: 0   -124> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                    Options.write_buffer_manager: 0x55b215c44960   -123> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:         Options.access_hint_on_compaction_start: 1   -122> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1   -121> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:           Options.random_access_max_buffer_size: 1048576   -120> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                      Options.use_adaptive_mutex: 0   -119> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                            Options.rate_limiter: (nil)   -118> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0   -117> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                       Options.wal_recovery_mode: 0   -116> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                  Options.enable_thread_tracking: 0   -115> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                  Options.enable_pipelined_write: 0   -114> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:         Options.allow_concurrent_memtable_write: 1   -113> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.enable_write_thread_adaptive_yield: 1   -112> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.write_thread_max_yield_usec: 100   -111> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:            Options.write_thread_slow_yield_usec: 3   -110> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                               Options.row_cache: None   -109> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                              Options.wal_filter: None   -108> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.avoid_flush_during_recovery: 0   -107> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.allow_ingest_behind: 0   -106> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.preserve_deletes: 0   -105> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.two_write_queues: 0   -104> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.manual_wal_flush: 0   -103> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.max_background_jobs: 2   -102> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.max_background_compactions: -1   -101> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.avoid_flush_during_shutdown: 0   -100> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:           Options.writable_file_max_buffer_size: 0    -99> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.delayed_write_rate : 16777216    -98> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.max_total_wal_size: 0    -97> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000    -96> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                   Options.stats_dump_period_sec: 600    -95> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                          Options.max_open_files: -1    -94> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                          Options.bytes_per_sync: 0    -93> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                      Options.wal_bytes_per_sync: 0    -92> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:       Options.compaction_readahead_size: 2097152    -91> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Compression algorithms supported:    -90> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kZSTDNotFinalCompression supported: 0    -89> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kZSTD supported: 0    -88> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kXpressCompression supported: 0    -87> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kLZ4HCCompression supported: 1    -86> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kLZ4Compression supported: 1    -85> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kBZip2Compression supported: 0    -84> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kZlibCompression supported: 1    -83> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:  kSnappyCompression supported: 1    -82> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Fast CRC32 supported: Supported on x86    -81> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-100046
   -80> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]:
   -79> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator    -78> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor    -77> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:        Options.compaction_filter: None    -76> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:        Options.compaction_filter_factory: None    -75> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:         Options.memtable_factory: SkipListFactory    -74> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:            Options.table_factory: BlockBasedTable    -73> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:            table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b2155b59c8) 
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x55b2156013c0
  block_cache_name: BinnedLRUCache
  block_cache_options:
    capacity : 429496729
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
   -72> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:        Options.write_buffer_size: 268435456    -71> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_write_buffer_number: 4    -70> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:          Options.compression: NoCompression    -69> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                  Options.bottommost_compression: Disabled    -68> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:       Options.prefix_extractor: nullptr    -67> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr    -66> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.num_levels: 7    -65> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1    -64> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_write_buffer_number_to_maintain: 0    -63> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:            Options.compression_opts.window_bits: -14    -62> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                  Options.compression_opts.level: -1    -61> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:               Options.compression_opts.strategy: 0    -60> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0    -59> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:         Options.compression_opts.zstd_max_train_bytes: 0    -58> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.level0_file_num_compaction_trigger: 4    -57> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20    -56> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:              Options.level0_stop_writes_trigger: 36    -55> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                   Options.target_file_size_base: 67108864    -54> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:             Options.target_file_size_multiplier: 1    -53> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.max_bytes_for_level_base: 268435456    -52> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0    -51> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000    -50> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1    -49> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1    -48> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1    -47> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1    -46> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1    -45> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1    -44> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1    -43> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8    -42> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                    Options.max_compaction_bytes: 1677721600    -41> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                        Options.arena_block_size: 33554432    -40> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736    -39> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944    -38> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100    -37> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.disable_auto_compactions: 0    -36> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                        Options.compaction_style: kCompactionStyleLevel    -35> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                          Options.compaction_pri: kByCompensatedSize    -34> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_universal.size_ratio: 1    -33> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2    -32> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295    -31> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200    -30> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1    -29> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize    -28> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824    -27> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0    -26> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.compaction_options_fifo.ttl: 0    -25> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                   Options.table_properties_collectors:    -24> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                   Options.inplace_update_support: 0    -23> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                 Options.inplace_update_num_locks: 10000    -22> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000    -21> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb: Options.memtable_huge_page_size: 0    -20> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                           Options.bloom_locality: 0    -19> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                    Options.max_successive_merges: 0    -18> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.optimize_filters_for_hits: 0    -17> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.paranoid_file_checks: 0    -16> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.force_consistency_checks: 0    -15> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                Options.report_bg_io_stats: 0    -14> 2019-05-06 15:11:45.672 7f28f4321d80  4 rocksdb:                               Options.ttl: 0    -13> 2019-05-06 15:11:45.702 7f28f4321d80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:db/MANIFEST-100046 succeeded,manifest_file_number is 100046, next_file_number is 100049, last_sequence is 1525341953, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 100044
   -12> 2019-05-06 15:11:45.702 7f28f4321d80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 100045
   -11> 2019-05-06 15:11:45.712 7f28f4321d80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1557148305719525, "job": 1, "event": "recovery_started", "log_files": [100047]}    -10> 2019-05-06 15:11:45.712 7f28f4321d80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/db_impl_open.cc:551] Recovering log #100047 mode 0     -9> 2019-05-06 15:11:45.712 7f28f4321d80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/version_set.cc:2863] Creating manifest 100049
    -8> 2019-05-06 15:11:45.712 7f28f4321d80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1557148305722323, "job": 1, "event": "recovery_finished"}     -7> 2019-05-06 15:11:45.712 7f28f4321d80  5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//MANIFEST-100046 type=3 #100046 -- OK
    -6> 2019-05-06 15:11:45.712 7f28f4321d80  5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/db_impl_files.cc:380] [JOB 2] Delete db//100047.log type=0 #100047 -- OK
    -5> 2019-05-06 15:11:45.712 7f28f4321d80  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x55b215bee000     -4> 2019-05-06 15:11:45.712 7f28f4321d80  1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 
    -3> 2019-05-06 15:11:45.732 7f28f4321d80  1 freelist init
    -2> 2019-05-06 15:11:45.742 7f28f4321d80  1 bluestore(/var/lib/ceph/osd/ceph-3) _open_alloc opening allocation metadata     -1> 2019-05-06 15:11:45.812 7f28f4321d80  1 bluestore(/var/lib/ceph/osd/ceph-3) _open_alloc loaded 149 GiB in 5011 extents      0> 2019-05-06 15:11:45.822 7f28f4321d80 -1 *** Caught signal (Segmentation fault) ** 
 in thread 7f28f4321d80 thread_name:ceph-osd
 ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic (stable) 
 1: (()+0x913410) [0x55b213c42410]
 2: (()+0xf5d0) [0x7f28e82845d0]
 3: (BitMapAreaIN::set_blocks_used_int(long, long)+0x41) [0x55b213c23d41] 
 4: (BitMapAreaIN::set_blocks_used(long, long)+0x35) [0x55b213c24025]
 5: (BitMapAreaIN::set_blocks_used_int(long, long)+0x7d) [0x55b213c23d7d] 
 6: (BitAllocator::set_blocks_used(long, long)+0x84) [0x55b213c25c84]
 7: (BlueStore::_open_alloc()+0x352) [0x55b213add752]
 8: (BlueStore::_mount(bool, bool)+0x642) [0x55b213b53822]
 9: (OSD::init()+0x28f) [0x55b2136fd08f]
 10: (main()+0x23a3) [0x55b2135db363]
 11: (__libc_start_main()+0xf5) [0x7f28e72913d5]
 12: (()+0x384ab0) [0x55b2136b3ab0]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.3.log
--- end dump of recent events ---



Something I can do about this?


Thanks!!

Kenneth

On 01/05/2019 09:24, Igor Fedotov wrote:

Hi Igor,
yeah, BlueStore allocators are absolutely interchangeable. You can switch between them for free. 


Thanks,

Igor


On 5/1/2019 8:59 AM, Igor Podlesny wrote:

On Tue, 30 Apr 2019 at 20:56, Igor Podlesny <ceph-user@xxxxxxxx> wrote:

On Tue, 30 Apr 2019 at 19:10, Denny Fuchs <linuxmail@xxxxxxxx> wrote:
[..]

Any suggestions ?

-- Try different allocator.

Ah, BTW, except memory allocator there's another option: recently
backported bitmap allocator.
Igor Fedotov wrote about it's expected to have lesser memory footprint
with time:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-April/034299.html 

Also I'm not sure though if it's okay to switch existent OSDs "on-fly"
-- changing config and restarting OSDs.
Igor (Fedotov), can you please elaborate on this matter?


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux