OSD crash after change of osd_memory_target

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,


I'm having troubles changing osd_memory_target on my test cluster. I've upgraded whole cluster from luminous to nautiuls, all OSDs are running bluestore. Because this testlab is short in RAM, I wanted to lower osd_memory_target to save some memory.

# ceph version
ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)

# ceph config set osd osd_memory_target 2147483648
# ceph config dump
WHO   MASK LEVEL    OPTION                    VALUE        RO
  mon      advanced auth_client_required      cephx        *
  mon      advanced auth_cluster_required     cephx        *
  mon      advanced auth_service_required     cephx        *
  mon      advanced mon_allow_pool_delete     true
  mon      advanced mon_max_pg_per_osd        500
  mgr      advanced mgr/balancer/active       true
  mgr      advanced mgr/balancer/mode         crush-compat
  osd      advanced osd_crush_update_on_start true
  osd      advanced osd_max_backfills         4
  osd      basic    osd_memory_target         2147483648

Now any OSD is unable to start/restart:

# /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

LOG /var/log/ceph/ceph-osd.0.log:

min_mon_release 14 (nautilus)
0: [v2:10.0.92.69:3300/0,v1:10.0.92.69:6789/0] mon.testlab-ceph-03
1: [v2:10.0.92.72:3300/0,v1:10.0.92.72:6789/0] mon.testlab-ceph-04
2: [v2:10.0.92.67:3300/0,v1:10.0.92.67:6789/0] mon.testlab-ceph-01
3: [v2:10.0.92.68:3300/0,v1:10.0.92.68:6789/0] mon.testlab-ceph-02

   -54> 2020-01-21 11:45:19.289 7f6aa5d78700  1 monclient:  mon.2 has (v2) addrs [v2:10.0.92.67:3300/0,v1:10.0.92.67:6789/0] but i'm connected to v1:10.0.92.67:6789/0, reconnecting
   -53> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient: _reopen_session rank -1
   -52> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient(hunting): picked mon.testlab-ceph-01 con 0x563319682880 addr [v2:10.0.92.67:3300/0,v1:10.0.92.67:6789/0]
   -51> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient(hunting): picked mon.testlab-ceph-04 con 0x563319682d00 addr [v2:10.0.92.72:3300/0,v1:10.0.92.72:6789/0]
   -50> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient(hunting): picked mon.testlab-ceph-02 con 0x563319683180 addr [v2:10.0.92.68:3300/0,v1:10.0.92.68:6789/0]
   -49> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient(hunting): start opening mon connection
   -48> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient(hunting): start opening mon connection
   -47> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient(hunting): start opening mon connection
   -46> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient(hunting): _renew_subs
   -45> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient(hunting): get_auth_request con 0x563319682880 auth_method 0
   -44> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2]
   -43> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient(hunting): _init_auth method 2
   -42> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient(hunting): handle_auth_reply_more payload 9
   -41> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9
   -40> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
   -39> 2020-01-21 11:45:19.289 7f6aa6579700 10 monclient(hunting): get_auth_request con 0x563319682d00 auth_method 0
   -38> 2020-01-21 11:45:19.289 7f6aa6579700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2]
   -37> 2020-01-21 11:45:19.289 7f6aa6579700 10 monclient(hunting): _init_auth method 2
   -36> 2020-01-21 11:45:19.289 7f6aa757b700 10 monclient(hunting): get_auth_request con 0x563319683180 auth_method 0
   -35> 2020-01-21 11:45:19.289 7f6aa757b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2]
   -34> 2020-01-21 11:45:19.289 7f6aa757b700 10 monclient(hunting): _init_auth method 2
   -33> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient(hunting): handle_auth_done global_id 5638238 payload 386
   -32> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient: _finish_hunting 0
   -31> 2020-01-21 11:45:19.289 7f6aa6d7a700  1 monclient: found mon.testlab-ceph-01
   -30> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient: _send_mon_message to mon.testlab-ceph-01 at v2:10.0.92.67:3300/0
   -29> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient: _finish_auth 0
   -28> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-01-21 11:44:49.293059)
   -27> 2020-01-21 11:45:19.289 7f6aa6d7a700 10 monclient: _send_mon_message to mon.testlab-ceph-01 at v2:10.0.92.67:3300/0
   -26> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient: handle_monmap mon_map magic: 0 v1
   -25> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient:  got monmap 17 from mon.testlab-ceph-01 (according to old e17)
   -24> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient: dump:
epoch 17
fsid f42082cc-c35a-44fe-b7ef-c2eb2ff1fe43
last_changed 2020-01-20 10:35:23.579081
created 2018-04-25 17:07:31.881451
min_mon_release 14 (nautilus)
0: [v2:10.0.92.69:3300/0,v1:10.0.92.69:6789/0] mon.testlab-ceph-03
1: [v2:10.0.92.72:3300/0,v1:10.0.92.72:6789/0] mon.testlab-ceph-04
2: [v2:10.0.92.67:3300/0,v1:10.0.92.67:6789/0] mon.testlab-ceph-01
3: [v2:10.0.92.68:3300/0,v1:10.0.92.68:6789/0] mon.testlab-ceph-02

   -23> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient: handle_config config(3 keys) v1
   -22> 2020-01-21 11:45:19.289 7f6aa7781c80 10 monclient: get_monmap_and_config success
   -21> 2020-01-21 11:45:19.289 7f6aa7781c80 10 monclient: shutdown
   -20> 2020-01-21 11:45:19.289 7f6aa4575700  4 set_mon_vals no callback set
   -19> 2020-01-21 11:45:19.289 7f6aa5d78700 10 monclient: discarding stray monitor message mon_map magic: 0 v1
   -18> 2020-01-21 11:45:19.289 7f6aa4575700 10 set_mon_vals osd_crush_update_on_start = true
   -17> 2020-01-21 11:45:19.289 7f6aa4575700 10 set_mon_vals osd_max_backfills = 4
   -16> 2020-01-21 11:45:19.289 7f6aa4575700 10 set_mon_vals osd_memory_target = 2147483648
   -15> 2020-01-21 11:45:19.297 7f6aa7781c80  0 set uid:gid to 64045:64045 (ceph:ceph)
   -14> 2020-01-21 11:45:19.297 7f6aa7781c80  0 ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable), process ceph-osd, pid 728019
   -13> 2020-01-21 11:45:20.681 7f6aa7781c80  0 pidfile_write: ignore empty --pid-file
   -12> 2020-01-21 11:45:20.685 7f6aa7781c80  5 asok(0x563319688000) init /var/run/ceph/ceph-osd.0.asok
   -11> 2020-01-21 11:45:20.685 7f6aa7781c80  5 asok(0x563319688000) bind_and_listen /var/run/ceph/ceph-osd.0.asok
   -10> 2020-01-21 11:45:20.685 7f6aa7781c80  5 asok(0x563319688000) register_command 0 hook 0x5633196003f0
    -9> 2020-01-21 11:45:20.685 7f6aa7781c80  5 asok(0x563319688000) register_command version hook 0x5633196003f0
    -8> 2020-01-21 11:45:20.685 7f6aa7781c80  5 asok(0x563319688000) register_command git_version hook 0x5633196003f0
    -7> 2020-01-21 11:45:20.685 7f6aa7781c80  5 asok(0x563319688000) register_command help hook 0x563319602220
    -6> 2020-01-21 11:45:20.685 7f6aa7781c80  5 asok(0x563319688000) register_command get_command_descriptions hook 0x563319602260
    -5> 2020-01-21 11:45:20.685 7f6aa4d76700  5 asok(0x563319688000) entry start
    -4> 2020-01-21 11:45:20.685 7f6aa7781c80  5 object store type is bluestore
    -3> 2020-01-21 11:45:20.689 7f6aa7781c80  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -2> 2020-01-21 11:45:20.689 7f6aa7781c80  1 bdev(0x56331a2d8000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -1> 2020-01-21 11:45:20.689 7f6aa7781c80  1 bdev(0x56331a2d8000 /var/lib/ceph/osd/ceph-0/block) open size 2000381018112 (0x1d1c0000000, 1.8 TiB) block_size 4096 (4 KiB) rotational discard not supported
     0> 2020-01-21 11:45:20.693 7f6aa7781c80 -1 *** Caught signal (Aborted) **
 in thread 7f6aa7781c80 thread_name:ceph-osd

 ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
 1: (()+0x12730) [0x7f6aa8229730]
 2: (gsignal()+0x10b) [0x7f6aa7d0d7bb]
 3: (abort()+0x121) [0x7f6aa7cf8535]
 4: (()+0x8c983) [0x7f6aa80c0983]
 5: (()+0x928c6) [0x7f6aa80c68c6]
 6: (()+0x92901) [0x7f6aa80c6901]
 7: (()+0x92b34) [0x7f6aa80c6b34]
 8: (()+0x5a3f53) [0x56330f0a0f53]
 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x81) [0x56330f0a6c91]
 10: (BlueStore::_set_cache_sizes()+0x15a) [0x56330f521d8a]
 11: (BlueStore::_open_bdev(bool)+0x173) [0x56330f524b23]
 12: (BlueStore::get_devices(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)+0xef) [0x56330f58b7ef]
 13: (BlueStore::get_numa_node(int*, std::set<int, std::less<int>, std::allocator<int> >*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)+0x7b) [0x56330f53371b]
 14: (main()+0x2870) [0x56330f06e440]
 15: (__libc_start_main()+0xeb) [0x7f6aa7cfa09b]
 16: (_start()+0x2a) [0x56330f0a0c6a]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 0 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
   1/ 5 prioritycache
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---

When I remove this option:
# ceph config rm osd osd_memory_target

OSD starts without any trouble. I've seen same behaviour when I wrote this parameter into /etc/ceph/ceph.conf

Is this a known bug? Am I doing something wrong?

Any help appreciated.

Best Regards,

Martin

-- 
Martin Mlynář
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux