Hi Boris,
highly likely you've faced https://tracker.ceph.com/issues/47751
It's fixed in upcoming Nautilus release but v14.2.16 still lacks the fix.
As a workaround you might want to switch back to bitmap or avl allocator.
Thanks,
Igor
On 2/17/2021 12:36 PM, Boris Behrens wrote:
Hi,
currently we experience osd daemon crashes and I can't pin the issue. I
hope someone can help me with it.
* We operate multiple cluster (440 SSD - 1PB, 36 SSD - 126TB, 40SSD 100TB,
84HDD - 680TB)
* All clusters were updated around the same time (2021-02-03)
* We restarted ALL ceph daemons (systemctl restart ceph.target) on
2021-02-11 after we added OOMScoreAdjust=-900 the all service files.
now in our main cluster (440SSD with 1PB) the OSD daemons begin to crash:
# ceph crash ls
ID ENTITY NEW
2020-03-06_17:37:54.031675Z_0bbbb807-ff2f-46df-9508-58d319b89bd6 osd.397
2020-05-28_12:23:27.677741Z_061f2449-9a36-4747-a2f8-624e72cd1ad0 osd.410
2021-02-05_07:03:35.943384Z_dffab245-4788-4de2-a677-76b735d5fc01 osd.403
2021-02-15_15:41:27.934194Z_97b57f8f-58f2-4390-9d3e-993874e0e000 osd.395
2021-02-15_18:01:19.774879Z_18160e65-4659-451f-8aae-def2984f1f29 osd.178
2021-02-17_04:51:05.101052Z_9f04c6e8-d0c7-442c-9a38-33d5164d2a83 osd.384
osd.384 and osd.395 are on the same node, which had some memory issues we
fixed 2021-02-16_12:00:00
osd.384 was marked as out for >24h when the daemon crashed, and there no
more misplaced objects in the cluster.
Here is the latest crash dump
--- begin dump of recent events ---
-9999> 2021-02-17 03:31:31.305 7fcf7e136700 1 do_command 'perf dump'
'result is 30067 bytes
-9998> 2021-02-17 03:31:31.626 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9997> 2021-02-17 03:31:32.634 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9996> 2021-02-17 03:31:33.639 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9995> 2021-02-17 03:31:34.647 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9994> 2021-02-17 03:31:35.651 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9993> 2021-02-17 03:31:36.654 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9992> 2021-02-17 03:31:37.657 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9991> 2021-02-17 03:31:38.676 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9990> 2021-02-17 03:31:39.680 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9989> 2021-02-17 03:31:40.684 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9988> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump' '
-9987> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump'
'result is 30067 bytes
<snip>
-31> 2021-02-17 05:50:41.158 7fcf7e136700 1 do_command 'perf dump' '
-30> 2021-02-17 05:50:41.159 7fcf7e136700 1 do_command 'perf dump'
'result is 30070 bytes
-29> 2021-02-17 05:50:41.804 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-28> 2021-02-17 05:50:42.813 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-27> 2021-02-17 05:50:43.820 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-26> 2021-02-17 05:50:44.825 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-25> 2021-02-17 05:50:45.831 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-24> 2021-02-17 05:50:46.837 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-23> 2021-02-17 05:50:47.840 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-22> 2021-02-17 05:50:48.843 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-21> 2021-02-17 05:50:49.847 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-20> 2021-02-17 05:50:50.853 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-19> 2021-02-17 05:50:51.524 7fcf7e136700 1 do_command 'perf dump' '
-18> 2021-02-17 05:50:51.525 7fcf7e136700 1 do_command 'perf dump'
'result is 30070 bytes
-17> 2021-02-17 05:50:51.859 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-16> 2021-02-17 05:50:52.862 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-15> 2021-02-17 05:50:53.871 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-14> 2021-02-17 05:50:54.875 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-13> 2021-02-17 05:50:55.886 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-12> 2021-02-17 05:50:56.891 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-11> 2021-02-17 05:50:57.905 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-10> 2021-02-17 05:50:58.911 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-9> 2021-02-17 05:50:59.917 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-8> 2021-02-17 05:51:00.929 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-7> 2021-02-17 05:51:01.566 7fcf7e136700 1 do_command 'perf dump' '
-6> 2021-02-17 05:51:01.567 7fcf7e136700 1 do_command 'perf dump'
'result is 30070 bytes
-5> 2021-02-17 05:51:01.935 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-4> 2021-02-17 05:51:02.943 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-3> 2021-02-17 05:51:03.949 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-2> 2021-02-17 05:51:04.967 7fcf73be6700 5 prioritycache tune_memory
target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208
old mem: 2845415832 new mem: 2845415832
-1> 2021-02-17 05:51:05.091 7fcf743e7700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'uint64_t AllocatorLevel02<T>::claim_free_to_right(uint64_t)
[with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread
7fcf743e7700 time 2021-02-17 05:51:04.998475
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h:
572: FAILED ceph_assert(available >= allocated)
ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x14a) [0x561c84cc2c7d]
2: (()+0x4d8e45) [0x561c84cc2e45]
3: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e)
[0x561c853167de]
4: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned
long, unsigned long, std::less<unsigned long>,
std::allocator<std::pair<unsigned long const, unsigned long> > > >
const&)+0x60) [0x561c85310b20]
5: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned
long, unsigned long, std::less<unsigned long>,
std::allocator<std::pair<unsigned long const, unsigned long> > > >
const&)+0x3a) [0x561c853143ca]
6: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f)
[0x561c851ee83f]
7: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be)
[0x561c8522f4ae]
8: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa)
[0x561c8522fe9a]
9: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4]
10: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed]
11: (()+0x7ea5) [0x7fcf840a2ea5]
12: (clone()+0x6d) [0x7fcf82f6596d]
0> 2021-02-17 05:51:05.145 7fcf743e7700 -1 *** Caught signal (Aborted)
**
in thread 7fcf743e7700 thread_name:bstore_kv_final
ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus
(stable)
1: (()+0xf630) [0x7fcf840aa630]
2: (gsignal()+0x37) [0x7fcf82e9d387]
3: (abort()+0x148) [0x7fcf82e9ea78]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x199) [0x561c84cc2ccc]
5: (()+0x4d8e45) [0x561c84cc2e45]
6: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e)
[0x561c853167de]
7: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned
long, unsigned long, std::less<unsigned long>,
std::allocator<std::pair<unsigned long const, unsigned long> > > >
const&)+0x60) [0x561c85310b20]
8: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned
long, unsigned long, std::less<unsigned long>,
std::allocator<std::pair<unsigned long const, unsigned long> > > >
const&)+0x3a) [0x561c853143ca]
9: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f)
[0x561c851ee83f]
10: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be)
[0x561c8522f4ae]
11: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa)
[0x561c8522fe9a]
12: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4]
13: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed]
14: (()+0x7ea5) [0x7fcf840a2ea5]
15: (clone()+0x6d) [0x7fcf82f6596d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
