Sage Weil writes:
>> 2019-06-12 23:40:43.555 7f724b27f0c0 1 rocksdb: do_open column families: [default]
>> Unrecognized command: stats
>> ceph-kvstore-tool: /build/ceph-14.2.1/src/rocksdb/db/version_set.cc:356: rocksdb::Version::~Version(): Assertion `path_id < cfd_->ioptions()->cf_paths.size()' failed.
>> *** Caught signal (Aborted) **
> Ah, this looks promising.. it looks like it got it open and has some
> problem with teh error/teardown path.
> Try 'compact' instead of 'stats'?
That run for a while and then crashed, also in the destructor for
rocksdb::Version, but with an otherwise different backtrace. I'm
attaching the log again.
--
Simon.
leinen@unil0047:/mnt/ceph/db$ sudo ceph-kvstore-tool rocksdb /mnt/ceph/db compact
2019-06-13 00:00:08.650 7f00f4c510c0 1 rocksdb: do_open column families: [default]
ceph-kvstore-tool: /build/ceph-14.2.1/src/rocksdb/db/version_set.cc:356: rocksdb::Version::~Version(): Assertion `path_id < cfd_->ioptions()->cf_paths.size()' failed.
*** Caught signal (Aborted) **
in thread 7f00e5788700 thread_name:rocksdb:low0
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f00ea641890]
2: (gsignal()+0xc7) [0x7f00e9531e97]
3: (abort()+0x141) [0x7f00e9533801]
4: (()+0x3039a) [0x7f00e952339a]
5: (()+0x30412) [0x7f00e9523412]
6: (rocksdb::Version::~Version()+0x224) [0x5603bd78bfe4]
7: (rocksdb::Version::Unref()+0x35) [0x5603bd78c065]
8: (rocksdb::Compaction::~Compaction()+0x25) [0x5603bd880f05]
9: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc46) [0x5603bd6dab76]
10: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x141) [0x5603bd6dcfa1]
11: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x5603bd6dd5f7]
12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x267) [0x5603bd8dc847]
13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x49) [0x5603bd8dca29]
14: (()+0xbd57f) [0x7f00e9f5757f]
15: (()+0x76db) [0x7f00ea6366db]
16: (clone()+0x3f) [0x7f00e961488f]
2019-06-13 00:05:09.471 7f00e5788700 -1 *** Caught signal (Aborted) **
in thread 7f00e5788700 thread_name:rocksdb:low0
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f00ea641890]
2: (gsignal()+0xc7) [0x7f00e9531e97]
3: (abort()+0x141) [0x7f00e9533801]
4: (()+0x3039a) [0x7f00e952339a]
5: (()+0x30412) [0x7f00e9523412]
6: (rocksdb::Version::~Version()+0x224) [0x5603bd78bfe4]
7: (rocksdb::Version::Unref()+0x35) [0x5603bd78c065]
8: (rocksdb::Compaction::~Compaction()+0x25) [0x5603bd880f05]
9: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc46) [0x5603bd6dab76]
10: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x141) [0x5603bd6dcfa1]
11: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x5603bd6dd5f7]
12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x267) [0x5603bd8dc847]
13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x49) [0x5603bd8dca29]
14: (()+0xbd57f) [0x7f00e9f5757f]
15: (()+0x76db) [0x7f00ea6366db]
16: (clone()+0x3f) [0x7f00e961488f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-23> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command assert hook 0x5603be844130
-22> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command abort hook 0x5603be844130
-21> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perfcounters_dump hook 0x5603be844130
-20> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command 1 hook 0x5603be844130
-19> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf dump hook 0x5603be844130
-18> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perfcounters_schema hook 0x5603be844130
-17> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf histogram dump hook 0x5603be844130
-16> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command 2 hook 0x5603be844130
-15> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf schema hook 0x5603be844130
-14> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf histogram schema hook 0x5603be844130
-13> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf reset hook 0x5603be844130
-12> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config show hook 0x5603be844130
-11> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config help hook 0x5603be844130
-10> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config set hook 0x5603be844130
-9> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config unset hook 0x5603be844130
-8> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config get hook 0x5603be844130
-7> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config diff hook 0x5603be844130
-6> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config diff get hook 0x5603be844130
-5> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command log flush hook 0x5603be844130
-4> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command log dump hook 0x5603be844130
-3> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command log reopen hook 0x5603be844130
-2> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command dump_mempools hook 0x5603bf6be068
-1> 2019-06-13 00:00:08.650 7f00f4c510c0 1 rocksdb: do_open column families: [default]
0> 2019-06-13 00:05:09.471 7f00e5788700 -1 *** Caught signal (Aborted) **
in thread 7f00e5788700 thread_name:rocksdb:low0
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f00ea641890]
2: (gsignal()+0xc7) [0x7f00e9531e97]
3: (abort()+0x141) [0x7f00e9533801]
4: (()+0x3039a) [0x7f00e952339a]
5: (()+0x30412) [0x7f00e9523412]
6: (rocksdb::Version::~Version()+0x224) [0x5603bd78bfe4]
7: (rocksdb::Version::Unref()+0x35) [0x5603bd78c065]
8: (rocksdb::Compaction::~Compaction()+0x25) [0x5603bd880f05]
9: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc46) [0x5603bd6dab76]
10: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x141) [0x5603bd6dcfa1]
11: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x5603bd6dd5f7]
12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x267) [0x5603bd8dc847]
13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x49) [0x5603bd8dca29]
14: (()+0xbd57f) [0x7f00e9f5757f]
15: (()+0x76db) [0x7f00ea6366db]
16: (clone()+0x3f) [0x7f00e961488f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file
--- end dump of recent events ---
--- begin dump of recent events ---
-23> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command assert hook 0x5603be844130
-22> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command abort hook 0x5603be844130
-21> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perfcounters_dump hook 0x5603be844130
-20> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command 1 hook 0x5603be844130
-19> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf dump hook 0x5603be844130
-18> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perfcounters_schema hook 0x5603be844130
-17> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf histogram dump hook 0x5603be844130
-16> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command 2 hook 0x5603be844130
-15> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf schema hook 0x5603be844130
-14> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf histogram schema hook 0x5603be844130
-13> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command perf reset hook 0x5603be844130
-12> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config show hook 0x5603be844130
-11> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config help hook 0x5603be844130
-10> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config set hook 0x5603be844130
-9> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config unset hook 0x5603be844130
-8> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config get hook 0x5603be844130
-7> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config diff hook 0x5603be844130
-6> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command config diff get hook 0x5603be844130
-5> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command log flush hook 0x5603be844130
-4> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command log dump hook 0x5603be844130
-3> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command log reopen hook 0x5603be844130
-2> 2019-06-13 00:00:08.602 7f00f4c510c0 5 asok(0x5603bebba000) register_command dump_mempools hook 0x5603bf6be068
-1> 2019-06-13 00:00:08.650 7f00f4c510c0 1 rocksdb: do_open column families: [default]
0> 2019-06-13 00:05:09.471 7f00e5788700 -1 *** Caught signal (Aborted) **
in thread 7f00e5788700 thread_name:rocksdb:low0
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f00ea641890]
2: (gsignal()+0xc7) [0x7f00e9531e97]
3: (abort()+0x141) [0x7f00e9533801]
4: (()+0x3039a) [0x7f00e952339a]
5: (()+0x30412) [0x7f00e9523412]
6: (rocksdb::Version::~Version()+0x224) [0x5603bd78bfe4]
7: (rocksdb::Version::Unref()+0x35) [0x5603bd78c065]
8: (rocksdb::Compaction::~Compaction()+0x25) [0x5603bd880f05]
9: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*)+0xc46) [0x5603bd6dab76]
10: (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x141) [0x5603bd6dcfa1]
11: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) [0x5603bd6dd5f7]
12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x267) [0x5603bd8dc847]
13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x49) [0x5603bd8dca29]
14: (()+0xbd57f) [0x7f00e9f5757f]
15: (()+0x76db) [0x7f00ea6366db]
16: (clone()+0x3f) [0x7f00e961488f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file /var/lib/ceph/crash/2019-06-12_22:05:09.477430Z_f3f94e69-6267-4cd1-8720-97680186829e/log
--- end dump of recent events ---
Aborted
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
