On 13.06.19 00:29, Sage Weil wrote:
On Thu, 13 Jun 2019, Simon Leinen wrote:
Sage Weil writes:
2019-06-12 23:40:43.555 7f724b27f0c0 1 rocksdb: do_open column families: [default]
Unrecognized command: stats
ceph-kvstore-tool: /build/ceph-14.2.1/src/rocksdb/db/version_set.cc:356: rocksdb::Version::~Version(): Assertion `path_id < cfd_->ioptions()->cf_paths.size()' failed.
*** Caught signal (Aborted) **
Ah, this looks promising.. it looks like it got it open and has some
problem with teh error/teardown path.
Try 'compact' instead of 'stats'?
That run for a while and then crashed, also in the destructor for
rocksdb::Version, but with an otherwise different backtrace. I'm
attaching the log again.
Hmm, I'm pretty sure this is a shutdown problem, but not certain. If you
do
ceph-kvstore-tool rocksdb /mnt/ceph/db list > keys
is the keys file huge? Can you send the head and tail of it so we can
make sure it looks complete?
It wrote 16GB, then it aborted. Attached key-list.log, keys.tail.
keys.head separately:
ceph-post-file: 5ee5ff3c-ec89-4c2c-aaf9-1ae8ab3f59e3
One last thing to check:
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-NNN list > keys
and see if that behaves similarly or crashes in the way it did before when
the OSD was starting.
This aborts without writing any keys. Attached keys-bs.log.
I tried this once more, after having added a big swap partition, but it
looks very much the same. It aborted after exactly the same time. So it
is not clear if more RAM would help?
Thanks!
Harry
If the exported version looks intact, I have a workaround that will
make the osd use that external rocksdb db instead of the embedded one...
basically,
- symlink the db, db.wal, db.slow files from the osd dir
(/var/lib/ceph/osd/ceph-NNN/db -> ... etc)
- ceph-bluestore-tool --dev /var/lib/ceph/osd/ceph-NNN/block set-label-key -k bluefs -v 0
- start osd
but be warned this is fragile: there isn't a bluefs import function, so
this OSD will be permanently in that weird state. The goal will be to get
it up and the PG/cluster behaving, and then eventually let rados recover
elsewhere and reprovision this osd.
But first, let's make sure the external rocksdb has a complete set of
keys!
sage
root@unil0047:/mnt/ceph# ceph-kvstore-tool rocksdb /mnt/ceph/db list > keys
2019-06-13 07:46:52.905 7f070f8c30c0 1 rocksdb: do_open column families: [default]
ceph-kvstore-tool: /build/ceph-14.2.1/src/rocksdb/db/version_set.cc:356: rocksdb::Version::~Version(): Assertion `path_id < cfd_->ioptions()->cf_paths.size()' failed.
*** Caught signal (Aborted) **
in thread 7f070f8c30c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f07052b3890]
2: (gsignal()+0xc7) [0x7f07041a3e97]
3: (abort()+0x141) [0x7f07041a5801]
4: (()+0x3039a) [0x7f070419539a]
5: (()+0x30412) [0x7f0704195412]
6: (rocksdb::Version::~Version()+0x224) [0x557b0961ffe4]
7: (rocksdb::Version::Unref()+0x35) [0x557b09620065]
8: (rocksdb::SuperVersion::Cleanup()+0x68) [0x557b09705328]
9: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0xf4) [0x557b097083d4]
10: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0xb8) [0x557b09708ba8]
11: (rocksdb::VersionSet::~VersionSet()+0x4d) [0x557b09613a5d]
12: (rocksdb::DBImpl::CloseHelper()+0x6a8) [0x557b09540868]
13: (rocksdb::DBImpl::~DBImpl()+0x65b) [0x557b0954bdeb]
14: (rocksdb::DBImpl::~DBImpl()+0x11) [0x557b0954be21]
15: (RocksDBStore::~RocksDBStore()+0xe9) [0x557b0935b349]
16: (RocksDBStore::~RocksDBStore()+0x9) [0x557b0935b599]
17: (main()+0x307) [0x557b091abfb7]
18: (__libc_start_main()+0xe7) [0x7f0704186b97]
19: (_start()+0x2a) [0x557b0928403a]
2019-06-13 07:55:42.091 7f070f8c30c0 -1 *** Caught signal (Aborted) **
in thread 7f070f8c30c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f07052b3890]
2: (gsignal()+0xc7) [0x7f07041a3e97]
3: (abort()+0x141) [0x7f07041a5801]
4: (()+0x3039a) [0x7f070419539a]
5: (()+0x30412) [0x7f0704195412]
6: (rocksdb::Version::~Version()+0x224) [0x557b0961ffe4]
7: (rocksdb::Version::Unref()+0x35) [0x557b09620065]
8: (rocksdb::SuperVersion::Cleanup()+0x68) [0x557b09705328]
9: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0xf4) [0x557b097083d4]
10: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0xb8) [0x557b09708ba8]
11: (rocksdb::VersionSet::~VersionSet()+0x4d) [0x557b09613a5d]
12: (rocksdb::DBImpl::CloseHelper()+0x6a8) [0x557b09540868]
13: (rocksdb::DBImpl::~DBImpl()+0x65b) [0x557b0954bdeb]
14: (rocksdb::DBImpl::~DBImpl()+0x11) [0x557b0954be21]
15: (RocksDBStore::~RocksDBStore()+0xe9) [0x557b0935b349]
16: (RocksDBStore::~RocksDBStore()+0x9) [0x557b0935b599]
17: (main()+0x307) [0x557b091abfb7]
18: (__libc_start_main()+0xe7) [0x7f0704186b97]
19: (_start()+0x2a) [0x557b0928403a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-23> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command assert hook 0x557b0b2d6130
-22> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command abort hook 0x557b0b2d6130
-21> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perfcounters_dump hook 0x557b0b2d6130
-20> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command 1 hook 0x557b0b2d6130
-19> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf dump hook 0x557b0b2d6130
-18> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perfcounters_schema hook 0x557b0b2d6130
-17> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf histogram dump hook 0x557b0b2d6130
-16> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command 2 hook 0x557b0b2d6130
-15> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf schema hook 0x557b0b2d6130
-14> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf histogram schema hook 0x557b0b2d6130
-13> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf reset hook 0x557b0b2d6130
-12> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config show hook 0x557b0b2d6130
-11> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config help hook 0x557b0b2d6130
-10> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config set hook 0x557b0b2d6130
-9> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config unset hook 0x557b0b2d6130
-8> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config get hook 0x557b0b2d6130
-7> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config diff hook 0x557b0b2d6130
-6> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config diff get hook 0x557b0b2d6130
-5> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command log flush hook 0x557b0b2d6130
-4> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command log dump hook 0x557b0b2d6130
-3> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command log reopen hook 0x557b0b2d6130
-2> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command dump_mempools hook 0x557b0c150068
-1> 2019-06-13 07:46:52.905 7f070f8c30c0 1 rocksdb: do_open column families: [default]
0> 2019-06-13 07:55:42.091 7f070f8c30c0 -1 *** Caught signal (Aborted) **
in thread 7f070f8c30c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f07052b3890]
2: (gsignal()+0xc7) [0x7f07041a3e97]
3: (abort()+0x141) [0x7f07041a5801]
4: (()+0x3039a) [0x7f070419539a]
5: (()+0x30412) [0x7f0704195412]
6: (rocksdb::Version::~Version()+0x224) [0x557b0961ffe4]
7: (rocksdb::Version::Unref()+0x35) [0x557b09620065]
8: (rocksdb::SuperVersion::Cleanup()+0x68) [0x557b09705328]
9: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0xf4) [0x557b097083d4]
10: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0xb8) [0x557b09708ba8]
11: (rocksdb::VersionSet::~VersionSet()+0x4d) [0x557b09613a5d]
12: (rocksdb::DBImpl::CloseHelper()+0x6a8) [0x557b09540868]
13: (rocksdb::DBImpl::~DBImpl()+0x65b) [0x557b0954bdeb]
14: (rocksdb::DBImpl::~DBImpl()+0x11) [0x557b0954be21]
15: (RocksDBStore::~RocksDBStore()+0xe9) [0x557b0935b349]
16: (RocksDBStore::~RocksDBStore()+0x9) [0x557b0935b599]
17: (main()+0x307) [0x557b091abfb7]
18: (__libc_start_main()+0xe7) [0x7f0704186b97]
19: (_start()+0x2a) [0x557b0928403a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file
--- end dump of recent events ---
--- begin dump of recent events ---
-23> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command assert hook 0x557b0b2d6130
-22> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command abort hook 0x557b0b2d6130
-21> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perfcounters_dump hook 0x557b0b2d6130
-20> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command 1 hook 0x557b0b2d6130
-19> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf dump hook 0x557b0b2d6130
-18> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perfcounters_schema hook 0x557b0b2d6130
-17> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf histogram dump hook 0x557b0b2d6130
-16> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command 2 hook 0x557b0b2d6130
-15> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf schema hook 0x557b0b2d6130
-14> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf histogram schema hook 0x557b0b2d6130
-13> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command perf reset hook 0x557b0b2d6130
-12> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config show hook 0x557b0b2d6130
-11> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config help hook 0x557b0b2d6130
-10> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config set hook 0x557b0b2d6130
-9> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config unset hook 0x557b0b2d6130
-8> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config get hook 0x557b0b2d6130
-7> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config diff hook 0x557b0b2d6130
-6> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command config diff get hook 0x557b0b2d6130
-5> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command log flush hook 0x557b0b2d6130
-4> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command log dump hook 0x557b0b2d6130
-3> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command log reopen hook 0x557b0b2d6130
-2> 2019-06-13 07:46:52.881 7f070f8c30c0 5 asok(0x557b0b64c000) register_command dump_mempools hook 0x557b0c150068
-1> 2019-06-13 07:46:52.905 7f070f8c30c0 1 rocksdb: do_open column families: [default]
0> 2019-06-13 07:55:42.091 7f070f8c30c0 -1 *** Caught signal (Aborted) **
in thread 7f070f8c30c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7f07052b3890]
2: (gsignal()+0xc7) [0x7f07041a3e97]
3: (abort()+0x141) [0x7f07041a5801]
4: (()+0x3039a) [0x7f070419539a]
5: (()+0x30412) [0x7f0704195412]
6: (rocksdb::Version::~Version()+0x224) [0x557b0961ffe4]
7: (rocksdb::Version::Unref()+0x35) [0x557b09620065]
8: (rocksdb::SuperVersion::Cleanup()+0x68) [0x557b09705328]
9: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0xf4) [0x557b097083d4]
10: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0xb8) [0x557b09708ba8]
11: (rocksdb::VersionSet::~VersionSet()+0x4d) [0x557b09613a5d]
12: (rocksdb::DBImpl::CloseHelper()+0x6a8) [0x557b09540868]
13: (rocksdb::DBImpl::~DBImpl()+0x65b) [0x557b0954bdeb]
14: (rocksdb::DBImpl::~DBImpl()+0x11) [0x557b0954be21]
15: (RocksDBStore::~RocksDBStore()+0xe9) [0x557b0935b349]
16: (RocksDBStore::~RocksDBStore()+0x9) [0x557b0935b599]
17: (main()+0x307) [0x557b091abfb7]
18: (__libc_start_main()+0xe7) [0x7f0704186b97]
19: (_start()+0x2a) [0x557b0928403a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file /var/lib/ceph/crash/2019-06-13_05:55:42.095268Z_289ce656-ee72-4be7-a7ab-977f51eea09f/log
--- end dump of recent events ---
Aborted
b %00%00%03%a1%8d%80%00%00
b %00%00%03%a1%8e%00%00%00
b %00%00%03%a1%8e%80%00%00
b %00%00%03%a1%8f%00%00%00
b %00%00%03%a1%8f%80%00%00
b %00%00%03%a1%90%00%00%00
b %00%00%03%a1%90%80%00%00
b %00%00%03%a1%91%00%00%00
b %00%00%03%a1%91%80%00%00
b %00%00%03%a1%92%00%00%00
b %00%00%03%a1%92%80%00%00
b %00%00%03%a1%93%00%00%00
b %00%00%03%a1%93%80%00%00
b %00%00%03%a1%94%00%00%00
b %00%00%03%a1%94%80%00%00
b %00%00%03%a1%95%00%00%00
b %00%00%03%a1%95%80%00%00
b %00%00%03%a1%96%00%00%00
b %00%00%03%a1%96%80%00%00
b %00%00%03%a1%97%00%00%00
b %00%00%03%a1%97%80%00%00
b %00%00%03%a1%98%00%00%00
b %00%00%03%a1%98%80%00%00
b %00%00%03%a1%99%00%00%00
b %00%00%03%a1%99%80%00%00
b %00%00%03%a1%9a%00%00%00
b %00%00%03%a1%9a%80%00%00
b %00%00%03%a1%9b%00%00%00
b %00%00%03%a1%9b%80%00%00
b %00%00%03%a1%9c%00%00%00
b %00%00%03%a1%9c%80%00%00
b %00%00%03%a1%9d%00%00%00
b %00%00%03%a1%9d%80%00%00
b %00%00%03%a1%9e%00%00%00
b %00%00%03%a1%9e%80%00%00
b %00%00%03%a1%9f%00%00%00
b %00%00%03%a1%9f%80%00%00
b %00%00%03%a1%a0%00%00%00
b %00%00%03%a1%a0%80%00%00
b %00%00%03%a1%a1%00%00%00
b %00%00%03%a1%a1%80%00%00
b %00%00%03%a1%a2%00%00%00
b %00%00%03%a1%a2%80%00%00
b %00%00%03%a1%a3%00%00%00
b %00%00%03%a1%a3%80%00%00
b %00%00%03%a1%a4%00%00%00
b %00%00%03%a1%a4%80%00%00
b %00%00%03%a1%a5%00%00%00
b %00%00%03%a1%a5%80%00%00
b %00%00%03%a1%a6%00%00%00
b %00%00%03%a1%a6%80%00%00
b %00%00%03%a1%a7%00%00%00
b %00%00%03%a1%a7%80%00%00
b %00%00%03%a1%a8%00%00%00
b %00%00%03%a1%a8%80%00%00
b %00%00%03%a1%a9%00%00%00
b %00%00%03%a1%a9%80%00%00
b %00%00%03%a1%aa%00%00%00
b %00%00%03%a1%aa%80%00%00
b %00%00%03%a1%ab%00%00%00
b %00%00%03%a1%ab%80%00%00
b %00%00%03%a1%ac%00%00%00
b %00%00%03%a1%ac%80%00%00
b %00%00%03%a1%ad%00%00%00
b %00%00%03%a1%ad%80%00%00
b %00%00%03%a1%ae%00%00%00
b %00%00%03%a1%ae%80%00%00
b %00%00%03%a1%af%00%00%00
b %00%00%03%a1%af%80%00%00
b %00%00%03%a1%b0%00%00%00
b %00%00%03%a1%b0%80%00%00
b %00%00%03%a1%b1%00%00%00
b %00%00%03%a1%b1%80%00%00
b %00%00%03%a1%b2%00%00%00
b %00%00%03%a1%b2%80%00%00
b %00%00%03%a1%b3%00%00%00
b %00%00%03%a1%b3%80%00%00
b %00%00%03%a1%b4%00%00%00
b %00%00%03%a1%b4%80%00%00
b %00%00%03%a1%b5%00%00%00
b %00%00%03%a1%b5%80%00%00
b %00%00%03%a1%b6%00%00%00
b %00%00%03%a1%b6%80%00%00
b %00%00%03%a1%b7%00%00%00
b %00%00%03%a1%b7%80%00%00
b %00%00%03%a1%b8%00%00%00
b %00%00%03%a1%b8%80%00%00
b %00%00%03%a1%b9%00%00%00
b %00%00%03%a1%b9%80%00%00
b %00%00%03%a1%ba%00%00%00
b %00%00%03%a1%ba%80%00%00
b %00%00%03%a1%bb%00%00%00
b %00%00%03%a1%bb%80%00%00
b %00%00%03%a1%bc%00%00%00
b %00%00%03%a1%bc%80%00%00
b %00%00%03%a1%bd%00%00%00
b %00%00%03%a1%bd%80%00%00
b %00%00%03%a1%be%00%00%00
b %00%00%03%a1%be%80%00%00
b %00%00%03%a1%bf%00%00%00
b %00%00%03%a1%bf%80%00%00
b %00%00%03%a1%c0%00%00%00
b %00%00%03%a1%c0%80%00%00
b %00%00%03%a1%c1%00%00%00
b %00%00%03%a1%c1%80%00%00
b %00%00%03%a1%c2%00%00%00
b %00%00%03%a1%c2%80%00%00
b %00%00%03%a1%c3%00%00%00
b %00%00%03%a1%c3%80%00%00
b %00%00%03%a1%c4%00%00%00
b %00%00%03%a1%c4%80%00%00
b %00%00%03%a1%c5%00%00%00
b %00%00%03%a1%c5%80%00%00
b %00%00%03%a1%c6%00%00%00
b %00%00%03%a1%c6%80%00%00
b %00%00%03%a1%c7%00%00%00
b %00%00%03%a1%c7%80%00%00
b %00%00%03%a1%c8%00%00%00
b %00%00%03%a1%c8%80%00%00
b %00%00%03%a1%c9%00%00%00
b %00%00%03%a1%c9%80%00%00
b %00%00%03%a1%ca%00%00%00
b %00%00%03%a1%ca%80%00%00
b %00%00%03%a1%cb%00%00%00
b %00%00%03%a1%cb%80%00%00
b %00%00%03%a1%cc%00%00%00
b %00%00%03%a1%cc%80%00%00
b %00%00%03%a1%cd%00%00%00
b %00%00%03%a1%cd%80%00%00
b %00%00%03%a1%ce%00%00%00
b %00%00%03%a1%ce%80%00%00
b %00%00%03%a1%cf%00%00%00
b %00%00%03%a1%cf%80%00%00
b %00%00%03%a1%d0%00%00%00
b %00%00%03%a1%d0%80%00%00
b %00%00%03%a1%d1%00%00%00
b %00%00%03%a1%d1%80%00%00
b %00%00%03%a1%d2%00%00%00
b %00%00%03%a1%d2%80%00%00
b %00%00%03%a1%d3%00%00%00
b %00%00%03%a1%d3%80%00%00
b %00%00%03%a1%d4%00%00%00
b %00%00%03%a1%d4%80%00%00
b %00%00%03%a1%d5%00%00%00
b %00%00%03%a1%d5%80%00%00
b %00%00%03%a1%d6%00%00%00
b %00%00%03%a1%d6%80%00%00
b %00%00%03%a1%d7%00%00%00
b %00%00%03%a1%d7%80%00%00
b %00%00%03%a1%d8%00%00%00
b %00%00%03%a1%d8%80%00%00
b %00%00%03%a1%d9%00%00%00
b %00%00%03%a1%d9%80%00%00
b %00%00%03%a1%da%00%00%00
b %00%00%03%a1%da%80%00%00
b %00%00%03%a1%db%00%00%00
b %00%00%03%a1%db%80%00%00
b %00%00%03%a1%dc%00%00%00
b %00%00%03%a1%dc%80%00%00
b %00%00%03%a1%dd%00%00%00
b %00%00%03%a1%dd%80%00%00
b %00%00%03%a1%de%00%00%00
b %00%00%03%a1%de%80%00%00
b %00%00%03%a1%df%00%00%00
b %00%00%03%a1%df%80%00%00
b %00%00%03%a1%e0%00%00%00
b %00%00%03%a1%e0%80%00%00
b %00%00%03%a1%e1%00%00%00
b %00%00%03%a1%e1%80%00%00
b %00%00%03%a1%e2%00%00%00
b %00%00%03%a1%e2%80%00%00
b %00%00%03%a1%e3%00%00%00
b %00%00%03%a1%e3%80%00%00
b %00%00%03%a1%e4%00%00%00
b %00%00%03%a1%e4%80%00%00
b %00%00%03%a1%e5%00%00%00
b %00%00%03%a1%e5%80%00%00
b %00%00%03%a1%e6%00%00%00
b %00%00%03%a1%e6%80%00%00
b %00%00%03%a1%e7%00%00%00
b %00%00%03%a1%e7%80%00%00
b %00%00%03%a1%e8%00%00%00
b %00%00%03%a1%e8%80%00%00
b %00%00%03%a1%e9%00%00%00
b %00%00%03%a1%e9%80%00%00
b %00%00%03%a1%ea%00%00%00
b %00%00%03%a1%ea%80%00%00
b %00%00%03%a1%eb%00%00%00
b %00%00%03%a1%eb%80%00%00
b %00%00%03%a1%ec%00%00%00
b %00%00%03%a1%ec%80%00%00
b %00%00%03%a1%ed%00%00%00
b %00%00%03%a1%ed%80%00%00
b %00%00%03%a1%ee%00%00%00
b %00%00%03%a1%ee%80%00%00
b %00%00%03%a1%ef%00%00%00
b %00%00%03%a1%ef%80%00%00
b %00%00%03%a1%f0%00%00%00
b %00%00%03%a1%f0%80%00%00
b %00%00%03%a1%f1%00%00%00
b %00%00%03%a1%f1%80%00%00
b %00%00%03%a1%f2%00%00%00
b %00%00%03%a1%f2%80%00%00
b %00%00%03%a1%f3%00%00%00
b %00%00%03%a1%f3%80%00%00
b %00%00%03%a1%f4%00%00%00
b %00%00%03%a1%f4%80%00%00
b %00%00%03%a1%f5%00%00%00
b %00%00%03%a1%f5%80%00%00
b %00%00%03%a1%f6%00%00%00
b %00%00%03%a1%f6%80%00%00
b %00%00%03%a1%f7%00%00%00
b %00%00%03%a1%f7%80%00%00
b %00%00%03%a1%f8%00%00%00
b %00%00%03%a1%f8%80%00%00
b %00%00%03%a1%f9%00%00%00
b %00%00%03%a1%f9%80%00%00
b %00%00%03%a1%fa%00%00%00
b %00%00%03%a1%fa%80%00%00
b %00%00%03%a1%fb%00%00%00
b %00%00%03%a1%fb%80%00%00
b %00%00%03%a1%fc%00%00%00
b %00%00%03%a1%fc%80%00%00
b %00%00%03%a1%fd%00%00%00
b %00%00%03%a1%fd%80%00%00
b %00%00%03%a1%fe%00%00%00
b %00%00%03%a1%fe%80%00%00
b %00%00%03%a1%ff%00%00%00
b %00%00%03%a1%ff%80%00%00
b %00%00%03%a2%00%00%00%00
b %00%00%03%a2%00%80%00%00
b %00%00%03%a2%01%00%00%00
b %00%00%03%a2%01%80%00%00
b %00%00%03%a2%02%00%00%00
b %00%00%03%a2%02%80%00%00
b %00%00%03%a2%03%00%00%00
b %00%00%03%a2%03%80%00%00
b %00%00%03%a2%04%00%00%00
b %00%00%03%a2%04%80%00%00
b %00%00%03%a2%05%00%00%00
b %00%00%03%a2%05%80%00%00
b %00%00%03%a2%06%00%00%00
b %00%00%03%a2%06%80%00%00
b %00%00%03%a2%07%00%00%00
b %00%00%03%a2%07%80%00%00
b %00%00%03%a2%08%00%00%00
b %00%00%03%a2%08%80%00%00
b %00%00%03%a2%09%00%00%00
b %00%00%03%a2%09%80%00%00
b %00%00%03%a2%0a%00%00%00
b %00%00%03%a2%0a%80%00%00
b %00%00%03%a2%0b%00%00%00
b %00%00%03%a2%0b%80%00%00
b %00%00%03%a2%0c%00%00%00
b %00%00%03%a2%0c%80%00%00
b %00%00%03%a2%0d%00%00%00
b %00%00%03%a2%0d%80%00%00
b %00%00%03%a2%0e%00%00%00
b %00%00%03%a2%0e%80%00%00
b %00%00%03%a2%0f%00%00%00
b %00%00%03%a2%0f%80%00%00
b %00%00%03%a2%10%00%00%00
b %00%00%03%a2%10%80%00%00
b %00%00%03%a2%11%00%00%00
b %00%00%03%a2%11%80%00%00
b %00%00%03%a2%12%00%00%00
b %00%00%03%a2%12%80%00%00
b %00%00%03%a2%13%00%00%00
b %00%00%03%a2%13%80%00%00
b %00%00%03%a2%14%00%00%00
b %00%00%03%a2%14%80%00%00
b %00%00%03%a2%15%00%00%00
b %00%00%03%a2%15%80%00%00
b %00%00%03%a2%16%00%00%00
b %00%00%03%a2%16%80%00%00
b %00%00%03%a2%17%00%00%00
b %00%00%03%a2%17%80%00%00
b %00%00%03%a2%18%00%00%00
b %00%00%03%a2%18%80%00%00
b %00%00%03%a2%19%00%00%00
b %00%00%03%a2%19%80%00%00
b %00%00%03%a2%1a%00%00%00
b %00%00%03%a2%1a%80%00%00
b %00%00%03%a2%1b%00%00%00
b %00%00%03%a2%1b%80%00%00
b %00%00%03%a2%1c%00%00%00
b %00%00%03%a2%1c%80%00%00
b %00%00%03%a2%1d%00%00%00
b %00%00%03%a2%1d%80%00%00
b %00%00%03%a2%1e%00%00%00
b %00%00%03%a2%1e%80%00%00
b %00%00%03%a2%1f%00%00%00
b %00%00%03%a2%1f%80%00%00
b %00%00%03%a2%20%00%00%00
b %00%00%03%a2%20%80%00%00
b %00%00%03%a2%21%00%00%00
b %00%00%03%a2%21%80%00%00
b %00%00%03%a2%22%00%00%00
b %00%00%03%a2%22%80%00%00
b %00%00%03%a2%23%00%00%00
b %00%00%03%a2%23%80%00%00
b %00%00%03%a2%24%00%00%00
b %00%00%03%a2%24%80%00%00
b %00%00%03%a2%25%00%00%00
b %00%00%03%a2%25%80%00%00
b %00%00%03%a2%26%00%00%00
b %00%00%03%a2%26%80%00%00
b %00%00%03%a2%27%00%00%00
b %00%00%03%a2%27%80%00%00
b %00%00%03%a2%28%00%00%00
b %00%00%03%a2%28%80%00%00
b %00%00%03%a2%29%00%00%00
b %00%00%03%a2%29%80%00%00
b %00%00%03%a2%2a%00%00%00
b %00%00%03%a2%2a%80%00%00
b %00%00%03%a2%2b%00%00%00
b %00%00%03%a2%2b%80%00%00
b %00%00%03%a2%2c%00%00%00
b %00%00%03%a2%2c%80%00%00
b %00%00%03%a2-%00%00%00
b %00%00%03%a2-%80%00%00
b %00%00%03%a2.%00%00%00
b %00%00%03%a2.%80%00%00
b %00%00%03%a2/%00%00%00
b %00%00%03%a2/%80%00%00
b %00%00%03%a20%00%00%00
b %00%00%03%a20%80%00%00
b %00%00%03%a21%00%00%00
b %00%00%03%a21%80%00%00
b %00%00%03%a22%00%00%00
b %00%00%03%a22%80%00%00
b %00%00%03%a23%00%00%00
b %00%00%03%a23%80%00%00
b %00%00%03%a24%00%00%00
b %00%00%03%a24%80%00%00
b %00%00%03%a25%00%00%00
b %00%00%03%a25%80%00%00
b %00%00%03%a26%00%00%00
b %00%00%03%a26%80%00%00
b %00%00%03%a27%00%00%00
b %00%00%03%a27%80%00%00
b %00%00%03%a28%00%00%00
b %00%00%03%a28%80%00%00
b %00%00%03%a29%00%00%00
b %00%00%03%a29%80%00%00
b %00%00%03%a2%3a%00%00%00
b %00%00%03%a2%3a%80%00%00
b %00%00%03%a2%3b%00%00%00
b %00%00%03%a2%3b%80%00%00
b %00%00%03%a2%3c%00%00%00
b %00%00%03%a2%3c%80%00%00
b %00%00%03%a2%3d%00%00%00
b %00%00%03%a2%3d%80%00%00
b %00%00%03%a2%3e%00%00%00
b %00%00%03%a2%3e%80%00%00
b %00%00%03%a2%3f%00%00%00
b %00%00%03%a2%3f%80%00%00
b %00%00%03%a2%40%00%00%00
b %00%00%03%a2%40%80%00%00
b %00%00%03%a2A%00%00%00
b %00%00%03%a2A%80%00%00
b %00%00%03%a2B%00%00%00
b %00%00%03%a2B%80%00%00
b %00%00%03%a2C%00%00%00
b %00%00%03%a2C%80%00%00
b %00%00%03%a2D%00%00%00
b %00%00%03%a2D%80%00%00
b %00%00%03%a2E%00%00%00
b %00%00%03%a2E%80%00%00
b %00%00%03%a2F%00%00%00
b %00%00%03%a2F%80%00%00
b %00%00%03%a2G%00%00%00
b %00%00%03%a2G%80%00%00
b %00%00%03%a2H%00%00%00
b %00%00%03%a2H%80%00%00
b %00%00%03%a2I%00%00%00
b %00%00%03%a2I%80%00%00
b %00%00%03%a2J%00%00%00
b %00%00%03%a2J%80%00%00
b %00%00%03%a2K%00%00%00
b %00%00%03%a2K%80%00%00
b %00%00%03%a2L%00%00%00
b %00%00%03%a2L%80%00%00
b %00%00%03%a2M%00%00%00
b %00%00%03%a2M%80%00%00
b %00%00%03%a2N%00%00%00
b %00%00%03%a2N%80%00%00
b %00%00%03%a2O%00%00%00
b %00%00%03%a2O%80%00%00
b %00%00%03%a2P%00%00%00
b %00%00%03%a2P%80%00%00
b %00%00%03%a2Q%00%00%00
b %00%00%03%a2Q%80%00%00
b %00%00%03%a2R%00%00%00
b %00%00%03%a2R%80%00%00
b %00%00%03%a2S%00%00%00
b %00%00%03%a2S%80%00%00
b %00%00%03%a2T%00%00%00
b %00%00%03%a2T%80%00%00
b %00%00%03%a2U%00%00%00
b %00%00%03%a2U%80%00%00
b %00%00%03%a2V%00%00%00
b %00%00%03%a2V%80%00%00
b %00%00%03%a2W%00%00%00
b %00%00%03%a2W%80%00%00
b %00%00%03%a2X%00%00%00
b %00%00%03%a2X%80%00%00
b %00%00%03%a2Y%00%00%00
b %00%00%03%a2Y%80%00%00
b %00%00%03%a2Z%00%00%00
b %00%00%03%a2Z%80%00%00
b %00%00%03%a2%5b%00%00%00
b %00%00%03%a2%5b%80%00%00
b %00%00%03%a2%5c%00%00%00
b %00%00%03%a2%5c%80%00%00
b %00%00%03%a2%5d%00%00%00
b %00%00%03%a2%5d%80%00%00
b %00%00%03%a2%5e%00%00%00
b %00%00%03%a2%5e%80%00%00
b %00%00%03%a2_%00%00%00
b %00%00%03%a2_%80%00%00
b %00%00%03%a2%60%00%00%00
b %00%00%03%a2%60%80%00%00
b %00%00%03%a2a%00%00%00
b %00%00%03%a2a%80%00%00
b %00%00%03%a2b%00%00%00
b %00%00%03%a2b%80%00%00
b %00%00%03%a2c%00%00%00
b %00%00%03%a2c%80%00%00
b %00%00%03%a2d%00%00%00
b %00%00%03%a2d%80%00%00
b %00%00%03%a2e%00%00%00
b %00%00%03%a2e%80%00%00
b %00%00%03%a2f%00%00%00
b %00%00%03%a2f%80%00%00
b %00%00%03%a2g%00%00%00
b %00%00%03%a2g%80%00%00
b %00%00%03%a2h%00%00%00
b %00%00%03%a2h%80%00%00
b %00%00%03%a2i%00%00%00
b %00%00%03%a2i%80%00%00
b %00%00%03%a2j%00%00%00
b %00%00%03%a2j%80%00%00
b %00%00%03%a2k%00%00%00
b %00%00%03%a2k%80%00%00
b %00%00%03%a2l%00%00%00
b %00%00%03%a2l%80%00%00
b %00%00%03%a2m%00%00%00
b %00%00%03%a2m%80%00%00
b %00%00%03%a2n%00%00%00
b %00%00%03%a2n%80%00%00
b %00%00%03%a2o%00%00%00
b %00%00%03%a2o%80%00%00
b %00%00%03%a2p%00%00%00
b %00%00%03%a2p%80%00%00
b %00%00%03%a2q%00%00%00
b %00%00%03%a2q%80%00%00
b %00%00%03%a2r%00%00%00
b %00%00%03%a2r%80%00%00
b %00%00%03%a2s%00%00%00
b %00%00%03%a2s%80%00%00
b %00%00%03%a2t%00%00%00
b %00%00%03%a2t%80%00%00
b %00%00%03%a2u%00%00%00
b %00%00%03%a2u%80%00%00
b %00%00%03%a2v%00%00%00
b %00%00%03%a2v%80%00%00
b %00%00%03%a2w%00%00%00
b %00%00%03%a2w%80%00%00
b %00%00%03%a2x%00%00%00
b %00%00%03%a2x%80%00%00
b %00%00%03%a2y%00%00%00
b %00%00%03%a2y%80%00%00
b %00%00%03%a2z%00%00%00
b %00%00%03%a2z%80%00%00
b %00%00%03%a2%7b%00%00%00
b %00%00%03%a2%7b%80%00%00
b %00%00%03%a2%7c%00%00%00
b %00%00%03%a2%7c%80%00%00
b %00%00%03%a2%7d%00%00%00
b %00%00%03%a2%7d%80%00%00
b %00%00%03%a2~%00%00%00
b %00%00%03%a2~%80%00%00
b %00%00%03%a2%7f%00%00%00
b %00%00%03%a2%7f%80%00%00
b %00%00%03%a2%80%00%00%00
b %00%00%03%a2%80%80%00%00
b %00%00%03%a2%81%00%00%00
b %00%00%03%a2%81%80%00%00
b %00%00%03%a2%82%00%00%00
b %00%00%03%a2%82%80%00%00
b %00%00%03%a2%83%00%00%00
b %00%00%03%a2%83%80%00%00
b %00%00%03%a2%84%00%00%00
b %00%00%03%a2%84%80%00%00
b %00%00%03%a2%85%00%00%00
b %00%00%03%a2%85%80%00%00
b %00%00%03%a2%86%00%00%00
b %00%00%03%a2%86%80%00%00
b %00%00%03%a2%87%00%00%00
b %00%00%03%a2%87%80%00%00
b %00%00%03%a2%88%00%00%00
b %00%00%03%a2%88%80%00%00
b %00%00%03%a2%89%00%00%00
b %00%00%03%a2%89%80%00%00
b %00%00%03%a2%8a%00%00%00
b %00%00%03%a2%8a%80%00%00
b %00%00%03%a2%8b%00%00%00
b %00%00%03%a2%8b%80%00%00
b %00%00%03%a2%8c%00%00%00
b %00%00%03%a2%8c%80%00%00
b %00%00%03%a2%8d%00%00%00
b %00%00%03%a2%8d%80%00%00
b %00%00%03%a2%8e%00%00%00
b %00%00%03%a2%8e%80%00%00
b %00%00%03%a2%8f%00%00%00
b %00%00%03%a2%8f%80%00%00
b %00%00%03%a2%90%00%00%00
b %00%00%03%a2%90%80%00%00
b %00%00%03%a2%91%00%00%00
b %00%00%03%a2%91%80%00%00
b %00%00%03%a2%92%00%00%00
b %00%00%03%a2%92%80%00%00
b %00%00%03%a2%93%00%00%00
b %00%00%03%a2%93%80%00%00
b %00%00%03%a2%94%00%00%00
b %00%00%03%a2%94%80%00%00
b %00%00%03%a2%95%00%00%00
b %00%00%03%a2%95%80%00%00
b %00%00%03%a2%96%00%00%00
b %00%00%03%a2%96%80%00%00
b %00%00%03%a2%97%00%00%00
b %00%00%03%a2%97%80%00%00
b %00%00%03%a2%98%00%00%00
b %00%00%03%a2%98%80%00%00
b %00%00%03%a2%99%00%00%00
b %00%00%03%a2%99%80%00%00
b %00%00%03%a2%9a%00%00%00
b %00%00%03%a2%9a%80%00%00
b %00%00%03%a2%9b%00%00%00
b %00%00%03%a2%9b%80%00%00
b %00%00%03%a2%9c%00%00%00
b %00%00%03%a2%9c%80%00%00
b %00%00%03%a2%9d%00%00%00
b %00%00%03%a2%9d%80%00%00
b %00%00%03%a2%9e%00%00%00
b %00%00%03%a2%9e%80%00%00
b %00%00%03%a2%9f%00%00%00
b %00%00%03%a2%9f%80%00%00
b %00%00%03%a2%a0%00%00%00
b %00%00%03%a2%a0%80%00%00
b %00%00%03%a2%a1%00%00%00
b %00%00%03%a2%a1%80%00%00
b %00%00%03%a2%a2%00%00%00
b %00%00%03%a2%a2%80%00%00
b %00%00%03%a2%a3%00%00%00
b %00%00%03%a2%a3%80%00%00
b %00%00%03%a2%a4%00%00%00
b %00%00%03%a2%a4%80%00%00
b %00%00%03%a2%a5%00%00%00
b %00%00%03%a2%a5%80%00%00
b %00%00%03%a2%a6%00%00%00
b %00%00%03%a2%a6%80%00%00
b %00%00%03%a2%a7%00%00%00
b %00%00%03%a2%a7%80%00%00
b %00%00%03%a2%a8%00%00%00
b %00%00%03%a2%a8%80%00%00
b %00%00%03%a2%a9%00%00%00
b %00%00%03%a2%a9%80%00%00
b %00%00%03%a2%aa%00%00%00
b %00%00%03%a2%aa%80%00%00
b %00%00%03%a2%ab%00%00%00
b %00%00%03%a2%ab%80%00%00
b %00%00%03%a2%ac%00%00%00
b %00%00%03%a2%ac%80%00%00
b %00%00%03%a2%ad%00%00%00
b %00%00%03%a2%ad%80%00%00
b %00%00%03%a2%ae%00%00%00
b %00%00%03%a2%ae%80%00%00
b %00%00%03%a2%af%00%00%00
b %00%00%03%a2%af%80%00%00
b %00%00%03%a2%b0%00%00%00
b %00%00%03%a2%b0%80%00%00
b %00%00%03%a2%b1%00%00%00
b %00%00%03%a2%b1%80%00%00
b %00%00%03%a2%b2%00%00%00
b %00%00%03%a2%b2%80%00%00
b %00%00%03%a2%b3%00%00%00
b %00%00%03%a2%b3%80%00%00
b %00%00%03%a2%b4%00%00%00
b %00%00%03%a2%b4%80%00%00
b %00%00%03%a2%b5%00%00%00
b %00%00%03%a2%b5%80%00%00
b %00%00%03%a2%b6%00%00%00
b %00%00%03%a2%b6%80%00%00
b %00%00%03%a2%b7%00%00%00
b %00%00%03%a2%b7%80%00%00
b %00%00%03%a2%b8%00%00%00
b %00%00%03%a2%b8%80%00%00
b %00%00%03%a2%b9%00%00%00
b %00%00%03%a2%b9%80%00%00
b %00%00%03%a2%ba%00%00%00
b %00%00%03%a2%ba%80%00%00
b %00%00%03%a2%bb%00%00%00
b %00%00%03%a2%bb%80%00%00
b %00%00%03%a2%bc%00%00%00
b %00%00%03%a2%bc%80%00%00
b %00%00%03%a2%bd%00%00%00
b %00%00%03%a2%bd%80%00%00
b %00%00%03%a2%be%00%00%00
b %00%00%03%a2%be%80%00%00
b %00%00%03%a2%bf%00%00%00
b %00%00%03%a2%bf%80%00%00
b %00%00%03%a2%c0%00%00%00
b %00%00%03%a2%c0%80%00%00
b %00%00%03%a2%c1%00%00%00
b %00%00%03%a2%c1%80%00%00
b %00%00%03%a2%c2%00%00%00
b %00%00%03%a2%c2%80%00%00
b %00%00%03%a2%c3%00%00%00
b %00%00%03%a2%c3%80%00%00
b %00%00%03%a2%c4%00%00%00
b %00%00%03%a2%c4%80%00%00
b %00%00%03%a2%c5%00%00%00
b %00%00%03%a2%c5%80%00%00
b %00%00%03%a2%c6%00%00%00
b %00%00%03%a2%c6%80%00%00
b %00%00%03%a2%c7%00%00%00
b %00%00%03%a2%c7%80%00%00
b %00%00%03%a2%c8%00%00%00
b %00%00%03%a2%c8%80%00%00
b %00%00%03%a2%c9%00%00%00
b %00%00%03%a2%c9%80%00%00
b %00%00%03%a2%ca%00%00%00
b %00%00%03%a2%ca%80%00%00
b %00%00%03%a2%cb%00%00%00
b %00%00%03%a2%cb%80%00%00
b %00%00%03%a2%cc%00%00%00
b %00%00%03%a2%cc%80%00%00
b %00%00%03%a2%cd%00%00%00
b %00%00%03%a2%cd%80%00%00
b %00%00%03%a2%ce%00%00%00
b %00%00%03%a2%ce%80%00%00
b %00%00%03%a2%cf%00%00%00
b %00%00%03%a2%cf%80%00%00
b %00%00%03%a2%d0%00%00%00
b %00%00%03%a2%d0%80%00%00
b %00%00%03%a2%d1%00%00%00
b %00%00%03%a2%d1%80%00%00
b %00%00%03%a2%d2%00%00%00
b %00%00%03%a2%d2%80%00%00
b %00%00%03%a2%d3%00%00%00
b %00%00%03%a2%d3%80%00%00
b %00%00%03%a2%d4%00%00%00
b %00%00%03%a2%d4%80%00%00
b %00%00%03%a2%d5%00%00%00
b %00%00%03%a2%d5%80%00%00
b %00%00%03%a2%d6%00%00%00
b %00%00%03%a2%d6%80%00%00
b %00%00%03%a2%d7%00%00%00
b %00%00%03%a2%d7%80%00%00
b %00%00%03%a2%d8%00%00%00
b %00%00%03%a2%d8%80%00%00
b %00%00%03%a2%d9%00%00%00
b %00%00%03%a2%d9%80%00%00
b %00%00%03%a2%da%00%00%00
b %00%00%03%a2%da%80%00%00
b %00%00%03%a2%db%00%00%00
b %00%00%03%a2%db%80%00%00
b %00%00%03%a2%dc%00%00%00
b %00%00%03%a2%dc%80%00%00
b %00%00%03%a2%dd%00%00%00
b %00%00%03%a2%dd%80%00%00
b %00%00%03%a2%de%00%00%00
b %00%00%03%a2%de%80%00%00
b %00%00%03%a2%df%00%00%00
b %00%00%03%a2%df%80%00%00
b %00%00%03%a2%e0%00%00%00
b %00%00%03%a2%e0%80%00%00
b %00%00%03%a2%e1%00%00%00
b %00%00%03%a2%e1%80%00%00
b %00%00%03%a2%e2%00%00%00
b %00%00%03%a2%e2%80%00%00
b %00%00%03%a2%e3%00%00%00
b %00%00%03%a2%e3%80%00%00
b %00%00%03%a2%e4%00%00%00
b %00%00%03%a2%e4%80%00%00
b %00%00%03%a2%e5%00%00%00
b %00%00%03%a2%e5%80%00%00
b %00%00%03%a2%e6%00%00%00
b %00%00%03%a2%e6%80%00%00
b %00%00%03%a2%e7%00%00%00
b %00%00%03%a2%e7%80%00%00
b %00%00%03%a2%e8%00%00%00
b %00%00%03%a2%e8%80%00%00
b %00%00%03%a2%e9%00%00%00
b %00%00%03%a2%e9%80%00%00
b %00%00%03%a2%ea%00%00%00
b %00%00%03%a2%ea%80%00%00
b %00%00%03%a2%eb%00%00%00
b %00%00%03%a2%eb%80%00%00
b %00%00%03%a2%ec%00%00%00
b %00%00%03%a2%ec%80%00%00
b %00%00%03%a2%ed%00%00%00
b %00%00%03%a2%ed%80%00%00
b %00%00%03%a2%ee%00%00%00
b %00%00%03%a2%ee%80%00%00
b %00%00%03%a2%ef%00%00%00
b %00%00%03%a2%ef%80%00%00
b %00%00%03%a2%f0%00%00%00
b %00%00%03%a2%f0%80%00%00
b %00%00%03%a2%f1%00%00%00
b %00%00%03%a2%f1%80%00%00
b %00%00%03%a2%f2%00%00%00
b %00%00%03%a2%f2%80%00%00
b %00%00%03%a2%f3%00%00%00
b %00%00%03%a2%f3%80%00%00
b %00%00%03%a2%f4%00%00%00
b %00%00%03%a2%f4%80%00%00
b %00%00%03%a2%f5%00%00%00
b %00%00%03%a2%f5%80%00%00
b %00%00%03%a2%f6%00%00%00
b %00%00%03%a2%f6%80%00%00
b %00%00%03%a2%f7%00%00%00
b %00%00%03%a2%f7%80%00%00
b %00%00%03%a2%f8%00%00%00
b %00%00%03%a2%f8%80%00%00
b %00%00%03%a2%f9%00%00%00
b %00%00%03%a2%f9%80%00%00
b %00%00%03%a2%fa%00%00%00
b %00%00%03%a2%fa%80%00%00
b %00%00%03%a2%fb%00%00%00
b %00%00%03%a2%fb%80%00%00
b %00%00%03%a2%fc%00%00%00
b %00%00%03%a2%fc%80%00%00
b %00%00%03%a2%fd%00%00%00
b %00%00%03%a2%fd%80%00%00
b %00%00%03%a2%fe%00%00%00
b %00%00%03%a2%fe%80%00%00
b %00%00%03%a2%ff%00%00%00
b %00%00%03%a2%ff%80%00%00
b %00%00%03%a3%00%00%00%00
b %00%00%03%a3%00%80%00%00
b %00%00%03%a3%01%00%00%00
b %00%00%03%a3%01%80%00%00
b %00%00%03%a3%02%00%00%00
b %00%00%03%a3%02%80%00%00
b %00%00%03%a3%03%00%00%00
b %00%00%03%a3%03%80%00%00
b %00%00%03%a3%04%00%00%00
b %00%00%03%a3%04%80%00%00
b %00%00%03%a3%05%00%00%00
b %00%00%03%a3%05%80%00%00
b %00%00%03%a3%06%00%00%00
b %00%00%03%a3%06%80%00%00
b %00%00%03%a3%07%00%00%00
b %00%00%03%a3%07%80%00%00
b %00%00%03%a3%08%00%00%00
b %00%00%03%a3%08%80%00%00
b %00%00%03%a3%09%00%00%00
b %00%00%03%a3%09%80%00%00
b %00%00%03%a3%0a%00%00%00
b %00%00%03%a3%0a%80%00%00
b %00%00%03%a3%0b%00%00%00
b %00%00%03%a3%0b%80%00%00
b %00%00%03%a3%0c%00%00%00
b %00%00%03%a3%0c%80%00%00
b %00%00%03%a3%0d%00%00%00
b %00%00%03%a3%0d%80%00%00
b %00%00%03%a3%0e%00%00%00
b %00%00%03%a3%0e%80%00%00
b %00%00%03%a3%0f%00%00%00
b %00%00%03%a3%0f%80%00%00
b %00%00%03%a3%10%00%00%00
b %00%00%03%a3%10%80%00%00
b %00%00%03%a3%11%00%00%00
b %00%00%03%a3%11%80%00%00
b %00%00%03%a3%12%00%00%00
b %00%00%03%a3%12%80%00%00
b %00%00%03%a3%13%00%00%00
b %00%00%03%a3%13%80%00%00
b %00%00%03%a3%14%00%00%00
b %00%00%03%a3%14%80%00%00
b %00%00%03%a3%15%00%00%00
b %00%00%03%a3%15%80%00%00
b %00%00%03%a3%16%00%00%00
b %00%00%03%a3%16%80%00%00
b %00%00%03%a3%17%00%00%00
b %00%00%03%a3%17%80%00%00
b %00%00%03%a3%18%00%00%00
b %00%00%03%a3%18%80%00%00
b %00%00%03%a3%19%00%00%00
b %00%00%03%a3%19%80%00%00
b %00%00%03%a3%1a%00%00%00
b %00%00%03%a3%1a%80%00%00
b %00%00%03%a3%1b%00%00%00
b %00%00%03%a3%1b%80%00%00
b %00%00%03%a3%1c%00%00%00
b %00%00%03%a3%1c%80%00%00
b %00%00%03%a3%1d%00%00%00
b %00%00%03%a3%1d%80%00%00
b %00%00%03%a3%1e%00%00%00
b %00%00%03%a3%1e%80%00%00
b %00%00%03%a3%1f%00%00%00
b %00%00%03%a3%1f%80%00%00
b %00%00%03%a3%20%00%00%00
b %00%00%03%a3%20%80%00%00
b %00%00%03%a3%21%00%00%00
b %00%00%03%a3%21%80%00%00
b %00%00%03%a3%22%00%00%00
b %00%00%03%a3%22%80%00%00
b %00%00%03%a3%23%00%00%00
b %00%00%03%a3%23%80%00%00
b %00%00%03%a3%24%00%00%00
b %00%00%03%a3%24%80%00%00
b %00%00%03%a3%25%00%00%00
b %00%00%03%a3%25%80%00%00
b %00%00%03%a3%26%00%00%00
b %00%00%03%a3%26%80%00%00
b %00%00%03%a3%27%00%00%00
b %00%00%03%a3%27%80%00%00
b %00%00%03%a3%28%00%00%00
b %00%00%03%a3%28%80%00%00
b %00%00%03%a3%29%00%00%00
b %00%00%03%a3%29%80%00%00
b %00%00%03%a3%2a%00%00%00
b %00%00%03%a3%2a%80%00%00
b %00%00%03%a3%2b%00%00%00
b %00%00%03%a3%2b%80%00%00
b %00%00%03%a3%2c%00%00%00
b %00%00%03%a3%2c%80%00%00
b %00%00%03%a3-%00%00%00
b %00%00%03%a3-%80%00%00
b %00%00%03%a3.%00%00%00
b %00%00%03%a3.%80%00%00
b %00%00%03%a3/%00%00%00
b %00%00%03%a3/%80%00%00
b %00%00%03%a30%00%00%00
b %00%00%03%a30%80%00%00
b %00%00%03%a31%00%00%00
b %00%00%03%a31%80%00%00
b %00%00%03%a32%00%00%00
b %00%00%03%a32%80%00%00
b %00%00%03%a33%00%00%00
b %00%00%03%a33%80%00%00
b %00%00%03%a34%00%00%00
b %00%00%03%a34%80%00%00
b %00%00%03%a35%00%00%00
b %00%00%03%a35%80%00%00
b %00%00%03%a36%00%00%00
b %00%00%03%a36%80%00%00
b %00%00%03%a37%00%00%00
b %00%00%03%a37%80%00%00
b %00%00%03%a38%00%00%00
b %00%00%03%a38%80%00%00
b %00%00%03%a39%00%00%00
b %00%00%03%a39%80%00%00
b %00%00%03%a3%3a%00%00%00
b %00%00%03%a3%3a%80%00%00
b %00%00%03%a3%3b%00%00%00
b %00%00%03%a3%3b%80%00%00
b %00%00%03%a3%3c%00%00%00
b %00%00%03%a3%3c%80%00%00
b %00%00%03%a3%3d%00%00%00
b %00%00%03%a3%3d%80%00%00
b %00%00%03%a3%3e%00%00%00
b %00%00%03%a3%3e%80%00%00
b %00%00%03%a3%3f%00%00%00
b %00%00%03%a3%3f%80%00%00
b %00%00%03%a3%40%00%00%00
b %00%00%03%a3%40%80%00%00
b %00%00%03%a3A%00%00%00
b %00%00%03%a3A%80%00%00
b %00%00%03%a3B%00%00%00
b %00%00%03%a3B%80%00%00
b %00%00%03%a3C%00%00%00
b %00%00%03%a3C%80%00%00
b %00%00%03%a3D%00%00%00
b %00%00%03%a3D%80%00%00
b %00%00%03%a3E%00%00%00
b %00%00%03%a3E%80%00%00
b %00%00%03%a3F%00%00%00
b %00%00%03%a3F%80%00%00
b %00%00%03%a3G%00%00%00
b %00%00%03%a3G%80%00%00
b %00%00%03%a3H%00%00%00
b %00%00%03%a3H%80%00%00
b %00%00%03%a3I%00%00%00
b %00%00%03%a3I%80%00%00
b %00%00%03%a3J%00%00%00
b %00%00%03%a3J%80%00%00
b %00%00%03%a3K%00%00%00
b %00%00%03%a3K%80%00%00
b %00%00%03%a3L%00%00%00
b %00%00%03%a3L%80%00%00
b %00%00%03%a3M%00%00%00
b %00%00%03%a3M%80%00%00
b %00%00%03%a3N%00%00%00
b %00%00%03%a3N%80%00%00
b %00%00%03%a3O%00%00%00
b %00%00%03%a3O%80%00%00
b %00%00%03%a3P%00%00%00
b %00%00%03%a3P%80%00%00
b %00%00%03%a3Q%00%00%00
b %00%00%03%a3Q%80%00%00
b %00%00%03%a3R%00%00%00
b %00%00%03%a3R%80%00%00
b %00%00%03%a3S%00%00%00
b %00%00%03%a3S%80%00%00
b %00%00%03%a3T%00%00%00
b %00%00%03%a3T%80%00%00
b %00%00%03%a3U%00%00%00
b %00%00%03%a3U%80%00%00
b %00%00%03%a3V%00%00%00
b %00%00%03%a3V%80%00%00
b %00%00%03%a3W%00%00%00
b %00%00%03%a3W%80%00%00
b %00%00%03%a3X%00%00%00
b %00%00%03%a3X%80%00%00
b %00%00%03%a3Y%00%00%00
b %00%00%03%a3Y%80%00%00
b %00%00%03%a3Z%00%00%00
b %00%00%03%a3Z%80%00%00
b %00%00%03%a3%5b%00%00%00
b %00%00%03%a3%5b%80%00%00
b %00%00%03%a3%5c%00%00%00
b %00%00%03%a3%5c%80%00%00
b %00%00%03%a3%5d%00%00%00
b %00%00%03%a3%5d%80%00%00
b %00%00%03%a3%5e%00%00%00
b %00%00%03%a3%5e%80%00%00
b %00%00%03%a3_%00%00%00
b %00%00%03%a3_%80%00%00
b %00%00%03%a3%60%00%00%00
b %00%00%03%a3%60%80%00%00
b %00%00%03%a3a%00%00%00
b %00%00%03%a3a%80%00%00
b %00%00%03%a3b%00%00%00
b %00%00%03%a3b%80%00%00
b %00%00%03%a3c%00%00%00
b %00%00%03%a3c%80%00%00
b %00%00%03%a3d%00%00%00
b %00%00%03%a3d%80%00%00
b %00%00%03%a3e%00%00%00
b %00%00%03%a3e%80%00%00
b %00%00%03%a3f%00%00%00
b %00%00%03%a3f%80%00%00
b %00%00%03%a3g%00%00%00
b %00%00%03%a3g%80%00%00
b %00%00%03%a3h%00%00%00
b %00%00%03%a3h%80%00%00
b %00%00%03%a3i%00%00%00
b %00%00%03%a3i%80%00%00
b %00%00%03%a3j%00%00%00
b %00%00%03%a3j%80%00%00
b %00%00%03%a3k%00%00%00
b %00%00%03%a3k%80%00%00
b %00%00%03%a3l%00%00%00
b %00%00%03%a3l%80%00%00
b %00%00%03%a3m%00%00%00
b %00%00%03%a3m%80%00%00
b %00%00%03%a3n%00%00%00
b %00%00%03%a3n%80%00%00
b %00%00%03%a3o%00%00%00
b %00%00%03%a3o%80%00%00
b %00%00%03%a3p%00%00%00
b %00%00%03%a3p%80%00%00
b %00%00%03%a3q%00%00%00
b %00%00%03%a3q%80%00%00
b %00%00%03%a3r%00%00%00
b %00%00%03%a3r%80%00%00
b %00%00%03%a3s%00%00%00
b %00%00%03%a3s%80%00%00
b %00%00%03%a3t%00%00%00
b %00%00%03%a3t%80%00%00
b %00%00%03%a3u%00%00%00
b %00%00%03%a3u%80%00%00
b %00%00%03%a3v%00%00%00
b %00%00%03%a3v%80%00%00
b %00%00%03%a3w%00%00%00
b %00%00%03%a3w%80%00%00
b %00%00%03%a3x%00%00%00
b %00%00%03%a3x%80%00%00
b %00%00%03%a3y%00%00%00
b %00%00%03%a3y%80%00%00
b %00%00%03%a3z%00%00%00
b %00%00%03%a3z%80%00%00
b %00%00%03%a3%7b%00%00%00
b %00%00%03%a3%7b%80%00%00
b %00%00%03%a3%7c%00%00%00
b %00%00%03%a3%7c%80%00%00
b %00%00%03%a3%7d%00%00%00
b %00%00%03%a3%7d%80%00%00
b %00%00%03%a3~%00%00%00
b %00%00%03%a3~%80%00%00
b %00%00%03%a3%7f%00%00%00
b %00%00%03%a3%7f%80%00%00
b %00%00%03%a3%80%00%00%00
b %00%00%03%a3%80%80%00%00
b %00%00%03%a3%81%00%00%00
root@unil0047:/mnt/ceph# time ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-49 list > keys-bs
2019-06-13 08:13:45.706 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _mount path /var/lib/ceph/osd/ceph-49
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0700 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0700 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
2019-06-13 08:13:45.718 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block.db type kernel
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) open path /var/lib/ceph/osd/ceph-49/block.db
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) open size 20975714304 (0x4e2400000, 20 GiB) block_size 4096 (4 KiB) non-rotational discard supported
2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-49/block.db size 20 GiB
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-49/block size 3.6 TiB
2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs mount
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
2019-06-13 08:13:45.810 7ff348fa70c0 1 rocksdb: do_open column families: [default]
tcmalloc: large alloc 2145263616 bytes == 0x556c1ba46000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d1a19f1 0x556b8cf6cad1 0x556b8cec77e7 0x556b8cee14c8 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 4290519040 bytes == 0x556c9b828000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d1a19f1 0x556b8cf6cad1 0x556b8cec77e7 0x556b8cee14c8 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 8581029888 bytes == 0x556d9bce0000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d1a19f1 0x556b8cf6cad1 0x556b8cec77e7 0x556b8cee14c8 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 17162051584 bytes == 0x556f9bc62000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d1a19f1 0x556b8cf6cad1 0x556b8cec77e7 0x556b8cee14c8 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 13559291904 bytes == 0x55739bb64000 @ 0x7ff348b63887 0x556b8ce927fd 0x556b8d243749 0x556b8d18d9d5 0x556b8d18f6b5 0x556b8d1a19f1 0x556b8cf6cad1 0x556b8cec77e7 0x556b8cee14c8 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
2019-06-13 08:15:48.650 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
2019-06-13 08:15:48.666 7ff348fa70c0 1 freelist init
2019-06-13 08:15:48.762 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_alloc opening allocation metadata
2019-06-13 08:15:49.354 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_alloc loaded 2.5 TiB in 252916 extents
2019-06-13 08:16:04.211 7ff348fa70c0 1 bluefs umount
2019-06-13 08:16:04.211 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e600 shutdown
2019-06-13 08:16:04.211 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e700 shutdown
2019-06-13 08:16:04.263 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) close
2019-06-13 08:16:04.663 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) close
2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block.db type kernel
2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) open path /var/lib/ceph/osd/ceph-49/block.db
2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) open size 20975714304 (0x4e2400000, 20 GiB) block_size 4096 (4 KiB) non-rotational discard supported
2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-49/block.db size 20 GiB
2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-49/block size 3.6 TiB
2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs mount
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
2019-06-13 08:16:04.903 7ff348fa70c0 1 rocksdb: do_open column families: [default]
tcmalloc: large alloc 2145263616 bytes == 0x556bdbbca000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d190a80 0x556b8d191ff4 0x556b8cf6bc9f 0x556b8cec5f44 0x556b8cee15fe 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 4290519040 bytes == 0x556d9bce0000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d190a80 0x556b8d191ff4 0x556b8cf6bc9f 0x556b8cec5f44 0x556b8cee15fe 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 8581029888 bytes == 0x55739bb64000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d190a80 0x556b8d191ff4 0x556b8cf6bc9f 0x556b8cec5f44 0x556b8cee15fe 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 17162051584 bytes == 0x556f9bc62000 @ 0x7ff348b63887 0x556b8cf2e6b9 0x556b8cf2e833 0x556b8d1e4e9c 0x556b8d18d992 0x556b8d18f6b5 0x556b8d190a80 0x556b8d191ff4 0x556b8cf6bc9f 0x556b8cec5f44 0x556b8cee15fe 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
tcmalloc: large alloc 13559291904 bytes == 0x55739bb64000 @ 0x7ff348b63887 0x556b8ce927fd 0x556b8d243749 0x556b8d18d9d5 0x556b8d18f6b5 0x556b8d190a80 0x556b8d191ff4 0x556b8cf6bc9f 0x556b8cec5f44 0x556b8cee15fe 0x556b8cf1e0aa 0x556b8ce8e1b2 0x556b8ce90e56 0x556b8cdb5f4c 0x7ff33d86ab97 0x556b8ce8e03a
2019-06-13 08:19:25.323 7ff348fa70c0 -1 rocksdb: Corruption: file is too short (6139497190 bytes) to be an sstabledb/079728.sst
2019-06-13 08:19:25.323 7ff348fa70c0 -1 bluestore(/var/lib/ceph/osd/ceph-49) _open_db erroring opening db:
2019-06-13 08:19:25.323 7ff348fa70c0 1 bluefs umount
2019-06-13 08:19:25.323 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e700 shutdown
2019-06-13 08:19:25.323 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e600 shutdown
2019-06-13 08:19:25.335 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) close
2019-06-13 08:19:25.891 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) close
2019-06-13 08:19:26.199 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6eb00 shutdown
2019-06-13 08:19:26.279 7ff348fa70c0 1 freelist shutdown
*** Caught signal (Segmentation fault) **
in thread 7ff348fa70c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7ff33e997890]
2: (StoreTool::traverse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::ostream*)+0x40) [0x556b8ce90120]
3: (main()+0x492) [0x556b8cdb6142]
4: (__libc_start_main()+0xe7) [0x7ff33d86ab97]
5: (_start()+0x2a) [0x556b8ce8e03a]
2019-06-13 08:19:26.283 7ff348fa70c0 -1 *** Caught signal (Segmentation fault) **
in thread 7ff348fa70c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7ff33e997890]
2: (StoreTool::traverse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::ostream*)+0x40) [0x556b8ce90120]
3: (main()+0x492) [0x556b8cdb6142]
4: (__libc_start_main()+0xe7) [0x7ff33d86ab97]
5: (_start()+0x2a) [0x556b8ce8e03a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-93> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command assert hook 0x556b8fd1e130
-92> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command abort hook 0x556b8fd1e130
-91> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perfcounters_dump hook 0x556b8fd1e130
-90> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command 1 hook 0x556b8fd1e130
-89> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf dump hook 0x556b8fd1e130
-88> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perfcounters_schema hook 0x556b8fd1e130
-87> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf histogram dump hook 0x556b8fd1e130
-86> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command 2 hook 0x556b8fd1e130
-85> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf schema hook 0x556b8fd1e130
-84> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf histogram schema hook 0x556b8fd1e130
-83> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf reset hook 0x556b8fd1e130
-82> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config show hook 0x556b8fd1e130
-81> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config help hook 0x556b8fd1e130
-80> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config set hook 0x556b8fd1e130
-79> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config unset hook 0x556b8fd1e130
-78> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config get hook 0x556b8fd1e130
-77> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config diff hook 0x556b8fd1e130
-76> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config diff get hook 0x556b8fd1e130
-75> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command log flush hook 0x556b8fd1e130
-74> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command log dump hook 0x556b8fd1e130
-73> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command log reopen hook 0x556b8fd1e130
-72> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command dump_mempools hook 0x556b90b98068
-71> 2019-06-13 08:13:45.706 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _mount path /var/lib/ceph/osd/ceph-49
-70> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
-69> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0700 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
-68> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0700 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
-67> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
-66> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block.db type kernel
-65> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) open path /var/lib/ceph/osd/ceph-49/block.db
-64> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) open size 20975714304 (0x4e2400000, 20 GiB) block_size 4096 (4 KiB) non-rotational discard supported
-63> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-49/block.db size 20 GiB
-62> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
-61> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
-60> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
-59> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-49/block size 3.6 TiB
-58> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs mount
-57> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-56> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-55> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-54> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-53> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-52> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-51> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-50> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-49> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-48> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-47> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-46> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-45> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-44> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-43> 2019-06-13 08:13:45.810 7ff348fa70c0 1 rocksdb: do_open column families: [default]
-42> 2019-06-13 08:15:48.650 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
-41> 2019-06-13 08:15:48.666 7ff348fa70c0 1 freelist init
-40> 2019-06-13 08:15:48.762 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_alloc opening allocation metadata
-39> 2019-06-13 08:15:49.354 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_alloc loaded 2.5 TiB in 252916 extents
-38> 2019-06-13 08:16:04.211 7ff348fa70c0 1 bluefs umount
-37> 2019-06-13 08:16:04.211 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e600 shutdown
-36> 2019-06-13 08:16:04.211 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e700 shutdown
-35> 2019-06-13 08:16:04.263 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) close
-34> 2019-06-13 08:16:04.663 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) close
-33> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block.db type kernel
-32> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) open path /var/lib/ceph/osd/ceph-49/block.db
-31> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) open size 20975714304 (0x4e2400000, 20 GiB) block_size 4096 (4 KiB) non-rotational discard supported
-30> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-49/block.db size 20 GiB
-29> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
-28> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
-27> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
-26> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-49/block size 3.6 TiB
-25> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs mount
-24> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-23> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-22> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-21> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-20> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-19> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-18> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-17> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-16> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-15> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-14> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-13> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-12> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-11> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-10> 2019-06-13 08:16:04.903 7ff348fa70c0 1 rocksdb: do_open column families: [default]
-9> 2019-06-13 08:19:25.323 7ff348fa70c0 -1 rocksdb: Corruption: file is too short (6139497190 bytes) to be an sstabledb/079728.sst
-8> 2019-06-13 08:19:25.323 7ff348fa70c0 -1 bluestore(/var/lib/ceph/osd/ceph-49) _open_db erroring opening db:
-7> 2019-06-13 08:19:25.323 7ff348fa70c0 1 bluefs umount
-6> 2019-06-13 08:19:25.323 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e700 shutdown
-5> 2019-06-13 08:19:25.323 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e600 shutdown
-4> 2019-06-13 08:19:25.335 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) close
-3> 2019-06-13 08:19:25.891 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) close
-2> 2019-06-13 08:19:26.199 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6eb00 shutdown
-1> 2019-06-13 08:19:26.279 7ff348fa70c0 1 freelist shutdown
0> 2019-06-13 08:19:26.283 7ff348fa70c0 -1 *** Caught signal (Segmentation fault) **
in thread 7ff348fa70c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7ff33e997890]
2: (StoreTool::traverse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::ostream*)+0x40) [0x556b8ce90120]
3: (main()+0x492) [0x556b8cdb6142]
4: (__libc_start_main()+0xe7) [0x7ff33d86ab97]
5: (_start()+0x2a) [0x556b8ce8e03a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file
--- end dump of recent events ---
--- begin dump of recent events ---
-93> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command assert hook 0x556b8fd1e130
-92> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command abort hook 0x556b8fd1e130
-91> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perfcounters_dump hook 0x556b8fd1e130
-90> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command 1 hook 0x556b8fd1e130
-89> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf dump hook 0x556b8fd1e130
-88> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perfcounters_schema hook 0x556b8fd1e130
-87> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf histogram dump hook 0x556b8fd1e130
-86> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command 2 hook 0x556b8fd1e130
-85> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf schema hook 0x556b8fd1e130
-84> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf histogram schema hook 0x556b8fd1e130
-83> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command perf reset hook 0x556b8fd1e130
-82> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config show hook 0x556b8fd1e130
-81> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config help hook 0x556b8fd1e130
-80> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config set hook 0x556b8fd1e130
-79> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config unset hook 0x556b8fd1e130
-78> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config get hook 0x556b8fd1e130
-77> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config diff hook 0x556b8fd1e130
-76> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command config diff get hook 0x556b8fd1e130
-75> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command log flush hook 0x556b8fd1e130
-74> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command log dump hook 0x556b8fd1e130
-73> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command log reopen hook 0x556b8fd1e130
-72> 2019-06-13 08:13:45.698 7ff348fa70c0 5 asok(0x556b90094000) register_command dump_mempools hook 0x556b90b98068
-71> 2019-06-13 08:13:45.706 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _mount path /var/lib/ceph/osd/ceph-49
-70> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
-69> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0700 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
-68> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0700 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
-67> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
-66> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block.db type kernel
-65> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) open path /var/lib/ceph/osd/ceph-49/block.db
-64> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) open size 20975714304 (0x4e2400000, 20 GiB) block_size 4096 (4 KiB) non-rotational discard supported
-63> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-49/block.db size 20 GiB
-62> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
-61> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
-60> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
-59> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-49/block size 3.6 TiB
-58> 2019-06-13 08:13:45.718 7ff348fa70c0 1 bluefs mount
-57> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-56> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-55> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-54> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-53> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-52> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-51> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-50> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-49> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-48> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-47> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-46> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-45> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-44> 2019-06-13 08:13:45.746 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-43> 2019-06-13 08:13:45.810 7ff348fa70c0 1 rocksdb: do_open column families: [default]
-42> 2019-06-13 08:15:48.650 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
-41> 2019-06-13 08:15:48.666 7ff348fa70c0 1 freelist init
-40> 2019-06-13 08:15:48.762 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_alloc opening allocation metadata
-39> 2019-06-13 08:15:49.354 7ff348fa70c0 1 bluestore(/var/lib/ceph/osd/ceph-49) _open_alloc loaded 2.5 TiB in 252916 extents
-38> 2019-06-13 08:16:04.211 7ff348fa70c0 1 bluefs umount
-37> 2019-06-13 08:16:04.211 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e600 shutdown
-36> 2019-06-13 08:16:04.211 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e700 shutdown
-35> 2019-06-13 08:16:04.263 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block.db) close
-34> 2019-06-13 08:16:04.663 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block) close
-33> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block.db type kernel
-32> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) open path /var/lib/ceph/osd/ceph-49/block.db
-31> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) open size 20975714304 (0x4e2400000, 20 GiB) block_size 4096 (4 KiB) non-rotational discard supported
-30> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-49/block.db size 20 GiB
-29> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev create path /var/lib/ceph/osd/ceph-49/block type kernel
-28> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) open path /var/lib/ceph/osd/ceph-49/block
-27> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
-26> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-49/block size 3.6 TiB
-25> 2019-06-13 08:16:04.815 7ff348fa70c0 1 bluefs mount
-24> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-23> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-22> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-21> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-20> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-19> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-18> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-17> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compaction_readahead_size = 2097152
-16> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option compression = kNoCompression
-15> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option max_write_buffer_number = 4
-14> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option min_write_buffer_number_to_merge = 1
-13> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option recycle_log_file_num = 4
-12> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option writable_file_max_buffer_size = 0
-11> 2019-06-13 08:16:04.839 7ff348fa70c0 0 set rocksdb option write_buffer_size = 268435456
-10> 2019-06-13 08:16:04.903 7ff348fa70c0 1 rocksdb: do_open column families: [default]
-9> 2019-06-13 08:19:25.323 7ff348fa70c0 -1 rocksdb: Corruption: file is too short (6139497190 bytes) to be an sstabledb/079728.sst
-8> 2019-06-13 08:19:25.323 7ff348fa70c0 -1 bluestore(/var/lib/ceph/osd/ceph-49) _open_db erroring opening db:
-7> 2019-06-13 08:19:25.323 7ff348fa70c0 1 bluefs umount
-6> 2019-06-13 08:19:25.323 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e700 shutdown
-5> 2019-06-13 08:19:25.323 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6e600 shutdown
-4> 2019-06-13 08:19:25.335 7ff348fa70c0 1 bdev(0x556b90bd0e00 /var/lib/ceph/osd/ceph-49/block.db) close
-3> 2019-06-13 08:19:25.891 7ff348fa70c0 1 bdev(0x556b90bd0a80 /var/lib/ceph/osd/ceph-49/block) close
-2> 2019-06-13 08:19:26.199 7ff348fa70c0 1 fbmap_alloc 0x556b8fd6eb00 shutdown
-1> 2019-06-13 08:19:26.279 7ff348fa70c0 1 freelist shutdown
0> 2019-06-13 08:19:26.283 7ff348fa70c0 -1 *** Caught signal (Segmentation fault) **
in thread 7ff348fa70c0 thread_name:ceph-kvstore-to
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
1: (()+0x12890) [0x7ff33e997890]
2: (StoreTool::traverse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::ostream*)+0x40) [0x556b8ce90120]
3: (main()+0x492) [0x556b8cdb6142]
4: (__libc_start_main()+0xe7) [0x7ff33d86ab97]
5: (_start()+0x2a) [0x556b8ce8e03a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 500
max_new 1000
log_file /var/lib/ceph/crash/2019-06-13_06:19:26.285494Z_7beb59ed-6822-46eb-8e36-6b61e35f49c9/log
--- end dump of recent events ---
Segmentation fault
real 5m42.561s
user 3m2.856s
sys 1m31.892s
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
