This is "supposed" to work, but the compression in Bluestore has less testing than most things there and is pretty invasive, so when I discussed this with Radoslaw (added) last week there were some obvious places to look. Hopefully it's not too hard to identify the problem from these backtraces and get a fix in? :)
-Greg
On Fri, Oct 20, 2017 at 12:16 AM Cassiano Pilipavicius <cassiano@xxxxxxxxxxx> wrote:
Hello, is it possible to use compression on a EC pool? I am trying to
enable this to use as a huge backup/archive disk, the data is almost
static and access to it is very sporadic, so, bad performance is not a
concern here.
I've created the RBD storing data to the EC pool (--data-pool option)
and enabled compression on the EC pool, but as soon as I start to write
data to the volume, I have a crash on 12 osds (my ec ruleset is k=9 m=3)
If I use only RBD+EC it works fine, if I use RBD+Compression (no erasure
coding) it also works fine.
This is what appears on the log of the crashed osds:
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e)
luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x110) [0x7f019377af20]
2: (bluestore_blob_t::map(unsigned long, unsigned long,
std::function<int (unsigned long, unsigned long)>) const+0xfe)
[0x7f019365f14e]
3: (BlueStore::_do_remove(BlueStore::TransContext*,
boost::intrusive_ptr<BlueStore::Collection>&,
boost::intrusive_ptr<BlueStore::Onode>)+0x1151) [0x7f019363d191]
4: (BlueStore::_remove(BlueStore::TransContext*,
boost::intrusive_ptr<BlueStore::Collection>&,
boost::intrusive_ptr<BlueStore::Onode>&)+0x94) [0x7f019363da64]
5: (BlueStore::_txc_add_transaction(BlueStore::TransContext*,
ObjectStore::Transaction*)+0x15af) [0x7f019365047f]
6: (BlueStore::queue_transactions(ObjectStore::Sequencer*,
std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&,
boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a0)
[0x7f01936513a0]
7:
(PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&,
boost::intrusive_ptr<OpRequest>)+0x65) [0x7f01933c1f35]
8: (ECBackend::handle_sub_write(pg_shard_t,
boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&,
Context*)+0x631) [0x7f01934e1cc1]
9: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x327)
[0x7f01934f2867]
10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50)
[0x7f01933f73d0]
11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x5ae) [0x7f0193362fbe]
12: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9)
[0x7f01931f33a9]
13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest>
const&)+0x57) [0x7f0193465797]
14: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0xfce) [0x7f019321e9ee]
15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839)
[0x7f0193780a39]
16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f01937829d0]
17: (()+0x7dc5) [0x7f019013adc5]
18: (clone()+0x6d) [0x7f018f22e28d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 5 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
0/ 0 journal
0/ 0 ms
1/ 5 mon
0/ 0 monc
1/ 5 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.8.log
--- end dump of recent events ---
2017-10-20 04:28:51.564373 7f0176684700 -1 *** Caught signal (Aborted) **
in thread 7f0176684700 thread_name:tp_osd_tp
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e)
luminous (stable)
1: (()+0xa29511) [0x7f019373c511]
2: (()+0xf100) [0x7f0190142100]
3: (gsignal()+0x37) [0x7f018f16d5f7]
4: (abort()+0x148) [0x7f018f16ece8]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x284) [0x7f019377b094]
6: (bluestore_blob_t::map(unsigned long, unsigned long,
std::function<int (unsigned long, unsigned long)>) const+0xfe)
[0x7f019365f14e]
7: (BlueStore::_do_remove(BlueStore::TransContext*,
boost::intrusive_ptr<BlueStore::Collection>&,
boost::intrusive_ptr<BlueStore::Onode>)+0x1151) [0x7f019363d191]
8: (BlueStore::_remove(BlueStore::TransContext*,
boost::intrusive_ptr<BlueStore::Collection>&,
boost::intrusive_ptr<BlueStore::Onode>&)+0x94) [0x7f019363da64]
9: (BlueStore::_txc_add_transaction(BlueStore::TransContext*,
ObjectStore::Transaction*)+0x15af) [0x7f019365047f]
10: (BlueStore::queue_transactions(ObjectStore::Sequencer*,
std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&,
boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a0)
[0x7f01936513a0]
11:
(PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&,
boost::intrusive_ptr<OpRequest>)+0x65) [0x7f01933c1f35]
12: (ECBackend::handle_sub_write(pg_shard_t,
boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&,
Context*)+0x631) [0x7f01934e1cc1]
13:
(ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x327)
[0x7f01934f2867]
14: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50)
[0x7f01933f73d0]
15: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x5ae) [0x7f0193362fbe]
16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9)
[0x7f01931f33a9]
17: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest>
const&)+0x57) [0x7f0193465797]
18: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0xfce) [0x7f019321e9ee]
19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839)
[0x7f0193780a39]
20: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f01937829d0]
21: (()+0x7dc5) [0x7f019013adc5]
22: (clone()+0x6d) [0x7f018f22e28d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
0> 2017-10-20 04:28:51.564373 7f0176684700 -1 *** Caught signal
(Aborted) **
in thread 7f0176684700 thread_name:tp_osd_tp
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e)
luminous (stable)
1: (()+0xa29511) [0x7f019373c511]
2: (()+0xf100) [0x7f0190142100]
3: (gsignal()+0x37) [0x7f018f16d5f7]
4: (abort()+0x148) [0x7f018f16ece8]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x284) [0x7f019377b094]
6: (bluestore_blob_t::map(unsigned long, unsigned long,
std::function<int (unsigned long, unsigned long)>) const+0xfe)
[0x7f019365f14e]
7: (BlueStore::_do_remove(BlueStore::TransContext*,
boost::intrusive_ptr<BlueStore::Collection>&,
boost::intrusive_ptr<BlueStore::Onode>)+0x1151) [0x7f019363d191]
8: (BlueStore::_remove(BlueStore::TransContext*,
boost::intrusive_ptr<BlueStore::Collection>&,
boost::intrusive_ptr<BlueStore::Onode>&)+0x94) [0x7f019363da64]
9: (BlueStore::_txc_add_transaction(BlueStore::TransContext*,
ObjectStore::Transaction*)+0x15af) [0x7f019365047f]
10: (BlueStore::queue_transactions(ObjectStore::Sequencer*,
std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&,
boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a0)
[0x7f01936513a0]
11:
(PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction,
std::allocator<ObjectStore::Transaction> >&,
boost::intrusive_ptr<OpRequest>)+0x65) [0x7f01933c1f35]
12: (ECBackend::handle_sub_write(pg_shard_t,
boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&,
Context*)+0x631) [0x7f01934e1cc1]
13:
(ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x327)
[0x7f01934f2867]
14: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50)
[0x7f01933f73d0]
15: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x5ae) [0x7f0193362fbe]
16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9)
[0x7f01931f33a9]
17: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest>
const&)+0x57) [0x7f0193465797]
18: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0xfce) [0x7f019321e9ee]
19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839)
[0x7f0193780a39]
20: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f01937829d0]
21: (()+0x7dc5) [0x7f019013adc5]
22: (clone()+0x6d) [0x7f018f22e28d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 5 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
0/ 0 journal
0/ 0 ms
1/ 5 mon
0/ 0 monc
1/ 5 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.8.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
