I have a pool that I every time I try to change it's crush_ruleset
crashes 2 out of my 3 mons, and it's always the same. I've tried
leaving the first one down and it crashes the second.
It's a replicated pool, and I have other pools that look exactly the same.
I've deep-scrub'ed all the PG's to make sure there was no corruption.
OTOH the issue is with .rgw.meta pool, that from what I've read could
(?) isn't really needed...
thanks,
Some more information:
ceph osd ls detail (on 2 pools)
pool 20 '.log' replicated size 4 min_size 2 crush_ruleset 4
object_hash rjenkins pg_num 16 pgp_num 16 last_change 35288 owner
18446744073709551615 flags hashpspool min_write_recency_for_promote 1
stripe_width 0
pool 26 '.rgw.meta' replicated size 4 min_size 2 crush_ruleset 0
object_hash rjenkins pg_num 16 pgp_num 16 last_change 62814 owner
18446744073709551615 flags hashpspool stripe_width 0
On the log I get (I've increase logging)
-8> 2016-11-14 14:28:55.049924 7f3cf3115700 10 --
10.33.40.37:6789/0 >> 10.252.24.104:0/3170739472 pipe(0x7f3d108da800
sd=23 :6789 s=2 pgs=1 cs=1 l=1 c=0x7f3d0e782600).write_ack 7
-7> 2016-11-14 14:28:55.049920 7f3cf561e700 1 --
10.33.40.37:6789/0 <== client.56744214 10.252.24.104:0/3170739472 7
==== mon_command({"var": "crush_ruleset", "prefix": "osd pool set",
"pool": ".rgw.meta", "val": "4"} v 0) v1 ==== 125+0+0 (3985264784 0 0)
0x7f3d100ca400 con 0x7f3d0e782600
-6> 2016-11-14 14:28:55.049929 7f3cf3115700 10 --
10.33.40.37:6789/0 >> 10.252.24.104:0/3170739472 pipe(0x7f3d108da800
sd=23 :6789 s=2 pgs=1 cs=1 l=1 c=0x7f3d0e782600).writer: state = open
policy.server=1
-5> 2016-11-14 14:28:55.049969 7f3cf561e700 0
mon.ed05sv38@0(leader) e13 handle_command mon_command({"var":
"crush_ruleset", "prefix": "osd pool set", "pool": ".rgw.meta", "val":
"4"} v 0) v1
-4> 2016-11-14 14:28:55.050003 7f3cf561e700 0 log_channel(audit)
log [INF] : from='client.? 10.252.24.104:0/3170739472'
entity='client.admin' cmd=[{"var": "crush_ruleset", "prefix": "osd
pool set", "pool": ".rgw.meta", "val": "4"}]: dispatch
-3> 2016-11-14 14:28:55.050008 7f3cf561e700 1 --
10.33.40.37:6789/0 --> 10.33.40.37:6789/0 -- log(1 entries from seq
105 at 2016-11-14 14:28:55.050005) v1 -- ?+0 0x7f3d0ea29a80 con
0x7f3d0d652280
-2> 2016-11-14 14:28:55.050021 7f3cf561e700 10
mon.ed05sv38@0(leader).paxosservice(osdmap 62242..62823) dispatch
0x7f3d100ca400 mon_command({"var": "crush_ruleset", "prefix": "osd
pool set", "pool": ".rgw.meta", "val": "4"} v 0) v1 from
client.56744214 10.252.24.104:0/3170739472 con 0x7f3d0e782600
-1> 2016-11-14 14:28:55.050026 7f3cf561e700 5
mon.ed05sv38@0(leader).paxos(paxos active c 79137289..79137877)
is_readable = 1 - now=2016-11-14 14:28:55.050027
lease_expire=2016-11-14 14:28:59.753490 has v0 lc 79137877
0> 2016-11-14 14:28:55.051509 7f3cf561e700 -1 *** Caught signal
(Segmentation fault) **
in thread 7f3cf561e700 thread_name:ms_dispatch
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (()+0x4f3022) [0x7f3d01f73022]
2: (()+0x10340) [0x7f3d00d76340]
3: (OSDMonitor::prepare_command_pool_set(std::map<std::string,
boost::variant<std::string, bool, long, double,
std::vector<std::string, std::allocator<std::string> >,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_>, std::less<std::string>,
std::allocator<std::pair<std::string const,
boost::variant<std::string, bool, long, double,
std::vector<std::string, std::allocator<std::string> >,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_> > > >&, std::basic_stringstream<char,
std::char_traits<char>, std::allocator<char> >&)+0x1228)
[0x7f3d01d8d358]
4: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>,
std::map<std::string, boost::variant<std::string, bool, long, double,
std::vector<std::string, std::allocator<std::string> >,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_>, std::less<std::string>,
std::allocator<std::pair<std::string const,
boost::variant<std::string, bool, long, double,
std::vector<std::string, std::allocator<std::string> >,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_> > > >&)+0xef4c) [0x7f3d01d9cf8c]
5: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x5f6)
[0x7f3d01da1e56]
6: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x2f7)
[0x7f3d01da3927]
7: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xdab)
[0x7f3d01d5324b]
8: (Monitor::handle_command(std::shared_ptr<MonOpRequest>)+0x1cba)
[0x7f3d01d116aa]
9: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0x32b) [0x7f3d01d1d2bb]
10: (Monitor::_ms_dispatch(Message*)+0x6c1) [0x7f3d01d1e5f1]
11: (Monitor::ms_dispatch(Message*)+0x23) [0x7f3d01d3d873]
12: (DispatchQueue::entry()+0x78b) [0x7f3d0217558b]
13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f3d0207268d]
14: (()+0x8182) [0x7f3d00d6e182]
15: (clone()+0x6d) [0x7f3cff0c247d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 0 journaler
0/ 5 objectcacher
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
0/ 0 journal
10/10 ms
0/ 0 mon
0/ 0 monc
10/10 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/antelope-mon.ed05sv38.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
