Hi guys,
Someone I worked with today pointed me to a quick and easy way to
bring down an entire cluster, by making all mons kill themselves in
mass suicide:
ceph osd setmaxosd 2147483647
2012-07-05 16:29:41.893862 b5962b70 0 monclient: hunting for new mon
I don't know what the actual threshold is, but setting your maxosd to
any sufficiently big number should do it. I had hoped 2^31-1 would be
fine, but evidently it's not.
This is what's in the mon log -- the first line is obviously only on
the leader at the time of the command, the others are on all mons.
-1> 2012-07-05 16:29:41.829470 b41a1b70 0 mon.daisy@0(leader) e1
handle_command mon_command(osd setmaxosd 2147483647 v 0) v1
0> 2012-07-05 16:29:41.887590 b41a1b70 -1 *** Caught signal (Aborted) **
in thread b41a1b70
ceph version 0.48argonaut (commit:c2b20ca74249892c8e5e40c12aa14446a2bf2030)
1: /usr/bin/ceph-mon() [0x816f461]
2: [0xb7738400]
3: [0xb7738424]
4: (gsignal()+0x51) [0xb731a781]
5: (abort()+0x182) [0xb731dbb2]
6: (__gnu_cxx::__verbose_terminate_handler()+0x14f) [0xb753b53f]
7: (()+0xbd405) [0xb7539405]
8: (()+0xbd442) [0xb7539442]
9: (()+0xbd581) [0xb7539581]
10: (()+0x11dea) [0xb7582dea]
11: (tc_new()+0x26) [0xb75a1636]
12: (std::vector<unsigned char, std::allocator<unsigned char>
>::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned char*,
std::vector<unsigned char, std::allocator<unsigned char> > >, unsigned
int, unsigned char const&)+0x79) [0x8185629]
13: (OSDMap::set_max_osd(int)+0x497) [0x817c6b7]
>From src/mon/OSDMonitor.cc:
int newmax = atoi(m->cmd[2].c_str());
if (newmax < osdmap.crush->get_max_devices()) {
err = -ERANGE;
ss << "cannot set max_osd to " << newmax << " which is < crush
max_devices "
<< osdmap.crush->get_max_devices();
goto out;
}
I think that counts as unchecked user input, or has cmd[2] been
sanitized at any time before it gets here?
Also, is there a way to recover from this, short of reinitializing all mons?
Cheers,
Florian
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
