Hi,
Every time I try an bring up an OSD, it crashes and I get the
following: "error (121) Remote I/O error not handled on operation 20"
The cluster is new and only has a little bit of data on it. Any ideas
what is going on? Does Remote I/O mean a network error? Full log
below:
-9> 2013-01-10 00:00:20.182237 7f2ddde8f910 0
filestore(/mnt/dist_j/ceph) error (121) Remote I/O error not handled
on operation 20 (12.0.0, or op 0, counting from 0)
-8> 2013-01-10 00:00:20.182275 7f2ddde8f910 0
filestore(/mnt/dist_j/ceph) unexpected error code
-7> 2013-01-10 00:00:20.182285 7f2ddde8f910 0
filestore(/mnt/dist_j/ceph) transaction dump:
{ "ops": [
{ "op_num": 0,
"op_name": "mkcoll",
"collection": "0.2c0_head"},
{ "op_num": 1,
"op_name": "collection_setattr",
"collection": "0.2c0_head",
"name": "info",
"length": 5},
{ "op_num": 2,
"op_name": "truncate",
"collection": "meta",
"oid": "a04c46e9\/pginfo_0.2c0\/0\/\/-1",
"offset": 0},
{ "op_num": 3,
"op_name": "write",
"collection": "meta",
"oid": "a04c46e9\/pginfo_0.2c0\/0\/\/-1",
"length": 531,
"offset": 0,
"bufferlist length": 531},
{ "op_num": 4,
"op_name": "remove",
"collection": "meta",
"oid": "1f9ede85\/pglog_0.2c0\/0\/\/-1"},
{ "op_num": 5,
"op_name": "write",
"collection": "meta",
"oid": "1f9ede85\/pglog_0.2c0\/0\/\/-1",
"length": 0,
"offset": 0,
"bufferlist length": 0},
{ "op_num": 6,
"op_name": "collection_setattr",
"collection": "0.2c0_head",
"name": "ondisklog",
"length": 34},
{ "op_num": 7,
"op_name": "nop"}]}
-6> 2013-01-10 00:00:20.183085 7f2dd5e7f910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-5> 2013-01-10 00:00:20.183108 7f2dd5e7f910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5b15600 con 0x34629a0
-4> 2013-01-10 00:00:20.183772 7f2dd6680910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-3> 2013-01-10 00:00:20.183797 7f2dd6680910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5f75600 con 0x34629a0
-2> 2013-01-10 00:00:20.184315 7f2dd5e7f910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-1> 2013-01-10 00:00:20.184338 7f2dd5e7f910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5b15400 con 0x34629a0
0> 2013-01-10 00:00:20.184755 7f2ddde8f910 -1 os/FileStore.cc: In
function 'unsigned int
FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int)'
thread 7f2ddde8f910 time 2013-01-10 00:00:20.182422
os/FileStore.cc: 2681: FAILED assert(0 == "unexpected error")
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int)+0x90a) [0x73e14a]
2: (FileStore::do_transactions(std::list<ObjectStore::Transaction*,
std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c)
[0x7455dc]
3: (FileStore::_do_op(FileStore::OpSequencer*)+0xab) [0x72428b]
4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x82b) [0x894feb]
5: (ThreadPool::WorkThread::entry()+0x10) [0x8977d0]
6: /lib/libpthread.so.0 [0x7f2de6d087aa]
7: (clone()+0x6d) [0x7f2de518159d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 100000
max_new 1000
log_file /var/log/ceph/ceph-osd.9.log
--- end dump of recent events ---
2013-01-10 00:00:20.227763 7f2ddde8f910 -1 *** Caught signal (Aborted) **
in thread 7f2ddde8f910
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: /cf/ceph/bin/ceph-osd [0x7a5309]
2: /lib/libpthread.so.0 [0x7f2de6d10a60]
3: (gsignal()+0x35) [0x7f2de50e7f05]
4: (abort()+0x180) [0x7f2de50ead10]
5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f2de596ed45]
6: /usr/lib/libstdc++.so.6 [0x7f2de596d176]
7: /usr/lib/libstdc++.so.6 [0x7f2de596d1a3]
8: /usr/lib/libstdc++.so.6 [0x7f2de596d29e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x7c9) [0x898029]
10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int)+0x90a) [0x73e14a]
11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*,
std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c)
[0x7455dc]
12: (FileStore::_do_op(FileStore::OpSequencer*)+0xab) [0x72428b]
13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x82b) [0x894feb]
14: (ThreadPool::WorkThread::entry()+0x10) [0x8977d0]
15: /lib/libpthread.so.0 [0x7f2de6d087aa]
16: (clone()+0x6d) [0x7f2de518159d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
-17> 2013-01-10 00:00:20.184913 7f2dd6680910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-16> 2013-01-10 00:00:20.184936 7f2dd6680910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5f75400 con 0x34629a0
-15> 2013-01-10 00:00:20.185444 7f2dd5e7f910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-14> 2013-01-10 00:00:20.185461 7f2dd5e7f910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5b15200 con 0x34629a0
-13> 2013-01-10 00:00:20.186028 7f2dd6680910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-12> 2013-01-10 00:00:20.186049 7f2dd6680910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5f75200 con 0x34629a0
-11> 2013-01-10 00:00:20.186585 7f2dd5e7f910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-10> 2013-01-10 00:00:20.186596 7f2dd5e7f910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5b15000 con 0x34629a0
-9> 2013-01-10 00:00:20.186956 7f2dd6680910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-8> 2013-01-10 00:00:20.186973 7f2dd6680910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x5f75000 con 0x34629a0
-7> 2013-01-10 00:00:20.187288 7f2dd5e7f910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-6> 2013-01-10 00:00:20.187298 7f2dd5e7f910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x387ce00 con 0x34629a0
-5> 2013-01-10 00:00:20.187671 7f2dd6680910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-4> 2013-01-10 00:00:20.187688 7f2dd6680910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x393ae00 con 0x34629a0
-3> 2013-01-10 00:00:20.187946 7f2dd5e7f910 10 monclient:
_send_mon_message to mon.a at 108.162.209.120:6789/0
-2> 2013-01-10 00:00:20.187957 7f2dd5e7f910 1 --
108.162.209.120:6834/6359 --> 108.162.209.120:6789/0 -- osd_pgtemp(e22
{0.110=[8,9],0.147=[3,9],0.155=[1,9],0.171=[0,9],0.194=[3,9],0.1ad=[10,9],0.1c2=[1,9],0.1cb=[7,9],0.1df=[6,9],0.1e8=[7,9],0.1e9=[11,9],0.1f1=[7,9]}
v22) v1 -- ?+0 0x387cc00 con 0x34629a0
-1> 2013-01-10 00:00:20.200448 7f2dcfb4d910 1 --
108.162.209.120:6836/6359 >> :/0 pipe(0x38616c0 sd=49 :6836 pgs=0 cs=0
l=0).accept sd=49 108.162.209.120:13844/0
0> 2013-01-10 00:00:20.227763 7f2ddde8f910 -1 *** Caught signal
(Aborted) **
in thread 7f2ddde8f910
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: /cf/ceph/bin/ceph-osd [0x7a5309]
2: /lib/libpthread.so.0 [0x7f2de6d10a60]
3: (gsignal()+0x35) [0x7f2de50e7f05]
4: (abort()+0x180) [0x7f2de50ead10]
5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f2de596ed45]
6: /usr/lib/libstdc++.so.6 [0x7f2de596d176]
7: /usr/lib/libstdc++.so.6 [0x7f2de596d1a3]
8: /usr/lib/libstdc++.so.6 [0x7f2de596d29e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x7c9) [0x898029]
10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int)+0x90a) [0x73e14a]
11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*,
std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c)
[0x7455dc]
12: (FileStore::_do_op(FileStore::OpSequencer*)+0xab) [0x72428b]
13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x82b) [0x894feb]
14: (ThreadPool::WorkThread::entry()+0x10) [0x8977d0]
15: /lib/libpthread.so.0 [0x7f2de6d087aa]
16: (clone()+0x6d) [0x7f2de518159d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 100000
max_new 1000
log_file /var/log/ceph/ceph-osd.9.log
--- end dump of recent events ---
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
