ceph-osd failure following 0.92 -> 0.94 upgrade

[Dirk Grunwald <Dirk.Grunwald@xxxxxxxxxxxx>]

Ceph cluster, U14.10 base system, OSD's using BTRFS, journal on same disk as partition

(done using ceph-deploy)

I had been running 0.92 without (significant) issue. I upgraded

to Hammer (0.94) be modifying /etc/apt/sources.list, apt-get update, apt-get upgrade

Upgraded and restarted ceph-mon and then ceph-osd

Most of the 50 OSD's are in a failure cycle with the error "os/Transaction.cc: 504: FAILED assert(ops == data.ops)"

Right now, the entire cluster is useless because of this.

Any suggestions?

--------------

root@zfs2:/var/log/ceph# ceph-osd -i 2 -f

starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal

 HDIO_DRIVE_CMD(identify) failed: Invalid argument

os/Transaction.cc: In function 'void ObjectStore::Transaction::_build_actions_from_tbl()' thread\

 7fa97d88c900 time 2015-04-09 14:57:33.763110

os/Transaction.cc: 504: FAILED assert(ops == data.ops)

 ceph version 0.94 (e61c4f093f88e44961d157f65091733580cea79a)

 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc26ab]

 2: (ObjectStore::Transaction::_build_actions_from_tbl()+0x3735) [0x99fa55]

 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHan\

dle*)+0x407b) [0x92497b]

 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore\

::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x9279e4]

 5: (JournalingObjectStore::journal_replay(unsigned long)+0x5cb) [0x94058b]

 6: (FileStore::mount()+0x3bb6) [0x910d46]

 7: (OSD::init()+0x259) [0x6c47b9]

 8: (main()+0x2860) [0x651fc0]

 9: (__libc_start_main()+0xf5) [0x7fa97a9ceec5]

 10: ceph-osd() [0x66aff7]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2015-04-09 14:57:33.766270 7fa97d88c900 -1 os/Transaction.cc: In function 'void ObjectStore::Tra\

nsaction::_build_actions_from_tbl()' thread 7fa97d88c900 time 2015-04-09 14:57:33.763110

os/Transaction.cc: 504: FAILED assert(ops == data.ops)

 cep

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com