after adding more osd's and having a big backfill running 2 of my osd's
keep on stopping.
We also ercently upgraded from 0.94.7 to 0.94.9 but i do not know if
that is related.
the log say.
0> 2016-09-12 10:31:08.288858 7f8749125880 -1 osd/PGLog.cc: In
function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t,
ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&,
PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&,
std::set<std::basic_string<char> >*)' thread 7f8749125880 time
2016-09-12 10:31:08.286337
osd/PGLog.cc: 984: FAILED assert(oi.version == i->first)
googeling led me to a bug that seems to be related to infernalis only.
dmesg does not show anything wrong with the hardware.
this is debian running hammer 0.94.9
and the osd is a software raid5 consisting of 5 3TB harddrives.
journal is a partition on ssd intel 3500
anyone have a clue to what can be wrong ?
kind regrads
Ronny Aasen
-- log debug_filestore=10 --
-19> 2016-09-12 10:31:08.070947 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/1df4bfdd/rb.0.392c.238e1f29.0000002bd134/head '_' = 266
-18> 2016-09-12 10:31:08.083111 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/deb5bfdd/rb.0.392c.238e1f29.0000002bc596/head '_' = 266
-17> 2016-09-12 10:31:08.096718 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/9be5dfdd/rb.0.392c.238e1f29.0000002bc2bf/head '_' = 266
-16> 2016-09-12 10:31:08.110048 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/cbf8ffdd/rb.0.392c.238e1f29.0000002b9d89/head '_' = 266
-15> 2016-09-12 10:31:08.126263 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/e49d0fdd/rb.0.392c.238e1f29.0000002b078e/head '_' = 266
-14> 2016-09-12 10:31:08.150199 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/e49d0fdd/rb.0.392c.238e1f29.0000002b078e/22 '_' = 259
-13> 2016-09-12 10:31:08.173223 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/d0827fdd/rb.0.392c.238e1f29.0000002b0373/head '_' = 266
-12> 2016-09-12 10:31:08.199192 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/d0827fdd/rb.0.392c.238e1f29.0000002b0373/22 '_' = 259
-11> 2016-09-12 10:31:08.232712 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/bf4effdd/rb.0.392c.238e1f29.0000002ae882/head '_' = 266
-10> 2016-09-12 10:31:08.265331 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/bf4effdd/rb.0.392c.238e1f29.0000002ae882/22 '_' = 259
-9> 2016-09-12 10:31:08.265456 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000b381ae__head_DB220FDD__1
with flags=2: (2) No such file or directory
-8> 2016-09-12 10:31:08.265475 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/db220fdd/rb.0.392c.238e1f29.000000b381ae/head '_' = -2
-7> 2016-09-12 10:31:08.265535 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000b381ae__21_DB220FDD__1
with flags=2: (2) No such file or directory
-6> 2016-09-12 10:31:08.265546 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/db220fdd/rb.0.392c.238e1f29.000000b381ae/21 '_' = -2
-5> 2016-09-12 10:31:08.265609 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000cf4057__head_12020FDD__1
with flags=2: (2) No such file or directory
-4> 2016-09-12 10:31:08.265628 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/12020fdd/rb.0.392c.238e1f29.000000cf4057/head '_' = -2
-3> 2016-09-12 10:31:08.265688 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) error opening file
/var/lib/ceph/osd/ceph-8/current/1.fdd_head/DIR_D/DIR_D/DIR_F/DIR_0/DIR_2/rb.0.392c.238e1f29.000000cf4057__21_12020FDD__1
with flags=2: (2) No such file or directory
-2> 2016-09-12 10:31:08.265700 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/12020fdd/rb.0.392c.238e1f29.000000cf4057/21 '_' = -2
-1> 2016-09-12 10:31:08.286313 7f8749125880 10
filestore(/var/lib/ceph/osd/ceph-8) getattr
1.fdd_head/1/882e0fdd/rb.0.392c.238e1f29.0000003c9802/21 '_' = 251
0> 2016-09-12 10:31:08.288858 7f8749125880 -1 osd/PGLog.cc: In
function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t,
ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&,
PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&,
std::set<std::basic_string<char> >*)' thread 7f8749125880 time
2016-09-12 10:31:08.286337
osd/PGLog.cc: 984: FAILED assert(oi.version == i->first)
ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x76) [0xc0f196]
2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t,
pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>,
std::allocator<std::pair<eversion_t const, hobject_t> > >&,
PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char,
std::char_traits<char>, std::allocator<char> >&, std::set<std::string,
std::less<std::string>, std::allocator<std::string> >*)+0x11ab) [0x76f9ab]
3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x1e2) [0x7f6c72]
4: (OSD::load_pgs()+0xac0) [0x6abd00]
5: (OSD::init()+0x14da) [0x6af54a]
6: (main()+0x2848) [0x6339f8]
7: (__libc_start_main()+0xf5) [0x7f8746443b45]
8: /usr/bin/ceph-osd() [0x64d687]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
10/10 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.8.log
--- end dump of recent events ---
2016-09-12 10:31:08.376098 7f8749125880 -1 *** Caught signal (Aborted) **
in thread 7f8749125880
ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
1: /usr/bin/ceph-osd() [0xb0c4d3]
2: (()+0xf8d0) [0x7f8747fb68d0]
3: (gsignal()+0x37) [0x7f8746457067]
4: (abort()+0x148) [0x7f8746458448]
5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f8746d44b3d]
6: (()+0x5ebb6) [0x7f8746d42bb6]
7: (()+0x5ec01) [0x7f8746d42c01]
8: (()+0x5ee19) [0x7f8746d42e19]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x247) [0xc0f367]
10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t,
pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>,
std::allocator<std::pair<eversion_t const, hobject_t> > >&,
PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char,
std::char_traits<char>, std::allocator<char> >&, std::set<std::string,
std::less<std::string>, std::allocator<std::string> >*)+0x11ab) [0x76f9ab]
11: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x1e2) [0x7f6c72]
12: (OSD::load_pgs()+0xac0) [0x6abd00]
13: (OSD::init()+0x14da) [0x6af54a]
14: (main()+0x2848) [0x6339f8]
15: (__libc_start_main()+0xf5) [0x7f8746443b45]
16: /usr/bin/ceph-osd() [0x64d687]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
0> 2016-09-12 10:31:08.376098 7f8749125880 -1 *** Caught signal
(Aborted) **
in thread 7f8749125880
ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
1: /usr/bin/ceph-osd() [0xb0c4d3]
2: (()+0xf8d0) [0x7f8747fb68d0]
3: (gsignal()+0x37) [0x7f8746457067]
4: (abort()+0x148) [0x7f8746458448]
5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f8746d44b3d]
6: (()+0x5ebb6) [0x7f8746d42bb6]
7: (()+0x5ec01) [0x7f8746d42c01]
8: (()+0x5ee19) [0x7f8746d42e19]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x247) [0xc0f367]
10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t,
pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>,
std::allocator<std::pair<eversion_t const, hobject_t> > >&,
PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char,
std::char_traits<char>, std::allocator<char> >&, std::set<std::string,
std::less<std::string>, std::allocator<std::string> >*)+0x11ab) [0x76f9ab]
11: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x1e2) [0x7f6c72]
12: (OSD::load_pgs()+0xac0) [0x6abd00]
13: (OSD::init()+0x14da) [0x6af54a]
14: (main()+0x2848) [0x6339f8]
15: (__libc_start_main()+0xf5) [0x7f8746443b45]
16: /usr/bin/ceph-osd() [0x64d687]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
10/10 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.8.log
--- end dump of recent events ---
root@ceph-osd5:~#
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
