[switching to ceph-devel]
On Fri, 15 Aug 2014, David Zafman wrote:
>
> The way my test works, is that when an osd goes down we export a pg from it.
> Then we look to see if other OSDs are already down to pick from. Since we
> are moving a pg from an OSD that has been up longer than the destination
> OSD, that one doesn?t have the maps that the newly imported pg needs.
>
> In load_pgs() the get_map() called here asserts:
>
> dout(10) << "pgid " << pgid << " coll " << coll_t(pgid) << dendl;
> bufferlist bl;
> epoch_t map_epoch = PG::peek_map_epoch(store, coll_t(pgid),
> service.infos_oid, &bl);
>
> PG *pg = _open_lock_pg(map_epoch == 0 ? osdmap :
> service.get_map(map_epoch), pgid);
>
> OSDMapRef get_map(epoch_t e) {
> OSDMapRef ret(try_get_map(e));
> assert(ret);
> return ret;
> }
>
> osd-3 log
> -8> 2014-08-15 10:40:08.516025 7ff2f931d7a0 10 osd.3 61 pgid 1.10 coll
> 1.10_head
> -7> 2014-08-15 10:40:08.516034 7ff2f931d7a0 15
> filestore(/var/lib/ceph/osd/ceph-3) collection_getattr
> /var/lib/ceph/osd/ceph-3/current/1.10_head 'info'
> -6> 2014-08-15 10:40:08.516048 7ff2f931d7a0 10
> filestore(/var/lib/ceph/osd/ceph-3) collection_getattr
> /var/lib/ceph/osd/ceph-3/current/1.10_head 'info' = 1
> -5> 2014-08-15 10:40:08.516056 7ff2f931d7a0 15
> filestore(/var/lib/ceph/osd/ceph-3) omap_get_values
> meta/16ef7597/infos/head//-1
> -4> 2014-08-15 10:40:08.516111 7ff2f931d7a0 20 osd.3 0 get_map 77 -
> loading and decoding 0x3aea400
> -3> 2014-08-15 10:40:08.516120 7ff2f931d7a0 15
> filestore(/var/lib/ceph/osd/ceph-3) read meta/64173f2c/osdmap.77/0//-1 0~0
> -2> 2014-08-15 10:40:08.516165 7ff2f931d7a0 10
> filestore(/var/lib/ceph/osd/ceph-3) error opening file
> /var/lib/ceph/osd/ceph-3/current/meta/osdmap.77__0_64173F2C__none with
> flags=2: (2) No such file or directory
> -1> 2014-08-15 10:40:08.516246 7ff2f931d7a0 10
> filestore(/var/lib/ceph/osd/ceph-3)
> FileStore::read(meta/64173f2c/osdmap.77/0//-1) open error: (2) No such file
> or directory
> 0> 2014-08-15 10:40:08.517093 7ff2f931d7a0 -1 osd/OSD.h: In function
> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7ff2f931d7a0 time 2014-08-15
> 10:40:08.516268
> osd/OSD.h: 729: FAILED assert(ret)
>
> ceph version 0.83-731-g50e1ed0 (50e1ed04a1a48d653f11829fdf7e0f3954a24f97)
> 1: (OSD::load_pgs()+0x23b1) [0x660551]
> 2: (OSD::init()+0x19d5) [0x663145]
> 3: (main()+0x35ec) [0x6019fc]
> 4: (__libc_start_main()+0xfd) [0x379ae1ed1d]
> 5: ceph-osd() [0x5fe059]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
Maybe we also need a export and import maps functions. Then there would
be a check on import that the osd has at least the map that the PG needs.
And on OSD startup, the existing code in the OSD will (I think) get all of
the PGs to consume the new maps if they are present.
The main challenge would then be to make sure that we export
enough maps with the PG that the import will succeed (so we don't have to
retry or something). Maybe the thrasher would grab the current epoch each
time it starts a new episode of thrashing and always export maps back to
that point (when we were last active+clean) to ensure it's enough ...
sage
