Dear Sage and Dan: Thanks for kindly reply. If we want to patch http://tracker.newdream.net/issues/2446 on kernel 3.2, can we just modify the file as below? We are afraid that it may cause other side effect. diff --git a/net/ceph/osdmap.c b/net/ceph/osdmap.c index 2592f3c..011a3a9 100644 --- a/net/ceph/osdmap.c +++ b/net/ceph/osdmap.c @@ -883,8 +883,12 @@ struct ceph_osdmap *osdmap_apply_incremental(void **p, void *end, pglen = ceph_decode_32(p); if (pglen) { - /* insert */ ceph_decode_need(p, end, pglen*sizeof(u32), bad); + + /* removing existing (if any) */ + __remove_pg_mapping(&map->pg_temp, pgid); + + /* insert */ pg = kmalloc(sizeof(*pg) + sizeof(u32)*pglen, GFP_NOFS); if (!pg) { err = -ENOMEM; -----Original Message----- From: Sage Weil [mailto:sage@xxxxxxxxxxx] Sent: Thursday, August 09, 2012 11:54 PM To: Eric YH Chen/WYHQ/Wiwynn Cc: ceph-devel@xxxxxxxxxxxxxxx Subject: Re: accident corruption in osdmap On Thu, 9 Aug 2012, Eric_YH_Chen@xxxxxxxxxx wrote: > Dear all: > > My Environment: two servers, and 12 hard-disk on each server. > Version: Ceph 0.48, Kernel: 3.2.0-27 > > We create a ceph cluster with 24 osd, 3 monitors > Osd.0 ~ osd.11 is on server1 > Osd.12 ~ osd.23 is on server2 > Mon.0 is on server1 > Mon.1 is on server2 > Mon.2 is on server3 which has no osd > > We create a rbd device and mount it as ext4 file system. > During read/write data on the rbd device, one of the storage server is shutdown by accident. > After reboot the server, we cannot access the rbd device any more. > One of the log shows the osdmap is corrupted. > > Aug 5 15:37:24 ubuntu-002 kernel: [78579.998582] libceph: corrupt inc > osdmap epoch 78 off 98 (ffffc9000177d07e of > ffffc9000177d01c-ffffc9000177edf2) > > We would like to know what kind of scenario would cause the corruption of osdmap and how to avoid it? > It seems that osdmap corruption cannot be recovered by the ceph cluster itself. Which kernel version are you running? > Is it the same issue with http://tracker.newdream.net/issues/2446? > In which version of kernel that we can find this patch? Thanks! If it is that issue, simply restarting the client is a workaround. It may crop up again on a newer epoch as the cluster recovers/rebalances data, but in each case a client restart will get past it. If that doesn't work, please attach a copy of the incremental osdmap ($mon_data/osdmap/78 from one of your monitors) and we can see what the actual corruption is. Thanks! sage > > > ============= /var/log/kern.log > ================================================================= > Aug 5 15:31:44 ubuntu-002 kernel: [78240.712542] libceph: osd11 down > Aug 5 15:31:49 ubuntu-002 kernel: [78244.817151] libceph: osd12 down > Aug 5 15:31:52 ubuntu-002 kernel: [78248.151815] libceph: osd13 down > Aug 5 15:31:52 ubuntu-002 kernel: [78248.151913] libceph: osd14 down > Aug 5 15:31:53 ubuntu-002 kernel: [78249.250991] libceph: get_reply > unknown tid 96452 from osd7 Aug 5 15:31:59 ubuntu-002 kernel: > [78254.833033] libceph: osd15 down Aug 5 15:31:59 ubuntu-002 kernel: > [78254.833037] libceph: osd16 down Aug 5 15:31:59 ubuntu-002 kernel: > [78254.833039] libceph: osd17 down Aug 5 15:31:59 ubuntu-002 kernel: > [78254.833040] libceph: osd18 down Aug 5 15:31:59 ubuntu-002 kernel: > [78254.833042] libceph: osd19 down Aug 5 15:31:59 ubuntu-002 kernel: > [78254.833062] libceph: osd20 down Aug 5 15:31:59 ubuntu-002 kernel: > [78254.833064] libceph: osd21 down Aug 5 15:36:46 ubuntu-002 kernel: > [78541.813963] libceph: osd11 weight 0x0 (out) Aug 5 15:37:09 > ubuntu-002 kernel: [78564.811236] libceph: osd12 weight 0x0 (out) Aug > 5 15:37:09 ubuntu-002 kernel: [78564.811238] libceph: osd13 weight 0x0 > (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811264] libceph: osd14 > weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811265] > libceph: osd15 weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: > [78564.811266] libceph: osd16 weight 0x0 (out) Aug 5 15:37:09 > ubuntu-002 kernel: [78564.811271] libceph: osd17 weight 0x0 (out) Aug > 5 15:37:09 ubuntu-002 kernel: [78564.811272] libceph: osd18 weight 0x0 > (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811273] libceph: osd19 > weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: [78564.811314] > libceph: osd20 weight 0x0 (out) Aug 5 15:37:09 ubuntu-002 kernel: > [78564.811315] libceph: osd21 weight 0x0 (out) Aug 5 15:37:24 > ubuntu-002 kernel: [78579.998582] libceph: corrupt inc osdmap epoch 78 > off 98 (ffffc9000177d07e of ffffc9000177d01c-ffffc9000177edf2) > Aug 5 15:37:24 ubuntu-002 kernel: [78579.998737] osdmap: 00000000: 05 00 70 d6 52 f9 b3 cc 44 c5 a2 eb c1 33 1d a2 ..p.R...D....3.. > Aug 5 15:37:24 ubuntu-002 kernel: [78579.998739] osdmap: 00000010: 45 3d 4e 00 00 00 b3 22 1e 50 d0 b3 f3 2d ff ff E=N....".P...-.. > Aug 5 15:37:24 ubuntu-002 kernel: [78579.998742] osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................ > ... > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
