Hi, All. I encountered such a problem. Was the status of one pg - inconsistent. RBD found this device and deleted it, now on the OSD get the following error: cod 0'0 active+inconsistent snaptrimq=[15~1,89~1]] exit Started/Primary/Active/Recovering 0.025609 1 0.000053 -8> 2014-07-23 12:03:13.386747 7f3617b02700 5 osd.94 pg_epoch: 35725 pg[80.3d6( v 35718'170614 (34929'167412,35718'170614] local-les=35724 n=2242 ec=9510 les/c 35724/35718 35723/35723/35723) [94,36] r=0 lpr=35723 pi=35713-35722/2 ml cod 0'0 active+inconsistent snaptrimq=[15~1,89~1]] enter Started/Primary/Active/Recovered -7> 2014-07-23 12:03:13.386783 7f3617b02700 5 osd.94 pg_epoch: 35725 pg[80.3d6( v 35718'170614 (34929'167412,35718'170614] local-les=35724 n=2242 ec=9510 les/c 35724/35718 35723/35723/35723) [94,36] r=0 lpr=35723 pi=35713-35722/2 ml cod 0'0 active+inconsistent snaptrimq=[15~1,89~1]] exit Started/Primary/Active/Recovered 0.000035 0 0.000000 -6> 2014-07-23 12:03:13.386795 7f3617b02700 5 osd.94 pg_epoch: 35725 pg[80.3d6( v 35718'170614 (34929'167412,35718'170614] local-les=35724 n=2242 ec=9510 les/c 35724/35718 35723/35723/35723) [94,36] r=0 lpr=35723 pi=35713-35722/2 ml cod 0'0 active+inconsistent snaptrimq=[15~1,89~1]] enter Started/Primary/Active/Clean -5> 2014-07-23 12:03:13.386932 7f3617b02700 5 osd.94 pg_epoch: 35725 pg[2.772( v 35722'486163 lc 35716'486156 (35141'483132,35722'486163] local-les=35724 n=1328 ec=1 les/c 35724/35718 35723/35723/35723) [94,38,59] r=0 lpr=35723 pi=3 5713-35722/2 lcod 0'0 mlcod 0'0 active+recovery_wait m=4] exit Started/Primary/Active/WaitLocalRecoveryReserved 4.377808 7 0.000096 -4> 2014-07-23 12:03:13.386956 7f3617b02700 5 osd.94 pg_epoch: 35725 pg[2.772( v 35722'486163 lc 35716'486156 (35141'483132,35722'486163] local-les=35724 n=1328 ec=1 les/c 35724/35718 35723/35723/35723) [94,38,59] r=0 lpr=35723 pi=3 5713-35722/2 lcod 0'0 mlcod 0'0 active+recovery_wait m=4] enter Started/Primary/Active/WaitRemoteRecoveryReserved -3> 2014-07-23 12:03:13.387282 7f36148fd700 -1 osd.94 pg_epoch: 35725 pg[80.3d6( v 35718'170614 (34929'167412,35718'170614] local-les=35724 n=2242 ec=9510 les/c 35724/35725 35723/35723/35723) [94,36] r=0 lpr=35723 mlcod 0'0 active+cl ean+inconsistent snaptrimq=[15~1,89~1]] *trim_objectcould not find coid * f022c7d6/rbd_data.3ed9c72ae8944a.0000000000000717/15//80 -2> 2014-07-23 12:03:13.388628 7f3617101700 5 osd.94 pg_epoch: 35725 pg[2.772( v 35722'486163 lc 35716'486156 (35141'483132,35722'486163] local-les=35724 n=1328 ec=1 les/c 35724/35718 35723/35723/35723) [94,38,59] r=0 lpr=35723 pi=3 5713-35722/2 lcod 0'0 mlcod 0'0 active+recovery_wait m=4] exit Started/Primary/Active/WaitRemoteRecoveryReserved 0.001672 2 0.000079 -1> 2014-07-23 12:03:13.388670 7f3617101700 5 osd.94 pg_epoch: 35725 pg[2.772( v 35722'486163 lc 35716'486156 (35141'483132,35722'486163] local-les=35724 n=1328 ec=1 les/c 35724/35718 35723/35723/35723) [94,38,59] r=0 lpr=35723 pi=3 5713-35722/2 lcod 0'0 mlcod 0'0 active+recovery_wait m=4] enter Started/Primary/Active/Recovering 0> 2014-07-23 12:03:13.389138 7f36148fd700 -1 osd/ReplicatedPG.cc: In function 'ReplicatedPG::RepGather* ReplicatedPG::trim_object(const hobject_t&)' thread 7f36148fd700 time 2014-07-23 12:03:13.387304 osd/ReplicatedPG.cc: 1824: FAILED assert(0) [root at ceph08 DIR_7]# find /var/lib/ceph/osd/ceph-94/ -name '*3ed9c72ae8944a.0000000000000717*' -ls 1074528377 0 -rw-r--r-- 1 root root 1 ??? 23 11:30 /var/lib/ceph/osd/ceph-94/current/80.3d6_head/DIR_6/DIR_D/DIR_7/rbd\\udata.3ed9c72ae8944a.0000000000000717__15_F022C7D6__50 How to make ceph forgot about the existence of this file? Ceph version 0.72.2 Thanks. -- ? ?????????, ??????? ???? ??????????? ???.: +79229045757 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140723/babbed25/attachment.htm>