Hi Sage,
Below are my findings for your fix.
### Log of the first OSD run after the reinstall
2017-08-25 17:14:52.199134 7fdcfeb5ccc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc loaded 3655 G in 28 extents
2017-08-25 17:14:52.723949 7fdcfeb5ccc0 1 bluefs fsck
2017-08-25 17:14:52.723966 7fdcfeb5ccc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck walking object keyspace
2017-08-25 17:24:22.861239 7fdcfeb5ccc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking shared_blobs
2017-08-25 17:24:22.861289 7fdcfeb5ccc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking for stray omap data
2017-08-25 17:24:25.214386 7fdcfeb5ccc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking deferred events
2017-08-25 17:24:25.214455 7fdcfeb5ccc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking freelist vs allocated
2017-08-25 17:24:39.531219 7fdcfeb5ccc0 -1 bluestore(/var/lib/ceph/osd/ceph-0) fsck warning: fixing leaked block 0x3a37b2d0000~1000 due to old bug
2017-08-25 17:24:39.531234 7fdcfeb5ccc0 -1 bluestore(/var/lib/ceph/osd/ceph-0) fsck fixing just the stray block at 0x3a37b2d0000~1000
2017-08-25 17:24:39.803856 7fdcfeb5ccc0 1 stupidalloc shutdown
2017-08-25 17:24:39.803868 7fdcfeb5ccc0 1 freelist shutdown
2017-08-25 17:24:39.803905 7fdcfeb5ccc0 4 rocksdb: [/build/nitin/nightly_builds/20170825_152334-wip-21089/ceph.git/rpmbuild/BUILD/ceph-0.71-48280.gc029a96/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2017-08-25 17:24:39.804647 7fdcfeb5ccc0 4 rocksdb: [/build/nitin/nightly_builds/20170825_152334-wip-21089/ceph.git/rpmbuild/BUILD/ceph-0.71-48280.gc029a96/src/rocksdb/db/db_impl.cc:343] Shutdown complete
2017-08-25 17:24:39.916850 7fdcfeb5ccc0 1 bluefs umount
2017-08-25 17:24:39.916865 7fdcfeb5ccc0 1 stupidalloc shutdown
2017-08-25 17:24:39.916868 7fdcfeb5ccc0 1 stupidalloc shutdown
2017-08-25 17:24:39.916869 7fdcfeb5ccc0 1 stupidalloc shutdown
2017-08-25 17:24:39.916900 7fdcfeb5ccc0 1 bdev(0x55e99ab30400 /var/lib/ceph/osd/ceph-0/block.wal) close
2017-08-25 17:24:40.090870 7fdcfeb5ccc0 1 bdev(0x55e99ab30000 /var/lib/ceph/osd/ceph-0/block.db) close
2017-08-25 17:24:40.346856 7fdcfeb5ccc0 1 bdev(0x55e99ab30200 /var/lib/ceph/osd/ceph-0/block) close
2017-08-25 17:24:40.594848 7fdcfeb5ccc0 1 bdev(0x55e99a6efe00 /var/lib/ceph/osd/ceph-0/block) close
2017-08-25 17:24:40.671046 7fdcfeb5ccc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck finish with 0 errors in 595.187882 seconds
### Log of the second OSD run after the reinstall
2017-08-25 17:29:33.852361 7fc8c21abcc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc loaded 3655 G in 23 extents
2017-08-25 17:29:34.358094 7fc8c21abcc0 1 bluefs fsck
2017-08-25 17:29:34.358108 7fc8c21abcc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck walking object keyspace
2017-08-25 17:39:01.538464 7fc8c21abcc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking shared_blobs
2017-08-25 17:39:01.538513 7fc8c21abcc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking for stray omap data
2017-08-25 17:39:03.883701 7fc8c21abcc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking deferred events
2017-08-25 17:39:03.883766 7fc8c21abcc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck checking freelist vs allocated
2017-08-25 17:39:17.885003 7fc8c21abcc0 1 stupidalloc shutdown
2017-08-25 17:39:17.885016 7fc8c21abcc0 1 freelist shutdown
2017-08-25 17:39:17.885057 7fc8c21abcc0 4 rocksdb: [/build/nitin/nightly_builds/20170825_152334-wip-21089/ceph.git/rpmbuild/BUILD/ceph-0.71-48280.gc029a96/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2017-08-25 17:39:17.885886 7fc8c21abcc0 4 rocksdb: [/build/nitin/nightly_builds/20170825_152334-wip-21089/ceph.git/rpmbuild/BUILD/ceph-0.71-48280.gc029a96/src/rocksdb/db/db_impl.cc:343] Shutdown complete
2017-08-25 17:39:17.985250 7fc8c21abcc0 1 bluefs umount
2017-08-25 17:39:17.985399 7fc8c21abcc0 1 stupidalloc shutdown
2017-08-25 17:39:17.985408 7fc8c21abcc0 1 stupidalloc shutdown
2017-08-25 17:39:17.985409 7fc8c21abcc0 1 stupidalloc shutdown
2017-08-25 17:39:17.985440 7fc8c21abcc0 1 bdev(0x55cb71236400 /var/lib/ceph/osd/ceph-0/block.wal) close
2017-08-25 17:39:18.290894 7fc8c21abcc0 1 bdev(0x55cb71236000 /var/lib/ceph/osd/ceph-0/block.db) close
2017-08-25 17:39:18.382827 7fc8c21abcc0 1 bdev(0x55cb71236200 /var/lib/ceph/osd/ceph-0/block) close
2017-08-25 17:39:18.574843 7fc8c21abcc0 1 bdev(0x55cb70df1e00 /var/lib/ceph/osd/ceph-0/block) close
2017-08-25 17:39:18.831024 7fc8c21abcc0 1 bluestore(/var/lib/ceph/osd/ceph-0) fsck finish with 0 errors in 587.459738 seconds
Congratulations! The fix is working as expected!
BTW, Ignore the weird ceph version in the log as it is coming from the non-official ceph repo.
Thanks,
Nitin
On 8/25/17, 3:40 PM, "Sage Weil" <sweil@xxxxxxxxxx> wrote:
On Fri, 25 Aug 2017, Kamble, Nitin A wrote:
> Hi Sage,
>
> I am planning to test in a normal setup. Do I need to set debug level 20
> for bluestore to see these messages?
You should see it at normal debug levels.
Thanks!
sage
>
> Nitin
>
> On 8/25/17, 3:17 PM, "ceph-devel-owner@xxxxxxxxxxxxxxx on behalf of Kamble, Nitin A" <ceph-devel-owner@xxxxxxxxxxxxxxx on behalf of Nitin.Kamble@xxxxxxxxxxxx> wrote:
>
>
>
> On 8/25/17, 3:11 PM, "Sage Weil" <sweil@xxxxxxxxxx> wrote:
> Okay, I have a patch that repairs this. See wip-21089-luminous, which
> should appear at
>
> https://shaman.ceph.com/builds/ceph/wip-21089-luminous/
>
> in the next hour or so. If you run fsck with this build you see a line
> like
>
> 2017-08-25 18:01:58.656446 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck warning: fixing leaked block 0x280000000~1000 due to old bug
> 2017-08-25 18:01:58.656451 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck freelist block and key count changed, fixing 0x280001000~7f000
>
> go by in the log on the next fsck, and no such message thereafter. Can
> you give it a try on one of those OSDs to confirm it works? I tested both
> cases I see locally but want to ensure it works in your case too.
>
> Thanks!
> sage
>
>
> Ok, will report my findings with this code.
>
> Thanks,
> Nitin
>
>
> ????{.n???+?????????????????????+%??????lzwm??????b???맲??????r??????yǩ???ׯzX?????????????ܨ}?????????Ơz???&j:+v??????????????????????zZ+??????+zf?????????h?????????~????????????i?????????z???????w??????????????????????&???)ߢ?f
>
>
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
