One more question.....did we hit the bug because we had hardware issues during the remapping or would it have happened regardless of the hardware issues? e.g. I'm not planning to add any additional hardware soon, but would the bug pop again on an (unpatched) system not subject to any remapping?
thanks,
jeff
On Tue, Mar 15, 2016 at 7:27 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
[back on list]
ceph-objectstore-tool has a whole bunch of machinery for modifying an
offline objectstore. It would be the easiest place to put it -- you
could add a
ceph-objectstore-tool --op filestore-repair-orphan-links [--dry-run] ...
command which would mount the filestore in a special mode and iterate
over all collections and repair them. If you want to go that route,
we'd be happy to help you get it written. Once it fixes your cluster,
we'd then be able to merge and backport it in case anyone else hits
it.
You'd probably be fine doing it while the OSD is live...but as a rule
I usually prefer to do my osd surgery offline. Journal doesn't matter
here, the orphaned files are basically invisible to the filestore
(except when doing a collection scan for scrub) since they are in the
wrong directory.
I don't think the orphans are necessarily going to be 0 size. There
might be quirk of how radosgw creates these objects that always causes
them to be created 0 size than then overwritten with a writefull -- if
that's true it might be the case that you would only see 0 size ones.
-Sam
On Tue, Mar 15, 2016 at 4:02 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote:
> Thanks, I can try to write a tool to do this. Does ceph-objectstore-tool
> provide a framework?
>
> Can I safely delete the files while the OSD is alive or should I take it
> offline? Any concerns about the journal?
>
> Are there any other properties of the orphans, e.g. will the orphans always
> be size 0?
>
> Thanks!
> Jeff
>
> On Tue, Mar 15, 2016 at 5:35 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>
>> Ok, a branch merged to master which should fix this
>> (https://github.com/ceph/ceph/pull/8136). It'll be backported in due
>> course. The problem is that that patch won't clean orphaned files
>> that already exist.
>>
>> Let me explain a bit about what the orphaned files look like. The
>> problem is files with object names that result in escaped filenames
>> longer than the max filename ceph will create (~250 iirc). Normally,
>> the name of the file is an escaped and sanitized version of the object
>> name:
>>
>>
>> ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/default.325674.107\u\ushadow\u.KePEE8heghHVnlb1\uEIupG0I5eROwRn\u77__head_C1DCD459__46_ffffffffffffffff_0
>>
>> corresponds to an object like
>>
>>
>> c1dcd459/default.325674.107__shadow_.KePEE8heghHVnlb1_EIupG0I5eROwRn_77/head//70
>>
>> the DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/ path is derived from the hash
>> starting with the last value: cd459 -> DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/
>>
>> It ends up in DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/ because that's the
>> longest path that exists (DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/DIR_D does not
>> exist -- if DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/ ever gets too full,
>> DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/DIR_D would be created and this file
>> would be moved into it).
>>
>> When the escaped filename gets too long, we truncate the filename, and
>> then append a hash and a number yielding a name like:
>>
>>
>> ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long
>>
>> The _long at the end is always present with files like this.
>> fa202ec9b4b3b217275a is the hash of the filename. The 0 indicates
>> that it's the 0th file with this prefix and this hash -- if there are
>> hash collisions with the same prefix, you'll see _1_ and _2_ and so on
>> to distinguish them (very very unlikely). When the filename has been
>> truncated as with this one, you will find the full file name in the
>> attrs (attr user.cephosd.lfn3):
>>
>>
>> ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long:
>> user.cephos.lfn3:
>>
>> default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkLf.4\u156__head_79CED459__46_ffffffffffffffff_0
>>
>> Let's look at one of the orphaned files (the one with the same
>> file-name as the previous one, actually):
>>
>>
>> ./DIR_9/DIR_5/DIR_4/DIR_D/default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long:
>> user.cephos.lfn3:
>>
>> default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkLf.4\u156__head_79CED459__46_3189d_0
>>
>> This one has the same filename as the previous object, but is an
>> orphan. What makes it an orphan is that it has hash 79CED459, but is
>> in ./DIR_9/DIR_5/DIR_4/DIR_D even though
>> ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_E exists (objects-files are always at
>> the farthest directory from the root matching their hash). All of the
>> orphans will be long-file-name objects (but most long-file-name
>> objects are fine and are neither orphans nor have duplicates -- it's a
>> fairly low occurrence bug). In your case, I think *all* of the
>> orphans will probably happen to have files with duplicate names in the
>> correct directory -- though might not if the object had actually been
>> deleted since the bug happened. When there are duplicates, the full
>> object names will either be the same or differ by the generation
>> number at the end (ffffffffffffffff_0 vs 3189d_0) in this case.
>>
>> Once the orphaned files are cleaned up, your cluster should be back to
>> normal. If you want to wait, someone might get time to build a patch
>> for ceph-objectstore-tool to automate this. You can try removing the
>> orphan we identified in pg 70.459 and re-scrubbing to confirm that
>> that fixes the pg.
>> -Sam
>>
>> On Wed, Mar 9, 2016 at 6:58 AM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote:
>> > Hi, I went back to the mon logs to see if I could illicit any additional
>> > information about this PG.
>> > Prior to 1/27/16, the deep-scrub on this OSD passes(then I see obsolete
>> > rollback objects found):
>> >
>> > ceph.log.4.gz:2016-01-20 09:43:36.195640 osd.307 10.31.0.67:6848/127170
>> > 538
>> > : cluster [INF] 70.459 deep-scrub ok
>> > ceph.log.4.gz:2016-01-27 09:51:49.952459 osd.307 10.31.0.67:6848/127170
>> > 583
>> > : cluster [INF] 70.459 deep-scrub starts
>> > ceph.log.4.gz:2016-01-27 10:10:57.196311 osd.108 10.31.0.69:6816/4283
>> > 335 :
>> > cluster [ERR] osd.108 pg 70.459s5 found obsolete rollback obj
>> >
>> > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/5
>> > generation < trimmed_to 130605'206504...repaired
>> > ceph.log.4.gz:2016-01-27 10:10:57.043942 osd.307 10.31.0.67:6848/127170
>> > 584
>> > : cluster [ERR] osd.307 pg 70.459s0 found obsolete rollback obj
>> >
>> > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/0
>> > generation < trimmed_to 130605'206504...repaired
>> > ceph.log.4.gz:2016-01-27 10:10:58.225017 osd.307 10.31.0.67:6848/127170
>> > 585
>> > : cluster [ERR] 70.459s0 shard 4(3) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-27 10:10:58.225068 osd.307 10.31.0.67:6848/127170
>> > 586
>> > : cluster [ERR] 70.459s0 shard 10(2) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-27 10:10:58.225088 osd.307 10.31.0.67:6848/127170
>> > 587
>> > : cluster [ERR] 70.459s0 shard 26(1) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-27 10:10:58.225127 osd.307 10.31.0.67:6848/127170
>> > 588
>> > : cluster [ERR] 70.459s0 shard 132(4) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-27 10:13:52.926032 osd.307 10.31.0.67:6848/127170
>> > 589
>> > : cluster [ERR] 70.459s0 deep-scrub stat mismatch, got 21324/21323
>> > objects,
>> > 0/0 clones, 21324/21323 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0
>> > whiteouts,
>> > 64313094166/64308899862 bytes,0/0 hit_set_archive bytes.
>> > ceph.log.4.gz:2016-01-27 10:13:52.927589 osd.307 10.31.0.67:6848/127170
>> > 590
>> > : cluster [ERR] 70.459s0 deep-scrub 1 missing, 0 inconsistent objects
>> > ceph.log.4.gz:2016-01-27 10:13:52.931250 osd.307 10.31.0.67:6848/127170
>> > 591
>> > : cluster [ERR] 70.459 deep-scrub 5 errors
>> > ceph.log.4.gz:2016-01-28 10:32:37.083809 osd.307 10.31.0.67:6848/127170
>> > 592
>> > : cluster [INF] 70.459 repair starts
>> > ceph.log.4.gz:2016-01-28 10:51:44.608297 osd.307 10.31.0.67:6848/127170
>> > 593
>> > : cluster [ERR] osd.307 pg 70.459s0 found obsolete rollback obj
>> >
>> > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/0
>> > generation < trimmed_to 130605'206504...repaired
>> > ceph.log.4.gz:2016-01-28 10:51:45.802549 osd.307 10.31.0.67:6848/127170
>> > 594
>> > : cluster [ERR] 70.459s0 shard 4(3) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-28 10:51:45.802933 osd.307 10.31.0.67:6848/127170
>> > 595
>> > : cluster [ERR] 70.459s0 shard 10(2) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-28 10:51:45.802978 osd.307 10.31.0.67:6848/127170
>> > 596
>> > : cluster [ERR] 70.459s0 shard 26(1) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-28 10:51:45.803039 osd.307 10.31.0.67:6848/127170
>> > 597
>> > : cluster [ERR] 70.459s0 shard 132(4) missing
>> >
>> > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70
>> > ceph.log.4.gz:2016-01-28 10:51:44.781639 osd.108 10.31.0.69:6816/4283
>> > 338 :
>> > cluster [ERR] osd.108 pg 70.459s5 found obsolete rollback obj
>> >
>> > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/5
>> > generation < trimmed_to 130605'206504...repaired
>> > ceph.log.4.gz:2016-01-28 11:01:18.119350 osd.26 10.31.0.103:6812/77378
>> > 2312
>> > : cluster [INF] 70.459s1 restarting backfill on osd.305(0) from
>> > (0'0,0'0]
>> > MAX to 130605'206506
>> > ceph.log.4.gz:2016-02-01 13:40:55.096030 osd.307 10.31.0.67:6848/13421
>> > 16 :
>> > cluster [INF] 70.459s0 restarting backfill on osd.210(1) from (0'0,0'0]
>> > MAX
>> > to 135195'206996
>> > ceph.log.4.gz:2016-02-01 13:41:10.623892 osd.307 10.31.0.67:6848/13421
>> > 27 :
>> > cluster [INF] 70.459s0 restarting backfill on osd.25(1) from (0'0,0'0]
>> > MAX
>> > to 135195'206996
>> > ...
>> >
>> >
>> > Regards,
>> > Jeff
>> >
>> >
>> > --
>> >
>> > Jeffrey McDonald, PhD
>> > Assistant Director for HPC Operations
>> > Minnesota Supercomputing Institute
>> > University of Minnesota Twin Cities
>> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>> > 117 Pleasant St SE phone: +1 612 625-6905
>> > Minneapolis, MN 55455 fax: +1 612 624-8861
>> >
>> >
>
>
>
>
> --
>
> Jeffrey McDonald, PhD
> Assistant Director for HPC Operations
> Minnesota Supercomputing Institute
> University of Minnesota Twin Cities
> 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
> 117 Pleasant St SE phone: +1 612 625-6905
> Minneapolis, MN 55455 fax: +1 612 624-8861
>
>
Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
