Ok, so images are regularly removed. In that case, these two objects probably are left over from previously removed images. Once ceph-objectstore-tool can dump the SnapSet from those two objects, you will probably find that those two snapdir objects each have only one bogus clone, in which case you'll probably just remove the images. -Sam On Thu, Aug 20, 2015 at 3:45 PM, Voloshanenko Igor <igor.voloshanenko@xxxxxxxxx> wrote: > Image? One? > > We start deleting images only to fix thsi (export/import)m before - 1-4 > times per day (when VM destroyed)... > > > > 2015-08-21 1:44 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >> >> Interesting. How often do you delete an image? I'm wondering if >> whatever this is happened when you deleted these two images. >> -Sam >> >> On Thu, Aug 20, 2015 at 3:42 PM, Voloshanenko Igor >> <igor.voloshanenko@xxxxxxxxx> wrote: >> > Sam, i try to understand which rbd contain this chunks.. but no luck. No >> > rbd >> > images block names started with this... >> > >> >> Actually, now that I think about it, you probably didn't remove the >> >> images for 3fac9490/rbd_data.eb5f22eb141f2.00000000000004ba/snapdir//2 >> >> and 22ca30c4/rbd_data.e846e25a70bf7.0000000000000307/snapdir//2 >> > >> > >> > >> > >> > 2015-08-21 1:36 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >> >> >> >> Actually, now that I think about it, you probably didn't remove the >> >> images for 3fac9490/rbd_data.eb5f22eb141f2.00000000000004ba/snapdir//2 >> >> and 22ca30c4/rbd_data.e846e25a70bf7.0000000000000307/snapdir//2, but >> >> other images (that's why the scrub errors went down briefly, those >> >> objects -- which were fine -- went away). You might want to export >> >> and reimport those two images into new images, but leave the old ones >> >> alone until you can clean up the on disk state (image and snapshots) >> >> and clear the scrub errors. You probably don't want to read the >> >> snapshots for those images either. Everything else is, I think, >> >> harmless. >> >> >> >> The ceph-objectstore-tool feature would probably not be too hard, >> >> actually. Each head/snapdir image has two attrs (possibly stored in >> >> leveldb -- that's why you want to modify the ceph-objectstore-tool and >> >> use its interfaces rather than mucking about with the files directly) >> >> '_' and 'snapset' which contain encoded representations of >> >> object_info_t and SnapSet (both can be found in src/osd/osd_types.h). >> >> SnapSet has a set of clones and related metadata -- you want to read >> >> the SnapSet attr off disk and commit a transaction writing out a new >> >> version with that clone removed. I'd start by cloning the repo, >> >> starting a vstart cluster locally, and reproducing the issue. Next, >> >> get familiar with using ceph-objectstore-tool on the osds in that >> >> vstart cluster. A good first change would be creating a >> >> ceph-objectstore-tool op that lets you dump json for the object_info_t >> >> and SnapSet (both types have format() methods which make that easy) on >> >> an object to stdout so you can confirm what's actually there. oftc >> >> #ceph-devel or the ceph-devel mailing list would be the right place to >> >> ask questions. >> >> >> >> Otherwise, it'll probably get done in the next few weeks. >> >> -Sam >> >> >> >> On Thu, Aug 20, 2015 at 3:10 PM, Voloshanenko Igor >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> > thank you Sam! >> >> > I also noticed this linked errors during scrub... >> >> > >> >> > Now all lools like reasonable! >> >> > >> >> > So we will wait for bug to be closed. >> >> > >> >> > do you need any help on it? >> >> > >> >> > I mean i can help with coding/testing/etc... >> >> > >> >> > 2015-08-21 0:52 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >> >> >> >> >> >> Ah, this is kind of silly. I think you don't have 37 errors, but 2 >> >> >> errors. pg 2.490 object >> >> >> 3fac9490/rbd_data.eb5f22eb141f2.00000000000004ba/snapdir//2 is >> >> >> missing >> >> >> snap 141. If you look at the objects after that in the log: >> >> >> >> >> >> 2015-08-20 20:15:44.865670 osd.19 10.12.2.6:6838/1861727 298 : >> >> >> cluster >> >> >> [ERR] repair 2.490 >> >> >> 68c89490/rbd_data.16796a3d1b58ba.0000000000000047/head//2 expected >> >> >> clone 2d7b9490/rbd_data.18f92c3d1b58ba.0000000000006167/141//2 >> >> >> 2015-08-20 20:15:44.865817 osd.19 10.12.2.6:6838/1861727 299 : >> >> >> cluster >> >> >> [ERR] repair 2.490 >> >> >> ded49490/rbd_data.11a25c7934d3d4.0000000000008a8a/head//2 expected >> >> >> clone 68c89490/rbd_data.16796a3d1b58ba.0000000000000047/141//2 >> >> >> >> >> >> The clone from the second line matches the head object from the >> >> >> previous line, and they have the same clone id. I *think* that the >> >> >> first error is real, and the subsequent ones are just scrub being >> >> >> dumb. Same deal with pg 2.c4. I just opened >> >> >> http://tracker.ceph.com/issues/12738. >> >> >> >> >> >> The original problem is that >> >> >> 3fac9490/rbd_data.eb5f22eb141f2.00000000000004ba/snapdir//2 and >> >> >> 22ca30c4/rbd_data.e846e25a70bf7.0000000000000307/snapdir//2 are both >> >> >> missing a clone. Not sure how that happened, my money is on a >> >> >> cache/tiering evict racing with a snap trim. If you have any >> >> >> logging >> >> >> or relevant information from when that happened, you should open a >> >> >> bug. The 'snapdir' in the two object names indicates that the head >> >> >> object has actually been deleted (which makes sense if you moved the >> >> >> image to a new image and deleted the old one) and is only being kept >> >> >> around since there are live snapshots. I suggest you leave the >> >> >> snapshots for those images alone for the time being -- removing them >> >> >> might cause the osd to crash trying to clean up the wierd on disk >> >> >> state. Other than the leaked space from those two image snapshots >> >> >> and >> >> >> the annoying spurious scrub errors, I think no actual corruption is >> >> >> going on though. I created a tracker ticket for a feature that >> >> >> would >> >> >> let ceph-objectstore-tool remove the spurious clone from the >> >> >> head/snapdir metadata. >> >> >> >> >> >> Am I right that you haven't actually seen any osd crashes or user >> >> >> visible corruption (except possibly on snapshots of those two >> >> >> images)? >> >> >> -Sam >> >> >> >> >> >> On Thu, Aug 20, 2015 at 10:07 AM, Voloshanenko Igor >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> >> > Inktank: >> >> >> > >> >> >> > >> >> >> > >> >> >> > https://download.inktank.com/docs/ICE%201.2%20-%20Cache%20and%20Erasure%20Coding%20FAQ.pdf >> >> >> > >> >> >> > Mail-list: >> >> >> > >> >> >> > https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg18338.html >> >> >> > >> >> >> > 2015-08-20 20:06 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >> >> >> >> >> >> >> >> Which docs? >> >> >> >> -Sam >> >> >> >> >> >> >> >> On Thu, Aug 20, 2015 at 9:57 AM, Voloshanenko Igor >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> >> >> > Not yet. I will create. >> >> >> >> > But according to mail lists and Inktank docs - it's expected >> >> >> >> > behaviour >> >> >> >> > when >> >> >> >> > cache enable >> >> >> >> > >> >> >> >> > 2015-08-20 19:56 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >> >> >> >> >> >> >> >> >> >> Is there a bug for this in the tracker? >> >> >> >> >> -Sam >> >> >> >> >> >> >> >> >> >> On Thu, Aug 20, 2015 at 9:54 AM, Voloshanenko Igor >> >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> >> >> >> > Issue, that in forward mode, fstrim doesn't work proper, and >> >> >> >> >> > when >> >> >> >> >> > we >> >> >> >> >> > take >> >> >> >> >> > snapshot - data not proper update in cache layer, and client >> >> >> >> >> > (ceph) >> >> >> >> >> > see >> >> >> >> >> > damaged snap.. As headers requested from cache layer. >> >> >> >> >> > >> >> >> >> >> > 2015-08-20 19:53 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >> >> >> >> >> >> >> >> >> >> >> >> What was the issue? >> >> >> >> >> >> -Sam >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Aug 20, 2015 at 9:41 AM, Voloshanenko Igor >> >> >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> >> >> >> >> > Samuel, we turned off cache layer few hours ago... >> >> >> >> >> >> > I will post ceph.log in few minutes >> >> >> >> >> >> > >> >> >> >> >> >> > For snap - we found issue, was connected with cache >> >> >> >> >> >> > tier.. >> >> >> >> >> >> > >> >> >> >> >> >> > 2015-08-20 19:23 GMT+03:00 Samuel Just >> >> >> >> >> >> > <sjust@xxxxxxxxxx>: >> >> >> >> >> >> >> >> >> >> >> >> >> >> Ok, you appear to be using a replicated cache tier in >> >> >> >> >> >> >> front >> >> >> >> >> >> >> of >> >> >> >> >> >> >> a >> >> >> >> >> >> >> replicated base tier. Please scrub both inconsistent >> >> >> >> >> >> >> pgs >> >> >> >> >> >> >> and >> >> >> >> >> >> >> post >> >> >> >> >> >> >> the >> >> >> >> >> >> >> ceph.log from before when you started the scrub until >> >> >> >> >> >> >> after. >> >> >> >> >> >> >> Also, >> >> >> >> >> >> >> what command are you using to take snapshots? >> >> >> >> >> >> >> -Sam >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Aug 20, 2015 at 3:59 AM, Voloshanenko Igor >> >> >> >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> >> >> >> >> >> > Hi Samuel, we try to fix it in trick way. >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > we check all rbd_data chunks from logs (OSD) which are >> >> >> >> >> >> >> > affected, >> >> >> >> >> >> >> > then >> >> >> >> >> >> >> > query >> >> >> >> >> >> >> > rbd info to compare which rbd consist bad rbd_data, >> >> >> >> >> >> >> > after >> >> >> >> >> >> >> > that >> >> >> >> >> >> >> > we >> >> >> >> >> >> >> > mount >> >> >> >> >> >> >> > this >> >> >> >> >> >> >> > rbd as rbd0, create empty rbd, and DD all info from >> >> >> >> >> >> >> > bad >> >> >> >> >> >> >> > volume >> >> >> >> >> >> >> > to >> >> >> >> >> >> >> > new >> >> >> >> >> >> >> > one. >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > But after that - scrub errors growing... Was 15 >> >> >> >> >> >> >> > errors.. >> >> >> >> >> >> >> > .Now >> >> >> >> >> >> >> > 35... >> >> >> >> >> >> >> > We >> >> >> >> >> >> >> > laos >> >> >> >> >> >> >> > try to out OSD which was lead, but after rebalancing >> >> >> >> >> >> >> > this >> >> >> >> >> >> >> > 2 >> >> >> >> >> >> >> > pgs >> >> >> >> >> >> >> > still >> >> >> >> >> >> >> > have >> >> >> >> >> >> >> > 35 scrub errors... >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > ceph osd getmap -o <outfile> - attached >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > 2015-08-18 18:48 GMT+03:00 Samuel Just >> >> >> >> >> >> >> > <sjust@xxxxxxxxxx>: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Is the number of inconsistent objects growing? Can >> >> >> >> >> >> >> >> you >> >> >> >> >> >> >> >> attach >> >> >> >> >> >> >> >> the >> >> >> >> >> >> >> >> whole ceph.log from the 6 hours before and after the >> >> >> >> >> >> >> >> snippet >> >> >> >> >> >> >> >> you >> >> >> >> >> >> >> >> linked above? Are you using cache/tiering? Can you >> >> >> >> >> >> >> >> attach >> >> >> >> >> >> >> >> the >> >> >> >> >> >> >> >> osdmap >> >> >> >> >> >> >> >> (ceph osd getmap -o <outfile>)? >> >> >> >> >> >> >> >> -Sam >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Aug 18, 2015 at 4:15 AM, Voloshanenko Igor >> >> >> >> >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> >> >> >> >> >> >> > ceph - 0.94.2 >> >> >> >> >> >> >> >> > Its happen during rebalancing >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > I thought too, that some OSD miss copy, but looks >> >> >> >> >> >> >> >> > like >> >> >> >> >> >> >> >> > all >> >> >> >> >> >> >> >> > miss... >> >> >> >> >> >> >> >> > So any advice in which direction i need to go >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > 2015-08-18 14:14 GMT+03:00 Gregory Farnum >> >> >> >> >> >> >> >> > <gfarnum@xxxxxxxxxx>: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From a quick peek it looks like some of the OSDs >> >> >> >> >> >> >> >> >> are >> >> >> >> >> >> >> >> >> missing >> >> >> >> >> >> >> >> >> clones >> >> >> >> >> >> >> >> >> of >> >> >> >> >> >> >> >> >> objects. I'm not sure how that could happen and >> >> >> >> >> >> >> >> >> I'd >> >> >> >> >> >> >> >> >> expect >> >> >> >> >> >> >> >> >> the >> >> >> >> >> >> >> >> >> pg >> >> >> >> >> >> >> >> >> repair to handle that but if it's not there's >> >> >> >> >> >> >> >> >> probably >> >> >> >> >> >> >> >> >> something >> >> >> >> >> >> >> >> >> wrong; what version of Ceph are you running? Sam, >> >> >> >> >> >> >> >> >> is >> >> >> >> >> >> >> >> >> this >> >> >> >> >> >> >> >> >> something >> >> >> >> >> >> >> >> >> you've seen, a new bug, or some kind of config >> >> >> >> >> >> >> >> >> issue? >> >> >> >> >> >> >> >> >> -Greg >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Aug 18, 2015 at 6:27 AM, Voloshanenko Igor >> >> >> >> >> >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >> >> >> >> >> >> >> >> >> > Hi all, at our production cluster, due high >> >> >> >> >> >> >> >> >> > rebalancing >> >> >> >> >> >> >> >> >> > ((( >> >> >> >> >> >> >> >> >> > we >> >> >> >> >> >> >> >> >> > have 2 >> >> >> >> >> >> >> >> >> > pgs in >> >> >> >> >> >> >> >> >> > inconsistent state... >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > root@temp:~# ceph health detail | grep inc >> >> >> >> >> >> >> >> >> > HEALTH_ERR 2 pgs inconsistent; 18 scrub errors >> >> >> >> >> >> >> >> >> > pg 2.490 is active+clean+inconsistent, acting >> >> >> >> >> >> >> >> >> > [56,15,29] >> >> >> >> >> >> >> >> >> > pg 2.c4 is active+clean+inconsistent, acting >> >> >> >> >> >> >> >> >> > [56,10,42] >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > From OSD logs, after recovery attempt: >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > root@test:~# ceph pg dump | grep -i incons | cut >> >> >> >> >> >> >> >> >> > -f >> >> >> >> >> >> >> >> >> > 1 >> >> >> >> >> >> >> >> >> > | >> >> >> >> >> >> >> >> >> > while >> >> >> >> >> >> >> >> >> > read >> >> >> >> >> >> >> >> >> > i; >> >> >> >> >> >> >> >> >> > do >> >> >> >> >> >> >> >> >> > ceph pg repair ${i} ; done >> >> >> >> >> >> >> >> >> > dumped all in format plain >> >> >> >> >> >> >> >> >> > instructing pg 2.490 on osd.56 to repair >> >> >> >> >> >> >> >> >> > instructing pg 2.c4 on osd.56 to repair >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:51:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.035910 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > f5759490/rbd_data.1631755377d7e.00000000000004da/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > 90c59490/rbd_data.eb486436f2beb.0000000000007a65/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:52:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.035960 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > fee49490/rbd_data.12483d3ba0794b.000000000000522f/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > f5759490/rbd_data.1631755377d7e.00000000000004da/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:53:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.036133 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > a9b39490/rbd_data.12483d3ba0794b.00000000000037b3/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > fee49490/rbd_data.12483d3ba0794b.000000000000522f/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:54:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.036243 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > bac19490/rbd_data.1238e82ae8944a.000000000000032e/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > a9b39490/rbd_data.12483d3ba0794b.00000000000037b3/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:55:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.036289 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > 98519490/rbd_data.123e9c2ae8944a.0000000000000807/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > bac19490/rbd_data.1238e82ae8944a.000000000000032e/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:56:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.036314 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > c3c09490/rbd_data.1238e82ae8944a.0000000000000c2b/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > 98519490/rbd_data.123e9c2ae8944a.0000000000000807/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:57:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.036363 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > 28809490/rbd_data.edea7460fe42b.00000000000001d9/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > c3c09490/rbd_data.1238e82ae8944a.0000000000000c2b/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:58:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:37.036432 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub >> >> >> >> >> >> >> >> >> > 2.490 >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > e1509490/rbd_data.1423897545e146.00000000000009a6/head//2 >> >> >> >> >> >> >> >> >> > expected >> >> >> >> >> >> >> >> >> > clone >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > 28809490/rbd_data.edea7460fe42b.00000000000001d9/141//2 >> >> >> >> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:59:2015-08-18 >> >> >> >> >> >> >> >> >> > 07:26:38.548765 >> >> >> >> >> >> >> >> >> > 7f94663b3700 >> >> >> >> >> >> >> >> >> > -1 >> >> >> >> >> >> >> >> >> > log_channel(cluster) log [ERR] : 2.490 >> >> >> >> >> >> >> >> >> > deep-scrub >> >> >> >> >> >> >> >> >> > 17 >> >> >> >> >> >> >> >> >> > errors >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > So, how i can solve "expected clone" situation >> >> >> >> >> >> >> >> >> > by >> >> >> >> >> >> >> >> >> > hand? >> >> >> >> >> >> >> >> >> > Thank in advance! >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > _______________________________________________ >> >> >> >> >> >> >> >> >> > ceph-users mailing list >> >> >> >> >> >> >> >> >> > ceph-users@xxxxxxxxxxxxxx >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> > >> > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |