Hi Josh,
On 03/14/2012 10:59 PM, Josh Durgin wrote:
On 03/14/2012 01:49 PM, Oliver Francke wrote:
Well,
nobody able to sched some light in?
Did some math and found out how to fill the size bytes.
Sorry I didn't respond faster.
But, one question never got answered:
- why is - with busy VMs - frequently the first block affected,
with the result of damaged
grub-loaders/partition-tables/filesystems?
Is this some NULL/zero pointer thingy in case of ceph-failure?
My guess is that this is not the first object affected, but it's where
the loss of an object is most easily noticeable - if an object doesn't
exist, it's treated as being full of zeros, which might go undetected
for a long time if it's e.g. some temp or log file that's not reread
and verified.
well, I responded to Sage with some more infos from one of the images
where the header is missing... Did not want to bother the list ;)
If you demand some broken images… we have many of them to investigate,
unfortunately.
We'd really like to find the root cause of the problem. One
possibility is some bad interaction between osds running different
versions. This caused one issue with recovery stxShadow saw yesterday,
for example (http://tracker.newdream.net/issues/2132). Had you been
doing rolling upgrades of osds before these problems appeared? If so,
do you know which versions you had running concurrently?
Are your osds often restarting?
What we'd need to diagnose this are osd logs during recovery with:
debug osd = 20
debug ms = 1
Once you detect the problem, a log from each replica storing the pg
the bad/missing object is in should be enough.
And just to make sure, you aren't writing to these rbd images from
multiple places, right? This wouldn't cause the missing header
objects, but is likely to cause corruption of the image data. This
could happen, for example, by rolling an image back to a snapshot
while a vm is running on it.
Currently we don't use snapshots. And of course ensure, a VM is running
once at a time ;-) And we had some "rolling upgrade", but this was
_after_ trouble/crashes occured.
Oliver.
Josh
Maybe this sounds a bit harsh, after the 5th night-shift trying to
repair images
and keep customers calm, I think this is forgivable.
Oliver.
Am 14.03.2012 um 16:05 schrieb Oliver Francke:
Hey,
anybody out there who could explain the structure of a rbd-header?
After
last crash we have about 10 images with a:
2012-03-14 15:22:47.998790 7f45a61e3760 librbd: Error reading
header: 2 No such file or directory
error opening image vm-266-disk-1.rbd: 2 No such file or directory
... error?
I understand the "rb.x.y"-prefix, the 2 ^ 16hex as block-size. But
the size/count encoding is not intuitive ;)
Besides one file, where I "created" a header and putted it via "rados
put" back into the pool, and got some files
back, many of the other images with lost headers have different sizes.
We got bad luck again, too many crashed VM's, too much data-loss...
Comments welcome ;)
Oliver.
--
To unsubscribe from this list: send the line "unsubscribe
ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Oliver Francke
filoo GmbH
Moltkestraße 25a
33330 Gütersloh
HRB4355 AG Gütersloh
Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz
Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
