That's awesome Josh. Thanks for looking into it. Good luck with the fix! - Travis On Fri, Mar 22, 2013 at 1:11 PM, Josh Durgin <josh.durgin@xxxxxxxxxxx> wrote: > I think I found the root cause based on your logs: > > http://tracker.ceph.com/issues/4531 > > Josh > > > On 03/20/2013 02:47 PM, Travis Rhoden wrote: >> >> Didn't take long to re-create with the detailed debugging (ms = 20). >> I'm sending Josh a link to the gzip'd log off-list, I"m not sure if >> the log will contain any CephX keys or anything like that. >> >> On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@xxxxxxxxx> wrote: >>> >>> Thanks Josh. I will respond when I have something useful! >>> >>> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@xxxxxxxxxxx> >>> wrote: >>>> >>>> On 03/20/2013 01:19 PM, Josh Durgin wrote: >>>>> >>>>> >>>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>>> In this case, they are format 2. And they are from cloned snapshots. >>>>>>> Exactly like the following: >>>>>>> >>>>>>> # rbd ls -l -p volumes >>>>>>> NAME SIZE >>>>>>> PARENT FMT PROT LOCK >>>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8 5120M >>>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap 2 >>>>>>> >>>>>>> I'm doing an OpenStack boot-from-volume setup. >>>>>> >>>>>> >>>>>> >>>>>> OK i've never used cloned snapshots so maybe this is the reason. >>>>>> >>>>>>>> strange i've never seen this. Which qemu version? >>>>>>> >>>>>>> >>>>>>> >>>>>>> # qemu-x86_64 -version >>>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 >>>>>>> Fabrice Bellard >>>>>>> >>>>>>> that's coming from Ubuntu 12.04 apt repos. >>>>>> >>>>>> >>>>>> >>>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm >>>>>> does >>>>>> not exist anymore it was merged into qemu with 1.3 or 1.4. >>>>> >>>>> >>>>> >>>>> This particular problem won't be solved by upgrading qemu. It's a ceph >>>>> bug. Disabling caching would work around the issue. >>>>> >>>>> Travis, could you get a log from qemu of this happening with: >>>>> >>>>> debug ms = 20 >>>>> debug objectcacher = 20 >>>>> debug rbd = 20 >>>>> log file = /path/writeable/by/qemu >>>> >>>> >>>> >>>> If it doesn't reproduce with those settings, try changing debug ms to 1 >>>> instead of 20. >>>> >>>> >>>>> From those we can tell whether the issue is on the client side at >>>>> least, >>>>> and hopefully what's causing it. >>>>> >>>>> Thanks! >>>>> Josh >>>> >>>> >>>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
