Re: Broken snapshots... CEPH 0.94.2

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Odd, did you happen to capture osd logs?
-Sam

On Thu, Aug 20, 2015 at 8:10 PM, Ilya Dryomov <idryomov@xxxxxxxxx> wrote:
> On Fri, Aug 21, 2015 at 2:02 AM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>> What's supposed to happen is that the client transparently directs all
>> requests to the cache pool rather than the cold pool when there is a
>> cache pool.  If the kernel is sending requests to the cold pool,
>> that's probably where the bug is.  Odd.  It could also be a bug
>> specific 'forward' mode either in the client or on the osd.  Why did
>> you have it in that mode?
>
> I think I reproduced this on today's master.
>
> Setup, cache mode is writeback:
>
> $ ./ceph osd pool create foo 12 12
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> pool 'foo' created
> $ ./ceph osd pool create foo-hot 12 12
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> pool 'foo-hot' created
> $ ./ceph osd tier add foo foo-hot
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> pool 'foo-hot' is now (or already was) a tier of 'foo'
> $ ./ceph osd tier cache-mode foo-hot writeback
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> set cache-mode for pool 'foo-hot' to writeback
> $ ./ceph osd tier set-overlay foo foo-hot
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> overlay for 'foo' is now (or already was) 'foo-hot'
>
> Create an image:
>
> $ ./rbd create --size 10M --image-format 2 foo/bar
> $ sudo ./rbd-fuse -p foo -c $PWD/ceph.conf /mnt
> $ sudo mkfs.ext4 /mnt/bar
> $ sudo umount /mnt
>
> Create a snapshot, take md5sum:
>
> $ ./rbd snap create foo/bar@snap
> $ ./rbd export foo/bar /tmp/foo-1
> Exporting image: 100% complete...done.
> $ ./rbd export foo/bar@snap /tmp/snap-1
> Exporting image: 100% complete...done.
> $ md5sum /tmp/foo-1
> 83f5d244bb65eb19eddce0dc94bf6dda  /tmp/foo-1
> $ md5sum /tmp/snap-1
> 83f5d244bb65eb19eddce0dc94bf6dda  /tmp/snap-1
>
> Set the cache mode to forward and do a flush, hashes don't match - the
> snap is empty - we bang on the hot tier and don't get redirected to the
> cold tier, I suspect:
>
> $ ./ceph osd tier cache-mode foo-hot forward
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> set cache-mode for pool 'foo-hot' to forward
> $ ./rados -p foo-hot cache-flush-evict-all
> rbd_data.100a6b8b4567.0000000000000002
> rbd_id.bar
> rbd_directory
> rbd_header.100a6b8b4567
> bar.rbd
> rbd_data.100a6b8b4567.0000000000000001
> rbd_data.100a6b8b4567.0000000000000000
> $ ./rados -p foo-hot cache-flush-evict-all
> $ ./rbd export foo/bar /tmp/foo-2
> Exporting image: 100% complete...done.
> $ ./rbd export foo/bar@snap /tmp/snap-2
> Exporting image: 100% complete...done.
> $ md5sum /tmp/foo-2
> 83f5d244bb65eb19eddce0dc94bf6dda  /tmp/foo-2
> $ md5sum /tmp/snap-2
> f1c9645dbc14efddc7d8a322685f26eb  /tmp/snap-2
> $ od /tmp/snap-2
> 0000000 000000 000000 000000 000000 000000 000000 000000 000000
> *
> 50000000
>
> Disable the cache tier and we are back to normal:
>
> $ ./ceph osd tier remove-overlay foo
> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
> there is now (or already was) no overlay for 'foo'
> $ ./rbd export foo/bar /tmp/foo-3
> Exporting image: 100% complete...done.
> $ ./rbd export foo/bar@snap /tmp/snap-3
> Exporting image: 100% complete...done.
> $ md5sum /tmp/foo-3
> 83f5d244bb65eb19eddce0dc94bf6dda  /tmp/foo-3
> $ md5sum /tmp/snap-3
> 83f5d244bb65eb19eddce0dc94bf6dda  /tmp/snap-3
>
> I first reproduced it with the kernel client, rbd export was just to
> take it out of the equation.
>
>
> Also, Igor sort of raised a question in his second message: if, after
> setting the cache mode to forward and doing a flush, I open an image
> (not a snapshot, so may not be related to the above) for write (e.g.
> with rbd-fuse), I get an rbd header object in the hot pool, even though
> it's in forward mode:
>
> $ sudo ./rbd-fuse -p foo -c $PWD/ceph.conf /mnt
> $ sudo mount /mnt/bar /media
> $ sudo umount /media
> $ sudo umount /mnt
> $ ./rados -p foo-hot ls
> rbd_header.100a6b8b4567
> $ ./rados -p foo ls | grep rbd_header
> rbd_header.100a6b8b4567
>
> It's been a while since I looked into tiering, is that how it's
> supposed to work?  It looks like it happens because rbd_header op
> replies don't redirect?
>
> Thanks,
>
>                 Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux