>>If I understand the QEMU docs correctly, cache=unsafe would immediately
>>ack the guest's fsync() - at the risk of data losss if the QEMU process
>>crashes.
That's not true with rbd block driver.
The qemu cache options only set rbd_cache=true|false
rbd.c
if (flags & BDRV_O_NOCACHE) {
rados_conf_set(s->cluster, "rbd_cache", "false");
} else {
rados_conf_set(s->cluster, "rbd_cache", "true");
}
bloc.c
int bdrv_parse_cache_flags(const char *mode, int *flags)
{
*flags &= ~BDRV_O_CACHE_MASK;
if (!strcmp(mode, "off") || !strcmp(mode, "none")) {
*flags |= BDRV_O_NOCACHE | BDRV_O_CACHE_WB;
} else if (!strcmp(mode, "directsync")) {
*flags |= BDRV_O_NOCACHE;
} else if (!strcmp(mode, "writeback")) {
*flags |= BDRV_O_CACHE_WB;
} else if (!strcmp(mode, "unsafe")) {
*flags |= BDRV_O_CACHE_WB;
*flags |= BDRV_O_NO_FLUSH;
} else if (!strcmp(mode, "writethrough")) {
/* this is the default */
} else {
return -1;
}
return 0;
}
so
cache=writethrough|writeback|unsafe = rbd_cache=true
cache=none|directsync = rbd_cache_false
----- Mail original -----
De: "Daniel Swarbrick" <daniel.swarbrick@xxxxxxxxxxxxxxxx>
À: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
Envoyé: Mardi 9 Juin 2015 10:05:21
Objet: Re: rbd cache + libvirt
I presume that since QEMU 1.2+ sets the default cache mode to writeback
if not otherwise specified, and since giant sets rbd_cache to true if
not otherwise specified, then the result should be to cache?
We have a fair number of VMs running on hosts where we don't specify
either explicitly, and I've always had the feeling that its _not_
caching... saving a small text file in the VM (e.g. with vim) always
seems to take much longer than it should - but I wonder if that's
because vim is doing an fsync().
If I understand the QEMU docs correctly, cache=unsafe would immediately
ack the guest's fsync() - at the risk of data losss if the QEMU process
crashes.
On 09/06/15 06:59, Alexandre DERUMIER wrote:
> oops, sorry, my bad, I had wrong settings when testing.
>
> you are right, remove rbd_cache from ceph.conf is enough to remove overloading
>
>
>
> host conf : no value : guest cache=writeback : result : cache
> host conf : rbd_cache=false : guest cache=writeback : result : nocache (wrong)
> host conf : rbd_cache=true : guest cache=writeback : result : cache
> host conf : no value : guest cache=none : result : nocache
> host conf : rbd_cache=false : guest cache=none : result : no cache
> host conf : rbd_cache=true : guest cache=none : result : cache (wrong)
>
>
> ----- Mail original -----
> De: "aderumier" <aderumier@xxxxxxxxx>
> À: "Jason Dillaman" <dillaman@xxxxxxxxxx>
> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
> Envoyé: Mardi 9 Juin 2015 06:33:49
> Objet: Re: rbd cache + libvirt
>
> previous matrix was with ceph < giant
>
>
> with ceph =>giant, rbd_cache=true by default, so cache=none not working if a ceph.conf exist.
>
>
> host conf : no value : guest cache=writeback : result : cache
> host conf : rbd_cache=false : guest cache=writeback : result : nocache (wrong)
> host conf : rbd_cache=true : guest cache=writeback : result : cache
> host conf : no value : guest cache=none : result : cache (wrong)
> host conf : rbd_cache=false : guest cache=none : result : no cache
> host conf : rbd_cache=true : guest cache=none : result : cache (wrong)
>
>
> ----- Mail original -----
> De: "aderumier" <aderumier@xxxxxxxxx>
> À: "Jason Dillaman" <dillaman@xxxxxxxxxx>
> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
> Envoyé: Mardi 9 Juin 2015 06:23:06
> Objet: Re: rbd cache + libvirt
>
>>> In the short-term, you can remove the "rbd cache" setting from your ceph.conf
>
> That's not true, you need to remove the ceph.conf file.
> Removing rbd_cache is not enough or default rbd_cache=false will apply.
>
>
> I have done tests, here the result matrix
>
>
> host ceph.conf : no rbd_cache : guest cache=writeback : result : nocache (wrong)
> host ceph.conf : rbd_cache=false : guest cache=writeback : result : nocache (wrong)
> host ceph.conf : rbd_cache=true : guest cache=writeback : result : cache
> host ceph.conf : no rbd_cache : guest cache=none : result : nocache
> host ceph.conf : rbd_cache=false : guest cache=none : result : no cache
> host ceph.conf : rbd_cache=true : guest cache=none : result : cache (wrong)
>
>
>
> ----- Mail original -----
> De: "Jason Dillaman" <dillaman@xxxxxxxxxx>
> À: "Andrey Korolyov" <andrey@xxxxxxx>
> Cc: "Josh Durgin" <jdurgin@xxxxxxxxxx>, "aderumier" <aderumier@xxxxxxxxx>, "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
> Envoyé: Lundi 8 Juin 2015 22:29:10
> Objet: Re: rbd cache + libvirt
>
>> On Mon, Jun 8, 2015 at 10:43 PM, Josh Durgin <jdurgin@xxxxxxxxxx> wrote:
>>> On 06/08/2015 11:19 AM, Alexandre DERUMIER wrote:
>>>>
>>>> Hi,
>>>>>>
>>>>>> looking at the latest version of QEMU,
>>>>
>>>>
>>>> It's seem that it's was already this behaviour since the add of rbd_cache
>>>> parsing in rbd.c by josh in 2012
>>>>
>>>>
>>>> http://git.qemu.org/?p=qemu.git;a=blobdiff;f=block/rbd.c;h=eebc3344620058322bb53ba8376af4a82388d277;hp=1280d66d3ca73e552642d7a60743a0e2ce05f664;hb=b11f38fcdf837c6ba1d4287b1c685eb3ae5351a8;hpb=166acf546f476d3594a1c1746dc265f1984c5c85
>>>>
>>>>
>>>> I'll do tests on my side tomorrow to be sure.
>>>
>>>
>>> It seems like we should switch the order so ceph.conf is overridden by
>>> qemu's cache settings. I don't remember a good reason to have it the
>>> other way around.
>>>
>>> Josh
>>>
>>
>> Erm, doesn`t this code *already* represent the right priorities?
>> Cache=none setting should set a BDRV_O_NOCACHE which is effectively
>> disabling cache in a mentioned snippet.
>>
>
> Yes, the override is applied (correctly) based upon your QEMU cache settings. However, it then reads your configuration file and re-applies the "rbd_cache" setting based upon what is in the file (if it exists). So in the case where a configuration file has "rbd cache = true", the override of "rbd cache = false" derived from your QEMU cache setting would get wiped out. The long term solution would be to, as Josh noted, switch the order (so long as there wasn't a use-case for applying values in this order). In the short-term, you can remove the "rbd cache" setting from your ceph.conf so that QEMU controls it (i.e. it cannot get overridden when reading the configuration file) or use a different ceph.conf for a drive which requires different cache settings from the default configuration's settings.
>
> Jason
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
