Re: CephFS concurrency question

Neville <neville.taylor@xxxxxxxxxxxxx> · Tue, 21 Apr 2015 16:13:55 +0100

Hi Robert,

I'm running the latest version of Icehouse - 2014.1.4. I know there was some talk of patching nova but not sure if this was ever done, if it was I assume it wasn't backported for Icehouse. I'm not specifically worried about needing the instances folder on shared storage to get it working as I was keen to try out CephFS anyway, it's more a question of whether what I've done should work in theory and trying to find out why it doesn't.

Thanks,

Neville

Date: Tue, 21 Apr 2015 08:26:15 -0600
Subject: Re:  CephFS concurrency question
From: robert@xxxxxxxxxxxxx
To: neville.taylor@xxxxxxxxxxxxx
CC: ceph-users@xxxxxxxxxxxxxx

I think your are using an old version of OpenStack. I seem to remember a discussion about a patch to remove the requirement of shared storage for live migration on Ceph RBD. Are you using librbd in open stack? 
Robert LeBlanc
Sent from a mobile device please excuse any typos.
On Apr 21, 2015 6:43 AM, "Neville" <neville.taylor@xxxxxxxxxxxxx> wrote:

I'm trying to setup live migration in Openstack using Ceph RBD backed volumes. From what I understand I also need to put the libvirt folder /var/lib/nova/instances on shared storage for it to work as Nova tests for this as part of the migration process. I decided to look at using CephFS for this purpose.

I've created a CephFS file system and mounted it on my two compute nodes as /var/lib/nova/instances but I'm getting some strange results. It seems like after one of the hosts accesses a file then it becomes locked and the other host can't access it. For example, I complete the following steps:

1\ Create new Openstack instance using the boot from image (create new volume) option. New instance is created with RBD backed volume as expected. On the relevant compute host I see the instance folder created under /var/lib/nova/instances/{instance id} with two files inside libvirt.xml and console.log. If I cat the libvirt.xml file it works as expected.
2\ Live migrate the instance to other host. Appears to work as expected although instance status in Horizon stays as migrating forever. I can see instance has moved to second host by running virsh list on both hosts.
4\ Now, if I attempt to cat the libvirt.xml file on the new host I get "Operation not permitted".

I'm assuming this isn't what's expected?

To test this further I tried the following basic tests:

On Host 2:

root@devops-kvm02:/var/lib/nova/instances# echo hello > test
root@devops-kvm02:/var/lib/nova/instances# cat test
hello
root@devops-kvm02:/var/lib/nova/instances#

Then from Host 1:

root@devops-kvm01:/var/lib/nova/instances# cat test
cat: test: Operation not permitted
root@devops-kvm01:/var/lib/nova/instances#

Then back on Host 2:

root@devops-kvm02:/var/lib/nova/instances# cat test
cat: test: Operation not permitted
root@devops-kvm02:/var/lib/nova/instances#

Should this even work? My understanding is CephFS allows concurrent access but I'm not sure if there is some file locking going on that I need to understand.

Thanks,

Neville

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com