On Mon, May 30, 2016 at 8:33 PM, Ilya Dryomov <idryomov@xxxxxxxxx> wrote:
> On Mon, May 30, 2016 at 4:12 PM, Jens Offenbach <wolle5050@xxxxxx> wrote:
>> Hallo,
>> in my OpenStack Mitaka, I have installed the additional service "Manila" with a CephFS backend. Everything is working. All shares are created successfully:
>>
>> manila show 9dd24065-97fb-4bcd-9ad1-ca63d40bf3a8
>> +-----------------------------+------------------------------------------------------------------------------------------------------------------------+
>> | Property | Value |
>> +-----------------------------+------------------------------------------------------------------------------------------------------------------------+
>> | status | available |
>> | share_type_name | cephfs |
>> | description | None |
>> | availability_zone | nova |
>> | share_network_id | None |
>> | export_locations | |
>> | | path = 10.152.132.71:6789,10.152.132.72:6789,10.152.132.73:6789:/volumes/_nogroup/b27ad01a-245f-49e2-8974-1ed0ce8e259e |
>> | | preferred = False |
>> | | is_admin_only = False |
>> | | id = 9b7d7e9e-d661-4fa0-89d7-9727efb75554 |
>> | | share_instance_id = b27ad01a-245f-49e2-8974-1ed0ce8e259e |
>> | share_server_id | None |
>> | host | os-sharedfs@cephfs#cephfs |
>> | access_rules_status | active |
>> | snapshot_id | None |
>> | is_public | False |
>> | task_state | None |
>> | snapshot_support | True |
>> | id | 9dd24065-97fb-4bcd-9ad1-ca63d40bf3a8 |
>> | size | 1 |
>> | name | cephshare1 |
>> | share_type | 2a62fda4-82ce-4798-9a85-c800736b01e5 |
>> | has_replicas | False |
>> | replication_type | None |
>> | created_at | 2016-05-30T13:09:11.000000 |
>> | share_proto | CEPHFS |
>> | consistency_group_id | None |
>> | source_cgsnapshot_member_id | None |
>> | project_id | cf03dbf6f7d04ff6bda0c65cb8395ded |
>> | metadata | {} |
>> +-----------------------------+------------------------------------------------------------------------------------------------------------------------+
>>
>> When I try to mount the share into a client machine via the kernel driver:
>> mount -t ceph os-storage01,os-storage02,os-storage03:/volumes/_nogroup/b27ad01a-245f-49e2-8974-1ed0ce8e259e /tmp/test -o name=testclient,secret=AQDKO0xX6UgQNhAAe/9nJH1RGes0VzTZ+I04eQ==
>>
>> I get the following error:
>> mount error 5 = Input/output error
>>
>> When I use "ceph-fuse" without any further modifications, everything is working:
>> ceph-fuse /tmp/test --id=testclient --conf=/etc/ceph/ceph.conf --keyring=/etc/ceph/ceph.client.testclient.keyring --client-mountpoint=/volumes/_nogroup/b27ad01a-245f-49e2-8974-1ed0ce8e259e
>>
>> I have tried to mount the "volumes" folder with kernel driver:
>> mount -t ceph os-storage01,os-storage02,os-storage03:/volumes /tmp/test -o name=testclient,secret=AQDKO0xX6UgQNhAAe/9nJH1RGes0VzTZ+I04eQ==
>>
>> It works so far, but when I try to list the folder's content, I get the "Input/output error":
>> cd /tmp/test/_nogroup
>> ls
>> ls: cannot access '4f6f4f58-6a54-4300-8cb3-6d3a7debad61': Input/output error
>> ls: cannot access '83de097f-4fd4-4168-82e3-697dab6fa645': Input/output error
>> ls: cannot access '9a27fbd1-5446-47a3-8de5-0db57f624d09': Input/output error
>> ls: cannot access 'b27ad01a-245f-49e2-8974-1ed0ce8e259e': Input/output error
>> 4f6f4f58-6a54-4300-8cb3-6d3a7debad61 83de097f-4fd4-4168-82e3-697dab6fa645 9a27fbd1-5446-47a3-8de5-0db57f624d09 b27ad01a-245f-49e2-8974-1ed0ce8e259e
>>
>> I am using Ubuntu Server 16.04 (Xenial) with the current kernel "4.4.0-23-generic".
>
> I think this is because the kernel client doesn't support cephfs
> namespaces yet. Mitaka shares created with the cephfs native driver
> use them instead of RADOS pools for security isolation.
>
> As a workaround, I was going to suggest "cephfs:data_isolated=true"
> when creating shares, so that a separate RADOS pool is created instead
> of a namespace within the existing pool. Pools are much more expensive
> than namespaces in terms of cluster resources though, so it's viable
> only if you are going to have a small number of shares.
>
> However, it seems that ceph.dir.layout.pool_namespace is set even if
> a separate pool is created:
>
> https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L472
>
> This seems like over-doing to me. It also prevents the above
> workaround from working. John, can we stick an else in there and fixup
> {de,}authorize()?
Using the kernel client with Manila is already a bit of a halfway
solution, because it doesn't have the quota enforcement, so it won't
respect the size of the Manila shares. We have rados namespace
support in the latest kclient code (in ceph-clients:testing), so this
will eventually make it into released kernels.
In the meantime, I don't think it's worth the complexity of extending
the Manila interfaces to make namespaces optional, and using
data_isolated is pretty undesirable anyway due to the overhead of pool
creation.
John
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
