I'm not using Ceph Ganesha but GPFS Ganesha, so YMMV > ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt > --fsname vol1 > --> nfs mount > mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph > > - Although I can mount the export I can't write on it You created a export, But there are no Clients setup https://docs.ceph.com/en/latest/mgr/nfs/#create-cephfs-export [--client_addr <value>...] It should be fine to set this to a ip range like/24 /8 or whatever you should endup with a client config like this https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/config_samples/export.txt#L264 > - I can't understand the concept of "pseudo path" Lets say your mount your exported fs to /mnt your pseudo path is /mnt, When you now mount on the machine you will end up with a folder /mnt/mnt We use this to seperate diffrent producs in gpfs. So we have a pseudo path set to /products/something, and on the machines we have the NFS mounted under /corp. the combination of both comes then to /corp/products/something On Thu, 25 Apr 2024 at 16:55, Roberto Maggi @ Debian <debian108@xxxxxxxxx> wrote: > first of all thanks to all! > > > Al supposed by Robert Sander, I get "permission denied" but even writing > with root privileges I get the same error. > > As soon as I can I'll test your suggestions and update the thread. > > > Thanks again > > On 4/24/24 16:05, Adam King wrote: > > > > - Although I can mount the export I can't write on it > > > > What error are you getting trying to do the write? The way you set > > things up doesn't look to different than one of our integration tests > > for ingress over nfs > > ( > https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress.yaml) > > > and that test tests a simple read/write to the export after > > creating/mounting it. > > > > - I can't understand how to use the sdc disks for journaling > > > > > > you should be able to specify a `journal_devices` section in an OSD > > spec. For example > > > > *service_type: osd > > service_id: foo > > placement: > > hosts: > > - vm-00 > > spec: > > data_devices: > > paths: > > - /dev/vdb > > journal_devices: > > paths: > > - /dev/vdc* > > that will make non-colocated OSDs where the devices from the > > journal_devices section are used as journal devices for the OSDs on > > the devices in the data_devices section. Although I'd recommend > > looking through > > > https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications > > < > https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and > > > see if there are any other filtering options than the path that can be > > used first. It's possible the path the device gets can change on > > reboot and you could end up with cepadm using a device you don't want > > it to for this as that other device gets the path another device held > > previously. > > > > - I can't understand the concept of "pseudo path" > > > > > > I don't know at a low level either, but it seems to just be the path > > nfs-ganesha will present to the user. There is another argument to > > `ceph nfs export create` which is just "path" rather than pseudo-path > > that marks what actual path within the cephfs the export is mounted > > on. It's optional and defaults to "/" (so the export you made is > > mounted at the root of the fs). I think that's the one that really > > matters. The pseudo-path seems to just act like a user facing name for > > the path. > > > > On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian > > <debian108@xxxxxxxxx> wrote: > > > > Hi you all, > > > > I'm almost new to ceph and I'm understanding, day by day, why the > > official support is so expansive :) > > > > > > I setting up a ceph nfs network cluster whose recipe can be found > > here > > below. > > > > ####################### > > > > --> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 > > --cluster-network 10.20.20.0/24 <http://10.20.20.0/24> --fsid > > $FSID --initial-dashboard-user adm \ > > --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate > > --allow-fqdn-hostname --ssl-dashboard-port 443 \ > > --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt > > <http://wildcard.it/wildcard.it.crt> --dashboard-key > > /etc/ssl/wildcard.it/wildcard.it.key > > <http://wildcard.it/wildcard.it.key> \ > > --allow-overwrite --cleanup-on-failure > > cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k > > /etc/ceph/ceph.client.admin.keyring > > cephadm add-repo --release reef && cephadm install ceph-common > > --> adding hosts and set labels > > for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do > > ssh-copy-id -f > > -i /etc/ceph/ceph.pub root@$IP ; done > > ceph orch host add cephstage01 10.20.20.81 --labels > > _admin,mon,mgr,prometheus,grafana > > ceph orch host add cephstage02 10.20.20.82 --labels > > _admin,mon,mgr,prometheus,grafana > > ceph orch host add cephstage03 10.20.20.83 --labels > > _admin,mon,mgr,prometheus,grafana > > ceph orch host add cephstagedatanode01 10.20.20.84 --labels > > osd,nfs,prometheus > > ceph orch host add cephstagedatanode02 10.20.20.85 --labels > > osd,nfs,prometheus > > ceph orch host add cephstagedatanode03 10.20.20.86 --labels > > osd,nfs,prometheus > > --> network setup and daemons deploy > > ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 > > <http://10.20.20.0/24,192.168.7.0/24> > > ceph orch apply mon > > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" > > ceph orch apply mgr > > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" > > ceph orch apply prometheus > > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" > > ceph orch apply grafana > > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" > > ceph orch apply node-exporter > > ceph orch apply alertmanager > > ceph config set mgr mgr/cephadm/secure_monitoring_stack true > > --> disks and osd setup > > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do > > ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f > > /dev/sdc"; done > > ceph config set mgr mgr/cephadm/device_enhanced_scan true > > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > > doceph orch device ls --hostname=$IP --wide --refresh ; done > > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > > doceph orch device zap $IP /dev/sdb; done > > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > > doceph orch device zap $IP /dev/sdc ; done > > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > > doceph orch daemon add osd $IP:/dev/sdb ; done > > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > > doceph orch daemon add osd $IP:/dev/sdc ; done > > --> ganesha nfs cluster > > ceph mgr module enable nfs > > ceph fs volume create vol1 > > ceph nfs cluster create nfs-cephfs > > "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" > > --ingress > > --virtual-ip 192.168.7.80 --ingress-mode default > > ceph nfs export create cephfs --cluster-id nfs-cephfs > > --pseudo-path /mnt > > --fsname vol1 > > --> nfs mount > > mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph > > > > > > is my recipe correct? > > > > > > the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the > > latters I installed one 3tb ssd, for the data, and one 300gb ssd > > for the > > journaling but > > > > my problems are : > > > > - Although I can mount the export I can't write on it > > > > - I can't understand how to use the sdc disks for journaling > > > > - I can't understand the concept of "pseudo path" > > > > > > here below you can find the json output of the exports > > > > --> check > > ceph nfs export ls nfs-cephfs > > ceph nfs export info nfs-cephfs /mnt > > ------------------------------------ > > json file > > --------- > > { > > "export_id": 1, > > "path": "/", > > "cluster_id": "nfs-cephfs", > > "pseudo": "/mnt", > > "access_type": "RW", > > "squash": "none", > > "security_label": true, > > "protocols": [ > > 4 > > ], > > "transports": [ > > "TCP" > > ], > > "fsal": { > > "name": "CEPH", > > "user_id": "nfs.nfs-cephfs.1", > > "fs_name": "vol1" > > }, > > "clients": [] > > } > > ------------------------------------ > > > > > > Thanks in advance > > > > Rob > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |