> > - Although I can mount the export I can't write on it > > What error are you getting trying to do the write? The way you set things up doesn't look to different than one of our integration tests for ingress over nfs ( https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress.yaml) and that test tests a simple read/write to the export after creating/mounting it. > - I can't understand how to use the sdc disks for journaling > > you should be able to specify a `journal_devices` section in an OSD spec. For example *service_type: osdservice_id: fooplacement: hosts: - vm-00spec: data_devices: paths: - /dev/vdb journal_devices: paths: - /dev/vdc* that will make non-colocated OSDs where the devices from the journal_devices section are used as journal devices for the OSDs on the devices in the data_devices section. Although I'd recommend looking through https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications <https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and see if there are any other filtering options than the path that can be used first. It's possible the path the device gets can change on reboot and you could end up with cepadm using a device you don't want it to for this as that other device gets the path another device held previously. - I can't understand the concept of "pseudo path" > I don't know at a low level either, but it seems to just be the path nfs-ganesha will present to the user. There is another argument to `ceph nfs export create` which is just "path" rather than pseudo-path that marks what actual path within the cephfs the export is mounted on. It's optional and defaults to "/" (so the export you made is mounted at the root of the fs). I think that's the one that really matters. The pseudo-path seems to just act like a user facing name for the path. On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian <debian108@xxxxxxxxx> wrote: > Hi you all, > > I'm almost new to ceph and I'm understanding, day by day, why the > official support is so expansive :) > > > I setting up a ceph nfs network cluster whose recipe can be found here > below. > > ####################### > > --> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 > --cluster-network 10.20.20.0/24 --fsid $FSID --initial-dashboard-user adm > \ > --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate > --allow-fqdn-hostname --ssl-dashboard-port 443 \ > --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt --dashboard-key > /etc/ssl/wildcard.it/wildcard.it.key \ > --allow-overwrite --cleanup-on-failure > cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k > /etc/ceph/ceph.client.admin.keyring > cephadm add-repo --release reef && cephadm install ceph-common > --> adding hosts and set labels > for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f > -i /etc/ceph/ceph.pub root@$IP ; done > ceph orch host add cephstage01 10.20.20.81 --labels > _admin,mon,mgr,prometheus,grafana > ceph orch host add cephstage02 10.20.20.82 --labels > _admin,mon,mgr,prometheus,grafana > ceph orch host add cephstage03 10.20.20.83 --labels > _admin,mon,mgr,prometheus,grafana > ceph orch host add cephstagedatanode01 10.20.20.84 --labels > osd,nfs,prometheus > ceph orch host add cephstagedatanode02 10.20.20.85 --labels > osd,nfs,prometheus > ceph orch host add cephstagedatanode03 10.20.20.86 --labels > osd,nfs,prometheus > --> network setup and daemons deploy > ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 > ceph orch apply mon > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" > ceph orch apply mgr > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" > ceph orch apply prometheus > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" > ceph orch apply grafana > > --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" > ceph orch apply node-exporter > ceph orch apply alertmanager > ceph config set mgr mgr/cephadm/secure_monitoring_stack true > --> disks and osd setup > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do > ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f > /dev/sdc"; done > ceph config set mgr mgr/cephadm/device_enhanced_scan true > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > doceph orch device ls --hostname=$IP --wide --refresh ; done > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > doceph orch device zap $IP /dev/sdb; done > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > doceph orch device zap $IP /dev/sdc ; done > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > doceph orch daemon add osd $IP:/dev/sdb ; done > for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; > doceph orch daemon add osd $IP:/dev/sdc ; done > --> ganesha nfs cluster > ceph mgr module enable nfs > ceph fs volume create vol1 > ceph nfs cluster create nfs-cephfs > "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress > --virtual-ip 192.168.7.80 --ingress-mode default > ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt > --fsname vol1 > --> nfs mount > mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph > > > is my recipe correct? > > > the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the > latters I installed one 3tb ssd, for the data, and one 300gb ssd for the > journaling but > > my problems are : > > - Although I can mount the export I can't write on it > > - I can't understand how to use the sdc disks for journaling > > - I can't understand the concept of "pseudo path" > > > here below you can find the json output of the exports > > --> check > ceph nfs export ls nfs-cephfs > ceph nfs export info nfs-cephfs /mnt > ------------------------------------ > json file > --------- > { > "export_id": 1, > "path": "/", > "cluster_id": "nfs-cephfs", > "pseudo": "/mnt", > "access_type": "RW", > "squash": "none", > "security_label": true, > "protocols": [ > 4 > ], > "transports": [ > "TCP" > ], > "fsal": { > "name": "CEPH", > "user_id": "nfs.nfs-cephfs.1", > "fs_name": "vol1" > }, > "clients": [] > } > ------------------------------------ > > > Thanks in advance > > Rob > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |