On 01/09/2020 08:15, Simon Sutter wrote:
Hello again
So I have changed the network configuration.
Now my Ceph is reachable from outside, this also means all osd’s of all nodes are reachable.
I still have the same behaviour which is a timeout.
The client can resolve all nodes with their hostnames.
The mon’s are still listening on the internal network so the nat rule is still there.
I have set “public bind addr” to the external ip and restarted the mon but it’s still not working.
It could be that the NAT is the problem here.
Just use routing and firewalling. That way clients and OSDs have direct
IP-access to each other. Will make your life much easier.
Wido
[root@testnode1 ~]# ceph config get mon.public_bind_addr
WHO MASK LEVEL OPTION VALUE RO
mon advanced public_bind_addr v2:[ext-addr]:0/0 *
Do I have to change them somewhere else too?
Thanks in advance,
Simon
Von: Janne Johansson [mailto:icepic.dz@xxxxxxxxx]
Gesendet: 27 August 2020 20:01
An: Simon Sutter <ssutter@xxxxxxxxxxx>
Betreff: Re: cephfs needs access from two networks
Den tors 27 aug. 2020 kl 12:05 skrev Simon Sutter <ssutter@xxxxxxxxxxx<mailto:ssutter@xxxxxxxxxxx>>:
Hello Janne
Oh I missed that point. No, the client cannot talk directly to the osds.
In this case it’s extremely difficult to set this up.
This is an absolute requirement to be a ceph client.
How is the mon telling the client, which host and port of the osd, it should connect to?
The same port and ip that the ODS called into the mon with when it started up and joined the clusster.
Can I have an influence on it?
Well, you set the ip on the OSD hosts, and the port ranges in use for OSDs are changeable/settable, but it would not really help the above-mentioned client.
Von: Janne Johansson [mailto:icepic.dz@xxxxxxxxx<mailto:icepic.dz@xxxxxxxxx>]
Gesendet: 26 August 2020 15:09
An: Simon Sutter <ssutter@xxxxxxxxxxx<mailto:ssutter@xxxxxxxxxxx>>
Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
Betreff: Re: cephfs needs access from two networks
Den ons 26 aug. 2020 kl 14:16 skrev Simon Sutter <ssutter@xxxxxxxxxxx<mailto:ssutter@xxxxxxxxxxx>>:
Hello,
So I know, the mon services can only bind to just one ip.
But I have to make it accessible to two networks because internal and external servers have to mount the cephfs.
The internal ip is 10.99.10.1 and the external is some public-ip.
I tried nat'ing it with this: "firewall-cmd --zone=public --add-forward-port=port=6789:proto=tcp:toport=6789:toaddr=10.99.10.1 -permanent"
So the nat is working, because I get a "ceph v027" (alongside with some gibberish) when I do a telnet "telnet *public-ip* 6789"
But when I try to mount it, I get just a timeout:
mount -vvvv -t ceph *public-ip*:6789:/testing /mnt -o name=test,secretfile=/root/ceph.client. test.key
mount error 110 = Connection timed out
The tcpdump also recognizes a "Ceph Connect" packet, coming from the mon.
How can I get around this problem?
Is there something I have missed?
Any ceph client will need direct access to all OSDs involved also. Your mail doesn't really say if the cephfs-mounting client can talk to OSDs?
In ceph, traffic is not shuffled via mons, mons only tell the client which OSDs it needs to talk to, then all IO goes directly from client to any involved OSD servers.
--
May the most significant bit of your life be positive.
--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
