Re: cephfs needs access from two networks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 01/09/2020 08:15, Simon Sutter wrote:
Hello again

So I have changed the network configuration.
Now my Ceph is reachable from outside, this also means all osd’s of all nodes are reachable.
I still have the same behaviour which is a timeout.

The client can resolve all nodes with their hostnames.
The mon’s are still listening on the internal network so the nat rule is still there.
I have set “public bind addr” to the external ip and restarted the mon but it’s still not working.

It could be that the NAT is the problem here.

Just use routing and firewalling. That way clients and OSDs have direct IP-access to each other. Will make your life much easier.

Wido


[root@testnode1 ~]# ceph config get mon.public_bind_addr
WHO     MASK  LEVEL     OPTION            VALUE                        RO
mon           advanced  public_bind_addr  v2:[ext-addr]:0/0         *

Do I have to change them somewhere else too?

Thanks in advance,
Simon


Von: Janne Johansson [mailto:icepic.dz@xxxxxxxxx]
Gesendet: 27 August 2020 20:01
An: Simon Sutter <ssutter@xxxxxxxxxxx>
Betreff: Re:  cephfs needs access from two networks

Den tors 27 aug. 2020 kl 12:05 skrev Simon Sutter <ssutter@xxxxxxxxxxx<mailto:ssutter@xxxxxxxxxxx>>:
Hello Janne

Oh I missed that point. No, the client cannot talk directly to the osds.
In this case it’s extremely difficult to set this up.

This is an absolute requirement to be a ceph client.

How is the mon telling the client, which host and port of the osd, it should connect to?

The same port and ip that the ODS called into the mon with when it started up and joined the clusster.

Can I have an influence on it?


Well, you set the ip on the OSD hosts, and the port ranges in use for OSDs are changeable/settable, but it would not really help the above-mentioned client.

Von: Janne Johansson [mailto:icepic.dz@xxxxxxxxx<mailto:icepic.dz@xxxxxxxxx>]
Gesendet: 26 August 2020 15:09
An: Simon Sutter <ssutter@xxxxxxxxxxx<mailto:ssutter@xxxxxxxxxxx>>
Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
Betreff: Re:  cephfs needs access from two networks

Den ons 26 aug. 2020 kl 14:16 skrev Simon Sutter <ssutter@xxxxxxxxxxx<mailto:ssutter@xxxxxxxxxxx>>:
Hello,
So I know, the mon services can only bind to just one ip.
But I have to make it accessible to two networks because internal and external servers have to mount the cephfs.
The internal ip is 10.99.10.1 and the external is some public-ip.
I tried nat'ing it  with this: "firewall-cmd --zone=public --add-forward-port=port=6789:proto=tcp:toport=6789:toaddr=10.99.10.1 -permanent"

So the nat is working, because I get a "ceph v027" (alongside with some gibberish) when I do a telnet "telnet *public-ip* 6789"
But when I try to mount it, I get just a timeout:
mount -vvvv -t ceph *public-ip*:6789:/testing /mnt -o name=test,secretfile=/root/ceph.client. test.key
mount error 110 = Connection timed out

The tcpdump also recognizes a "Ceph Connect" packet, coming from the mon.

How can I get around this problem?
Is there something I have missed?

Any ceph client will need direct access to all OSDs involved also. Your mail doesn't really say if the cephfs-mounting client can talk to OSDs?

In ceph, traffic is not shuffled via mons, mons only tell the client which OSDs it needs to talk to, then all IO goes directly from client to any involved OSD servers.

--
May the most significant bit of your life be positive.


--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux