On 20-12-2016 16:06, Mykola Golub wrote: > This is due to SO_REUSEADDR (not SO_REUSEPORT) socket option set. You > should have mentioned that you were talking about FreeBSD. Hi Mykola, Sorry, I normally do. Since I know there are subtile differences. > Note, although osd-0 and osd-1 processes are bound to the same port, > they have different addresses: wildcard (*) for osd-1, and 127.0.0.1 > for rebound osd-0. On FreeBSD if SO_REUSEADDR is set, it fails to bind > only when both address and port are the same, and wildcard is > considered as a different address here. On Linux bind fails in such > case. I suspected something like this, but if I try to simulate this in a program. I still get 'address in use' when I first bind to *:port, and then to 127.0.0.1:port. So that makes it rather vague. > See, for example this for more details: > > http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t Interesting article, archived it with the other stuff I already have from the FreeBSD lists where there is also a lot off misunderstanding on this topic. > The question is though why it rebinds to 127.0.0.1, and not to '*'? I > suppose this is wrong. How does it behave on Linux? Similar: First round of binds: 2016-12-12 23:57:37.615409 7f36c2be6940 1 -- 0.0.0.0:6800/13799 _finish_bind bind my_inst.addr is 0.0.0.0:6800/13799 2016-12-12 23:57:37.615739 7f36c2be6940 1 -- 0.0.0.0:6801/13799 _finish_bind bind my_inst.addr is 0.0.0.0:6801/13799 2016-12-12 23:57:37.616090 7f36c2be6940 1 -- 0.0.0.0:6802/13799 _finish_bind bind my_inst.addr is 0.0.0.0:6802/13799 2016-12-12 23:57:37.616452 7f36c2be6940 1 -- 0.0.0.0:6803/13799 _finish_bind bind my_inst.addr is 0.0.0.0:6803/13799 So that is to INADDR_ANY rebinds: 2016-12-12 23:57:50.094446 7f36b5ac6700 1 -- 127.0.0.1:6812/1013799 _finish_bind bind my_inst.addr is 127.0.0.1:6812/1013799 2016-12-12 23:57:50.094956 7f36b5ac6700 1 -- 127.0.0.1:6813/1013799 _finish_bind bind my_inst.addr is 127.0.0.1:6813/1013799 2016-12-12 23:57:50.095477 7f36b5ac6700 1 -- 127.0.0.1:6814/1013799 _finish_bind bind my_inst.addr is 127.0.0.1:6814/1013799 so that is on the hostname as specified in the config. So your suggestion would be to not bind on INADDR_ANY but on the config hostname with the initial bind as well?? Also i got Email from Sage, stating that SO_REUSEADDR not working is not too bad, since it is mainly to prevent running out of ports when they are cycled thru high speed. --WjW > > On Tue, Dec 20, 2016 at 11:21:19AM +0100, Willem Jan Withagen wrote: >> Hi, >> >> I've been banging my head against the wall for some time now. >> But rebinding OSD.0 (in cephtool-test-mon.sh) does not quite work. >> >> When rebinding it connects to the ports of OSD.1 because those ports are >> the first not in the avoid_list. That should be refused since these >> sockets belong to a different process. >> UNLESS SO_REUSEPORT is set: >> SO_REUSEPORT allows completely duplicate bindings by multiple processes >> if they all set SO_REUSEPORT before binding the port. This option >> permits multiple instances of a program to each receive UDP/IP >> multicast or broadcast datagrams destined for the bound port. >> >> Which seems that that happens. >> Output from sockstat in this state: >> wjw ceph-osd-0 43305 14 tcp4 *:6800 *:* >> wjw ceph-osd-0 43305 15 tcp4 127.0.0.1:6804 *:* >> wjw ceph-osd-0 43305 16 tcp4 127.0.0.1:6805 *:* >> wjw ceph-osd-0 43305 45 tcp4 127.0.0.1:6806 *:* >> wjw ceph-osd-1 43318 14 tcp4 *:6804 *:* >> wjw ceph-osd-1 43318 15 tcp4 *:6805 *:* >> wjw ceph-osd-1 43318 16 tcp4 *:6806 *:* >> wjw ceph-osd-1 43318 17 tcp4 *:6807 *:* >> >> Which clearly demonstrates the mess. >> How ever that option is nowhere set in the ceph-code, neither is it a >> setting that "just" gets set. >> >> Any suggestions where to look for this option to get set in an >> incidental/bug way would be much appreciated. >> Or a suggestion on how to easily debug this. >> >> Thanx, >> --WjW >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
