On Tue, 30 Apr 2019, Roman Penyaev wrote: > On 2019-04-19 16:31, Liu, Changcheng wrote: > > Hi Roman, > > I found that why ceph/msg/async/rdma/iwarp(x722) doesn't work on > > ceph master branch. > > The problem is triggered by below commit: > > > > https://github.com/ceph/ceph/pull/20172/commits/fdde016301ae329f76c621337c384ac60aa0d210 > > > > Below is the basic program model extracted from > > ceph/msg/async/rdma/iwarp to show how the problem is triggered: > > Hi Changcheng, > > Indeed fork() also changes credentials (see copy_creds() in kernel for > details), > like setuid() does, so there are two known places in ceph, after which uverbs > calls return -EACESS: > > o setuid() (see global_init()) > o daemon() (see global_init_daemonize()) > > My question is why you daemonize your ceph services and do not rely on > systemd, > which does fork() on its own and runs each service with '-f' flag, which means > do not daemonize? So I would not daemonize services and this can be a simple > solution. The daemonize behavior predates systemd (and upstart). At this point it is only there for legacy reasons and to avoid breaking things for the Devuan sysvinit hold-outs. (And vstart still daemonizes.) We could probably get away with ripping it out... sage > > With setuid() is not that easy. The most straightforward way is to move > mc_bootstrap.get_monmap_and_config() after setuid() call. At the bottom of > the email there is a small patch which can fix the problem (I hope does not > introduce something new). Would be great if you can check it. > > -- > Roman > > > diff --git a/src/global/global_init.cc b/src/global/global_init.cc > index eb8bbfd1a4db..de647be768bd 100644 > --- a/src/global/global_init.cc > +++ b/src/global/global_init.cc > @@ -147,18 +147,6 @@ void global_pre_init( > cct->_log->start(); > } > > - if (!conf->no_mon_config) { > - // make sure our mini-session gets legacy values > - conf.apply_changes(nullptr); > - > - MonClient mc_bootstrap(g_ceph_context); > - if (mc_bootstrap.get_monmap_and_config() < 0) { > - cct->_log->flush(); > - cerr << "failed to fetch mon config (--no-mon-config to skip)" > - << std::endl; > - _exit(1); > - } > - } > if (!cct->_log->is_started()) { > cct->_log->start(); > } > @@ -313,6 +301,28 @@ global_init(const std::map<std::string,std::string> > *defaults, > } > #endif > > + // > + // Utterly important to run first network connection after setuid(). > + // In case of rdma transport uverbs kernel module starts returning > + // -EACCESS on each operation if credentials has been changed, see > + // callers of ib_safe_file_access() for details. > + // > + // fork() syscall also matters, so daemonization won't work in case > + // of rdma. > + // > + if (!g_conf()->no_mon_config) { > + // make sure our mini-session gets legacy values > + g_conf().apply_changes(nullptr); > + > + MonClient mc_bootstrap(g_ceph_context); > + if (mc_bootstrap.get_monmap_and_config() < 0) { > + g_ceph_context->_log->flush(); > + cerr << "failed to fetch mon config (--no-mon-config to skip)" > + << std::endl; > + _exit(1); > + } > + } > + > > > >
