On 2019-04-19 16:31, Liu, Changcheng wrote:
Hi Roman,
I found that why ceph/msg/async/rdma/iwarp(x722) doesn't work on
ceph master branch.
The problem is triggered by below commit:
https://github.com/ceph/ceph/pull/20172/commits/fdde016301ae329f76c621337c384ac60aa0d210
Below is the basic program model extracted from
ceph/msg/async/rdma/iwarp to show how the problem is triggered:
Hi Changcheng,
Indeed fork() also changes credentials (see copy_creds() in kernel for
details),
like setuid() does, so there are two known places in ceph, after which
uverbs
calls return -EACESS:
o setuid() (see global_init())
o daemon() (see global_init_daemonize())
My question is why you daemonize your ceph services and do not rely on
systemd,
which does fork() on its own and runs each service with '-f' flag, which
means
do not daemonize? So I would not daemonize services and this can be a
simple
solution.
With setuid() is not that easy. The most straightforward way is to move
mc_bootstrap.get_monmap_and_config() after setuid() call. At the bottom
of
the email there is a small patch which can fix the problem (I hope does
not
introduce something new). Would be great if you can check it.
--
Roman
diff --git a/src/global/global_init.cc b/src/global/global_init.cc
index eb8bbfd1a4db..de647be768bd 100644
--- a/src/global/global_init.cc
+++ b/src/global/global_init.cc
@@ -147,18 +147,6 @@ void global_pre_init(
cct->_log->start();
}
- if (!conf->no_mon_config) {
- // make sure our mini-session gets legacy values
- conf.apply_changes(nullptr);
-
- MonClient mc_bootstrap(g_ceph_context);
- if (mc_bootstrap.get_monmap_and_config() < 0) {
- cct->_log->flush();
- cerr << "failed to fetch mon config (--no-mon-config to skip)"
- << std::endl;
- _exit(1);
- }
- }
if (!cct->_log->is_started()) {
cct->_log->start();
}
@@ -313,6 +301,28 @@ global_init(const std::map<std::string,std::string>
*defaults,
}
#endif
+ //
+ // Utterly important to run first network connection after setuid().
+ // In case of rdma transport uverbs kernel module starts returning
+ // -EACCESS on each operation if credentials has been changed, see
+ // callers of ib_safe_file_access() for details.
+ //
+ // fork() syscall also matters, so daemonization won't work in case
+ // of rdma.
+ //
+ if (!g_conf()->no_mon_config) {
+ // make sure our mini-session gets legacy values
+ g_conf().apply_changes(nullptr);
+
+ MonClient mc_bootstrap(g_ceph_context);
+ if (mc_bootstrap.get_monmap_and_config() < 0) {
+ g_ceph_context->_log->flush();
+ cerr << "failed to fetch mon config (--no-mon-config to skip)"
+ << std::endl;
+ _exit(1);
+ }
+ }
+
