--
*Kevin Weiler*
IT
IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | Chicago, IL
60606 | http://imc-chicago.com/
Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail:
_kevin.weiler@xxxxxxxxxxxxxxx <mailto:Kevin.Weiler@xxxxxxxxxxxxxxx>_
From: Gregory Farnum <greg@xxxxxxxxxxx <mailto:greg@xxxxxxxxxxx>>
Date: Friday, October 18, 2013 10:26 AM
To: Omar Marquez <Omar.Marquez@xxxxxxxxxxxxxxx
<mailto:Omar.Marquez@xxxxxxxxxxxxxxx>>
Cc: Kyle Bader <kyle.bader@xxxxxxxxx <mailto:kyle.bader@xxxxxxxxx>>,
Kevin Weiler <kevin.weiler@xxxxxxxxxxxxxxx
<mailto:kevin.weiler@xxxxxxxxxxxxxxx>>, "ceph-users@xxxxxxxxxxxxxx
<mailto:ceph-users@xxxxxxxxxxxxxx>" <ceph-users@xxxxxxxxxxxxxx
<mailto:ceph-users@xxxxxxxxxxxxxx>>, Khalid Goudeaux
<Khalid.Goudeaux@xxxxxxxxxxxxxxx <mailto:Khalid.Goudeaux@xxxxxxxxxxxxxxx>>
Subject: Re: mounting RBD in linux containers
What kernel are you running, and which format is the RBD image? I
thought we had a special return code for when the kernel doesn't support
the features used by that image, but that could be the problem.
-Greg
On Thursday, October 17, 2013, Omar Marquez wrote:
Strace produces below:
…
futex(0xb5637c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xb56378,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0xb562f8, FUTEX_WAKE_PRIVATE, 1) = 1
add_key(0x424408, 0x7fff82c4e210, 0x7fff82c4e140, 0x22,
0xfffffffe) = 607085216
stat("/sys/bus/rbd", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
*open("/sys/bus/rbd/add", O_WRONLY) = 3*
*write(3, "10.198.41.6:6789
<http://10.198.41.6:6789>,10.198.41.8:678
<http://10.198.41.8:678>"..., 96) = -1 EINVAL (Invalid argument)*
close(3) = 0
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7fbf8a7efa90},
{SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER,
0x7fbf8a7efa90}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD,
parent_tidptr=0x7fff82c4e040) = 22
wait4(22, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fbf8a7efa90},
NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER,
0x7fbf8a7efa90}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
write(2, "rbd: add failed: ", 17rbd: add failed: ) = 17
write(2, "(22) Invalid argument", 21(22) Invalid argument) = 21
write(2, "\n", 1
) = 1
exit_group(1) = ?
+++ exited with 1 +++
The app is run inside the container with setuid = 0 and the
container is able to mount all required filesystems … could this
still be a capability problem ? Also I do not see any call to
capset() in the strafe log …
--
Om
From: Kyle Bader <kyle.bader@xxxxxxxxx>
Date: Thursday, October 17, 2013 5:08 PM
To: Kevin Weiler <Kevin.Weiler@xxxxxxxxxxxxxxx>
Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>, Omar
Marquez <omar.marquez@xxxxxxxxxxxxxxx>, Khalid Goudeaux
<Khalid.Goudeaux@xxxxxxxxxxxxxxx>
Subject: Re: mounting RBD in linux containers
My first guess would be that it's due to LXC dropping capabilities,
I'd investigate whether CAP_SYS_ADMIN is being dropped. You need
CAP_SYS_ADMIN for mount and block ioctls, if the container doesn't
have those privs a map will likely fail. Maybe try tracing the
command with strace?
On Thu, Oct 17, 2013 at 2:45 PM, Kevin Weiler
<Kevin.Weiler@xxxxxxxxxxxxxxx> wrote:
Hi all,
We're trying to mount an rbd image inside of a linux container
that has been created with docker (https://www.docker.io/). We
seem to have access to the rbd kernel module from inside the
container:
# lsmod | grep ceph
libceph 218854 1 rbd
libcrc32c 12603 3 xfs,libceph,dm_persistent_data
And we can query the pool for available rbds and create rbds
from inside the container:
# rbd -p dockers --id dockers --keyring
/etc/ceph/ceph.client.dockers.keyring create lxctest --size 51200
# rbd -p dockers --id dockers --keyring
/etc/ceph/ceph.client.dockers.keyring ls
lxctest
But for some reason, we can't seem to map the device to the
container:
# rbd -p dockers --id dockers --keyring
/etc/ceph/ceph.client.dockers.keyring map lxctest
rbd: add failed: (22) Invalid argument
I don't see anything particularly interesting in dmesg or
messages on either the container or the host box. Any ideas on
how to troubleshoot this?
Thanks!
--
*Kevin Weiler*
IT
IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 |
Chicago, IL 60606 | http://imc-chicago.com/
Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail:
_kevin.weiler@imc-chicago.com_
------------------------------------------------------------------------
The information in this e-mail is intended only for the person
or entity to which it is addressed.
It may contain confidential and /or privileged material. If
someone other than the intended recipient should receive this
e-mail, he / she shall not be entitled to read, disseminate,
disclose or duplicate it.
If you receive this e-mail unintentionally, please inform us
immediately by "re
------------------------------------------------------------------------
The information in this e-mail is intended only for the person or
entity to which it is addressed.
It may contain confidential and /or privileged material. If someone
other than the intended recipient should receive this e-mail, he /
she shall not be entitled to read, disseminate, disclose or
duplicate it.
If you receive this e-mail unintentionally, please inform us
immediately by "reply" and then delete it from your system. Although
this information has been compiled with great care, neither IMC
Financial Markets & Asset Management nor any of its related entities
shall accept any responsibility for any errors, omissions or other
inaccuracies in this information or for the consequences thereof,
nor shall it be bound in any way by the contents of this e-mail or
its attachments. In the event of incomplete or incorrect
transmission, please return the e-mail to the sender and permanently
delete this message and any attachments.
Messages and attachments are scanned for all known viruses. Always
scan attachments before opening them.
--
Software Engineer #42 @ http://inktank.com | http://ceph.com
------------------------------------------------------------------------
The information in this e-mail is intended only for the person or entity
to which it is addressed.
It may contain confidential and /or privileged material. If someone
other than the intended recipient should receive this e-mail, he / she
shall not be entitled to read, disseminate, disclose or duplicate it.
If you receive this e-mail unintentionally, please inform us immediately
by "reply" and then delete it from your system. Although this
information has been compiled with great care, neither IMC Financial
Markets & Asset Management nor any of its related entities shall accept
any responsibility for any errors, omissions or other inaccuracies in
this information or for the consequences thereof, nor shall it be bound
in any way by the contents of this e-mail or its attachments. In the
event of incomplete or incorrect transmission, please return the e-mail
to the sender and permanently delete this message and any attachments.
Messages and attachments are scanned for all known viruses. Always scan
attachments before opening them.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
