Re: mounting RBD in linux containers

Omar Marquez <Omar.Marquez@xxxxxxxxxxxxxxx> · Thu, 17 Oct 2013 22:49:19 +0000

Strace produces below:

…

futex(0xb5637c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xb56378, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0xb562f8, FUTEX_WAKE_PRIVATE, 1)  = 1
add_key(0x424408, 0x7fff82c4e210, 0x7fff82c4e140, 0x22, 0xfffffffe) = 607085216
stat("/sys/bus/rbd", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
open("/sys/bus/rbd/add", O_WRONLY)      = 3
write(3, "10.198.41.6:6789,10.198.41.8:678"..., 96) = -1 EINVAL (Invalid argument)
close(3)                                = 0
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7fbf8a7efa90}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7fbf8a7efa90}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fff82c4e040) = 22
wait4(22, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fbf8a7efa90}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fbf8a7efa90}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
write(2, "rbd: add failed: ", 17rbd: add failed: )       = 17
write(2, "(22) Invalid argument", 21(22) Invalid argument)   = 21
write(2, "\n", 1
)                       = 1
exit_group(1)                           = ?
+++ exited with 1 +++

The app is run inside the container with setuid = 0 and the container is able to mount all required filesystems … could this still be a capability problem ? Also I do not see any call to capset() in the strafe log …

--
Om

From: Kyle Bader <kyle.bader@xxxxxxxxx>

Date: Thursday, October 17, 2013 5:08 PM

To: Kevin Weiler <Kevin.Weiler@xxxxxxxxxxxxxxx>

Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>, Omar Marquez <omar.marquez@xxxxxxxxxxxxxxx>,
 Khalid Goudeaux <Khalid.Goudeaux@xxxxxxxxxxxxxxx>

Subject: Re:  mounting RBD in linux containers

My first guess would be that it's due to LXC dropping capabilities, I'd investigate whether CAP_SYS_ADMIN is being dropped. You need CAP_SYS_ADMIN for mount and block ioctls, if the container doesn't have those privs a map will likely
 fail. Maybe try tracing the command with strace?

On Thu, Oct 17, 2013 at 2:45 PM, Kevin Weiler 
<Kevin.Weiler@xxxxxxxxxxxxxxx> wrote:

Hi all,

We're trying to mount an rbd image inside of a linux container that has been created with docker (https://www.docker.io/). We seem to have access to the rbd kernel module from inside the container:

# lsmod | grep ceph
libceph               218854  1 rbd
libcrc32c              12603  3 xfs,libceph,dm_persistent_data

And we can query the pool for available rbds and create rbds from inside the container:

# rbd -p dockers --id dockers --keyring /etc/ceph/ceph.client.dockers.keyring create lxctest --size 51200
# rbd -p dockers --id dockers --keyring /etc/ceph/ceph.client.dockers.keyring ls
lxctest

But for some reason, we can't seem to map the device to the container:

# rbd -p dockers --id dockers --keyring /etc/ceph/ceph.client.dockers.keyring map lxctest
rbd: add failed: (22) Invalid argument

I don't see anything particularly interesting in dmesg or messages on either the container or the host box. Any ideas on how to troubleshoot this?

Thanks!

-- 

Kevin Weiler

IT

IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | Chicago, IL 60606 | http://imc-chicago.com/

Phone:
+1 312-204-7439 | Fax:
+1 312-244-3301 | E-Mail: kevin.weiler@xxxxxxxxxxxxxxx

The information in this e-mail is intended only for the person or entity to which it is addressed.

It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it.

If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities
 shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect
 transmission, please return the e-mail to the sender and permanently delete this message and any attachments.

Messages and attachments are scanned for all known viruses. Always scan attachments before opening them.

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 

Kyle 

The information in this e-mail is intended only for the person or entity to which it is addressed.

It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it.

If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities
 shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect
 transmission, please return the e-mail to the sender and permanently delete this message and any attachments.

Messages and attachments are scanned for all known viruses. Always scan attachments before opening them.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com