Hi Josh, We did map it directly to the host, and it seems to work just fine. I think this is a problem with how the container is accessing the rbd module. -- Kevin Weiler IT IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | Chicago, IL 60606 | http://imc-chicago.com/ Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail: kevin.weiler@xxxxxxxxxxxxxxx On 10/18/13 7:50 PM, "Josh Durgin" <josh.durgin@xxxxxxxxxxx> wrote: >On 10/18/2013 10:04 AM, Kevin Weiler wrote: >> The kernel is 3.11.4-201.fc19.x86_64, and the image format is 1. I did, >> however, try a map with an RBD that was format 2. I got the same error. > >To rule out any capability drops as the culprit, can you map an rbd >image on the same host outside of a container? > >Josh > >> -- >> >> *Kevin Weiler* >> >> IT >> >> IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | Chicago, IL >> 60606 | http://imc-chicago.com/ >> >> Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail: >> _kevin.weiler@xxxxxxxxxxxxxxx <mailto:Kevin.Weiler@xxxxxxxxxxxxxxx>_ >> >> >> From: Gregory Farnum <greg@xxxxxxxxxxx <mailto:greg@xxxxxxxxxxx>> >> Date: Friday, October 18, 2013 10:26 AM >> To: Omar Marquez <Omar.Marquez@xxxxxxxxxxxxxxx >> <mailto:Omar.Marquez@xxxxxxxxxxxxxxx>> >> Cc: Kyle Bader <kyle.bader@xxxxxxxxx <mailto:kyle.bader@xxxxxxxxx>>, >> Kevin Weiler <kevin.weiler@xxxxxxxxxxxxxxx >> <mailto:kevin.weiler@xxxxxxxxxxxxxxx>>, "ceph-users@xxxxxxxxxxxxxx >> <mailto:ceph-users@xxxxxxxxxxxxxx>" <ceph-users@xxxxxxxxxxxxxx >> <mailto:ceph-users@xxxxxxxxxxxxxx>>, Khalid Goudeaux >> <Khalid.Goudeaux@xxxxxxxxxxxxxxx >><mailto:Khalid.Goudeaux@xxxxxxxxxxxxxxx>> >> Subject: Re: mounting RBD in linux containers >> >> What kernel are you running, and which format is the RBD image? I >> thought we had a special return code for when the kernel doesn't support >> the features used by that image, but that could be the problem. >> -Greg >> >> On Thursday, October 17, 2013, Omar Marquez wrote: >> >> >> Strace produces below: >> >> Š >> >> futex(0xb5637c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xb56378, >> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 >> futex(0xb562f8, FUTEX_WAKE_PRIVATE, 1) = 1 >> add_key(0x424408, 0x7fff82c4e210, 0x7fff82c4e140, 0x22, >> 0xfffffffe) = 607085216 >> stat("/sys/bus/rbd", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 >> *open("/sys/bus/rbd/add", O_WRONLY) = 3* >> *write(3, "10.198.41.6:6789 >> <http://10.198.41.6:6789>,10.198.41.8:678 >> <http://10.198.41.8:678>"..., 96) = -1 EINVAL (Invalid >>argument)* >> close(3) = 0 >> rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7fbf8a7efa90}, >> {SIG_DFL, [], 0}, 8) = 0 >> rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, >> 0x7fbf8a7efa90}, {SIG_DFL, [], 0}, 8) = 0 >> rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0 >> clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, >> parent_tidptr=0x7fff82c4e040) = 22 >> wait4(22, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22 >> rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fbf8a7efa90}, >> NULL, 8) = 0 >> rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, >> 0x7fbf8a7efa90}, NULL, 8) = 0 >> rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0 >> write(2, "rbd: add failed: ", 17rbd: add failed: ) = 17 >> write(2, "(22) Invalid argument", 21(22) Invalid argument) = >>21 >> write(2, "\n", 1 >> ) = 1 >> exit_group(1) = ? >> +++ exited with 1 +++ >> >> >> The app is run inside the container with setuid = 0 and the >> container is able to mount all required filesystems Š could this >> still be a capability problem ? Also I do not see any call to >> capset() in the strafe log Š >> >> -- >> Om >> >> >> From: Kyle Bader <kyle.bader@xxxxxxxxx> >> Date: Thursday, October 17, 2013 5:08 PM >> To: Kevin Weiler <Kevin.Weiler@xxxxxxxxxxxxxxx> >> Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>, Omar >> Marquez <omar.marquez@xxxxxxxxxxxxxxx>, Khalid Goudeaux >> <Khalid.Goudeaux@xxxxxxxxxxxxxxx> >> Subject: Re: mounting RBD in linux containers >> >> My first guess would be that it's due to LXC dropping capabilities, >> I'd investigate whether CAP_SYS_ADMIN is being dropped. You need >> CAP_SYS_ADMIN for mount and block ioctls, if the container doesn't >> have those privs a map will likely fail. Maybe try tracing the >> command with strace? >> >> On Thu, Oct 17, 2013 at 2:45 PM, Kevin Weiler >> <Kevin.Weiler@xxxxxxxxxxxxxxx> wrote: >> >> Hi all, >> >> We're trying to mount an rbd image inside of a linux container >> that has been created with docker (https://www.docker.io/). We >> seem to have access to the rbd kernel module from inside the >> container: >> >> # lsmod | grep ceph >> libceph 218854 1 rbd >> libcrc32c 12603 3 xfs,libceph,dm_persistent_data >> >> And we can query the pool for available rbds and create rbds >> from inside the container: >> >> # rbd -p dockers --id dockers --keyring >> /etc/ceph/ceph.client.dockers.keyring create lxctest --size >>51200 >> # rbd -p dockers --id dockers --keyring >> /etc/ceph/ceph.client.dockers.keyring ls >> lxctest >> >> But for some reason, we can't seem to map the device to the >> container: >> >> # rbd -p dockers --id dockers --keyring >> /etc/ceph/ceph.client.dockers.keyring map lxctest >> rbd: add failed: (22) Invalid argument >> >> I don't see anything particularly interesting in dmesg or >> messages on either the container or the host box. Any ideas on >> how to troubleshoot this? >> >> Thanks! >> >> >> -- >> >> *Kevin Weiler* >> >> IT >> >> IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | >> Chicago, IL 60606 | http://imc-chicago.com/ >> >> Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail: >> _kevin.weiler@imc-chicago.com_ >> >> >> >>------------------------------------------------------------------------ >> >> The information in this e-mail is intended only for the person >> or entity to which it is addressed. >> >> It may contain confidential and /or privileged material. If >> someone other than the intended recipient should receive this >> e-mail, he / she shall not be entitled to read, disseminate, >> disclose or duplicate it. >> >> If you receive this e-mail unintentionally, please inform us >> immediately by "re >> >> >> >>------------------------------------------------------------------------ >> >> The information in this e-mail is intended only for the person or >> entity to which it is addressed. >> >> It may contain confidential and /or privileged material. If someone >> other than the intended recipient should receive this e-mail, he / >> she shall not be entitled to read, disseminate, disclose or >> duplicate it. >> >> If you receive this e-mail unintentionally, please inform us >> immediately by "reply" and then delete it from your system. Although >> this information has been compiled with great care, neither IMC >> Financial Markets & Asset Management nor any of its related entities >> shall accept any responsibility for any errors, omissions or other >> inaccuracies in this information or for the consequences thereof, >> nor shall it be bound in any way by the contents of this e-mail or >> its attachments. In the event of incomplete or incorrect >> transmission, please return the e-mail to the sender and permanently >> delete this message and any attachments. >> >> Messages and attachments are scanned for all known viruses. Always >> scan attachments before opening them. >> >> >> >> -- >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> ------------------------------------------------------------------------ >> >> The information in this e-mail is intended only for the person or entity >> to which it is addressed. >> >> It may contain confidential and /or privileged material. If someone >> other than the intended recipient should receive this e-mail, he / she >> shall not be entitled to read, disseminate, disclose or duplicate it. >> >> If you receive this e-mail unintentionally, please inform us immediately >> by "reply" and then delete it from your system. Although this >> information has been compiled with great care, neither IMC Financial >> Markets & Asset Management nor any of its related entities shall accept >> any responsibility for any errors, omissions or other inaccuracies in >> this information or for the consequences thereof, nor shall it be bound >> in any way by the contents of this e-mail or its attachments. In the >> event of incomplete or incorrect transmission, please return the e-mail >> to the sender and permanently delete this message and any attachments. >> >> Messages and attachments are scanned for all known viruses. Always scan >> attachments before opening them. >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ________________________________ The information in this e-mail is intended only for the person or entity to which it is addressed. It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it. If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect transmission, please return the e-mail to the sender and permanently delete this message and any attachments. Messages and attachments are scanned for all known viruses. Always scan attachments before opening them. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |