Will do. We haven't really had any stability issues once the containers are running - they are the same daemons that would run on your host OS, they just happen to be in a container. We have seen a few cases during startup where there are placement groups stuck inactive/peering, which prevents the metadata servers from coming up. Restarting the daemon containers resolves this, although I haven't figured out why it's occurring.
Chris Armstrong
Head of Services & Community
OpDemand / Deis.io
GitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/
On Fri, Nov 14, 2014 at 3:31 PM, Robert LeBlanc <robert@xxxxxxxxxxxxx> wrote:
Ceph in Docker is very intriguing to me, but I understood that there were still a number of stability and implementation issues. What is your experience? Please post a link to your blog when you are done, I'd be interested in reading it.On Fri, Nov 14, 2014 at 4:15 PM, Christopher Armstrong <chris@xxxxxxxxxxxx> wrote:Hi Loic,Thanks for the reply! I started implementing these first using ceph-deploy, but ran into lots of issues because there is no upstart daemon running within the containers. So while ceph-deploy would successfully log into the containers remotely (I had to run sshd in the containers), it would fail when it tried to start the daemons. I spent a few days hitting a wall with this before I found Seán McCord's repository at https://github.com/ulexus/docker-ceph which configures/starts everything without ceph-deploy, and went that route. So our containers are based on his work, but we added a lot of service coordination/discovery within CoreOS.I'm still planning on writing a blog post about our use of Ceph in containers, but haven't found the time. I'm hoping to get to it next week. In the meantime, let me know if you have any questions about the implementation!ChrisChris Armstrong
Head of Services & Community
OpDemand / Deis.ioGitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/
On Thu, Oct 30, 2014 at 5:36 AM, Hunter Nield <hunter@xxxxxxxx> wrote:http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.comGreat idea Loic!I'd forgotten about the storage-driver side but is a great fit with CephOn Thu, Oct 30, 2014 at 6:50 PM, Loic Dachary <loic@xxxxxxxxxxx> wrote:Hi,
It would also be great to have a Ceph docker storage driver.
https://github.com/docker/docker/issues/8854
Cheers
On 30/10/2014 11:06, Hunter Nield wrote:
> Great to see this discussion starting. There is work being done in this repo for Ceph in Docker - https://github.com/Ulexus/docker-ceph
>
> Independently of this, we're using RBD as backing for the Docker containers but still installing Ceph as part of the system and mounting outside of the container before launch. Ceph in Docker is in the backlog for us but becoming important.
>
> There is a proposal for volume drivers in Docker which would also be a great fit for RBD (via fuse?). Hopefully it gets moving soon - https://github.com/docker/docker/issues/7249
>
>
>
> On Thu, Oct 30, 2014 at 5:18 PM, Loic Dachary <loic@xxxxxxxxxxx <mailto:loic@xxxxxxxxxxx>> wrote:
>
> Hi Christopher,
>
> Very interesting setup :-) Last week-end I discussed this in theory with Johan Euphrosine and did not know you had something already. Deploying a mon in a container is fairly straightforward and I wonder if the boot script
>
> https://github.com/deis/deis/blob/master/store/monitor/bin/boot
>
> could somehow be coupled to ceph-deploy to benefit from its flexibility. Deploying an OSD with
>
> https://github.com/deis/deis/blob/master/store/daemon/bin/boot
>
> is more complicated and I don't quite understand how it uses the udev rules and their relationship with the upstart / systemd scripts that play an important role in the OSD lifecycle
>
> https://github.com/ceph/ceph/tree/giant/udev
> https://github.com/ceph/ceph/tree/giant/src/upstart
> https://github.com/ceph/ceph/blob/giant/src/upstart/ceph-osd.conf
>
> Cheers
>
> On 30/10/2014 06:16, Christopher Armstrong wrote:
> > Hey Patrick,
> >
> > We recently added a new component to Deis which is based entirely on running Ceph in containers. We're running mons, OSDs, and MDSes in containers, and consuming from containers with radosgw as well as CephFS. See the source here: https://github.com/deis/deis/tree/master/store
> >
> > I'm pretty proud of the work, and would be more than happy to write a blog post about it if you'd like.
> >
> > Chris
> >
> > *Chris Armstrong
> > *Head of Services
> > OpDemand / Deis.io
> >
> > GitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/
> >
> >
> > On Wed, Oct 29, 2014 at 8:26 PM, Patrick McGarry <patrick@xxxxxxxxxxx <mailto:patrick@xxxxxxxxxxx> <mailto:patrick@xxxxxxxxxxx <mailto:patrick@xxxxxxxxxxx>>> wrote:
> >
> > Hey cephers,
> >
> > Given some of the recent interest in utilizing Docker with Ceph I'm
> > taking another survey of the landscape. I know that Loic recently got
> > Teuthology running with Docker (http://dachary.org/?p=3330) but I'd
> > like to look at running a containerized Ceph setup as well.
> >
> > So far I see that Sebastien did an experiment back in 2013:
> >
> > http://www.sebastien-han.fr/blog/2013/09/19/how-I-barely-got-my-first-ceph-mon-running-in-docker/
> >
> > and Lorieri had a CoreOS experiment:
> >
> > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-August/042063.html
> >
> > but there have been a few people that mentioned their experiments to
> > me in passing at cons and other places. I'd love to gather any
> > experience that people have gleaned in this area by aggregating blog
> > entries and other notes. So, if you have a successful Ceph+Docker
> > setup and would be willing to write a short doc via email/blog/wiki I
> > would greatly appreciate it. Thanks!
> >
> >
> > Best Regards,
> >
> > Patrick McGarry
> > Director Ceph Community || Red Hat
> > http://ceph.com || http://community.redhat.com
> > @scuttlemonkey || @ceph
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
--
Loïc Dachary, Artisan Logiciel Libre
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
