Hi Tijn, I had also created my first CEPH storage cluster almost as you have created. I had 3 VMs for OSD nodes and 1 VM for Monitor node. All 3 OSD VMs were having one 10 GB virtual disks. so I faced almost same problem as you are facing right now. Then changing disk space from 10 GB to 20 GB solved my problem. I don't know if dynamic disks will create any problem. But I think instead of having 6 OSDs you can have 3 OSDs, one OSD per VM and can increase disk size from 10 GB to 20 GB for 3 OSDs. I don't know this will solve your problem or not but worthy to try. I mean 3 OSDs are enough for testing purpose initially. Regards, Pratik On 8/5/2014 12:37 PM, Tijn Buijs wrote: > Hello Pratik, > > I'm using virtual disks as OSDs. I prefer virtual disks over > directories because this resembles the production environment a bit > better. > I'm using VirtualBox for virtualisation. The OSDs are dynamic disks, > not pre-allocated, but this shouldn't be a problem, right? I don't > have the diskspace on my iMac to have all 6 OSDs pre-allocated :). > I've made the virtual OSD disks 10 GB each, by the way, so that should > be enough for a first test, imho. > > Met vriendelijke groet/With kind regards, > > Tijn Buijs > > Cloud.nl logo > > tijn at cloud.nl <mailto:tijn at cloud.nl> | T. 0800-CLOUDNL / +31 (0)162 > 820 000 | F. +31 (0)162 820 001 > Cloud.nl B.V. | Minervum 7092D | 4817 ZK Breda | www.cloud.nl > <http://www.cloud.nl> > On 04/08/14 14:51, Pratik Rupala wrote: >> Hi, >> >> You mentioned that you have 3 hosts which are VMs. Are you using >> simple directories as OSDs or virtual disks as OSDs? >> >> I had same problem few days back where enough space was not available >> from OSD for the cluster. >> >> Try to increase the size of disks if you are using virtual disks and >> if you are using directories as OSDs then check whether you have >> enough space on root device using df -h command on OSD node. >> >> Regards, >> Pratik >> >> On 8/4/2014 4:11 PM, Tijn Buijs wrote: >>> Hi Everybody, >>> >>> My idea was that maybe I was inpatient or something, so I let my >>> Ceph cluster running over the weekend. So from friday 15:00 until >>> now (it is monday morning 11:30 here now) it kept on running. And it >>> didn't help :). It still needs to create 192 PGs. >>> I've reinstalled my entier cluster a few times now. I switched over >>> from CentOS 6.5 to Ubuntu 14.04.1 LTS and back to CentOS again, and >>> every time I get exactly the same results. The PGs are getting in >>> the incomplete, stuck inactive, stuk unclean state. What am I doing >>> wrong? :). >>> >>> For the moment I'm running with 6 OSDs evenly divided over 3 hosts >>> (so each host has 2 OSDs). I've only got 1 monitor configured in my >>> current cluster. I hit some other problem when trying to add monitor >>> 2 and 3 again. And to not complicate things with multiple problems >>> at the same time I've switched back to only 1 monitor. The cluster >>> should work that way, right? >>> >>> To make things clear for everybody, here is the output of ceph >>> health and ceph -s: >>> $ ceph health >>> HEALTH_WARN 192 pgs incomplete; 192 pgs stuck inactive; 192 pgs >>> stuck unclean >>> $ ceph -s >>> cluster 43d5f48b-d034-4f50-bec8-5c4f3ad8276f >>> health HEALTH_WARN 192 pgs incomplete; 192 pgs stuck inactive; >>> 192 pgs stuck unclean >>> monmap e1: 1 mons at {ceph-mon1=10.28.28.71:6789/0}, election >>> epoch 1, quorum 0 ceph-mon1 >>> osdmap e20: 6 osds: 6 up, 6 in >>> pgmap v40: 192 pgs, 3 pools, 0 bytes data, 0 objects >>> 197 MB used, 30456 MB / 30653 MB avail >>> 192 incomplete >>> >>> I hope somebody has an idea for me to try :). >>> >>> Met vriendelijke groet/With kind regards, >>> >>> Tijn Buijs >>> >>> Cloud.nl logo >>> >>> tijn at cloud.nl <mailto:tijn at cloud.nl> | T. 0800-CLOUDNL / +31 (0)162 >>> 820 000 | F. +31 (0)162 820 001 >>> Cloud.nl B.V. | Minervum 7092D | 4817 ZK Breda | www.cloud.nl >>> <http://www.cloud.nl> >>> On 31/07/14 17:19, Alfredo Deza wrote: >>>> >>>> >>>> >>>> On Thu, Jul 31, 2014 at 10:36 AM, Tijn Buijs <tijn at cloud.nl >>>> <mailto:tijn at cloud.nl>> wrote: >>>> >>>> Hello everybody, >>>> >>>> At cloud.nl <http://cloud.nl> we are going to use Ceph. So I >>>> find it a good idea to get some handson experience with it, so >>>> I can work with it :). So I'm installing a testcluster in a few >>>> VirtualBox machines on my iMac, which runs OS X 10.9.4 >>>> offcourse. I know I will get a lousy performance, but that's >>>> not the objective here. The objective is to get some experience >>>> with Ceph, to see how it works. >>>> >>>> But I hit an issue during the initial setup of the cluster. >>>> When I'm done installing everything and following the howto's >>>> on ceph.com <http://ceph.com> (the preflight >>>> <http://ceph.com/docs/master/start/quick-start-preflight/> and >>>> the Storage Cluster quick start >>>> <http://ceph.com/docs/master/start/quick-ceph-deploy/>) I need >>>> to run ceph health to see that everything is running perfectly. >>>> But it doesn't run perfectly, I get the following output: >>>> ceph at ceph-admin:~$ ceph health >>>> HEALTH_WARN 192 pgs incomplete; 192 pgs stuck inactive; 192 pgs >>>> stuck unclean >>>> >>>> And it stays at this information, it never ever changes. So >>>> everything is really stuck. But I don't know what is stuck >>>> exactly and how I can fix it. Some more info about my cluster: >>>> ceph at ceph-admin:~$ ceph -s >>>> cluster d31586a5-6dd6-454e-8835-0d6d9e204612 >>>> health HEALTH_WARN 192 pgs incomplete; 192 pgs stuck >>>> inactive; 192 pgs stuck unclean >>>> monmap e3: 3 mons at >>>> {ceph-mon1=10.28.28.18:6789/0,ceph-mon2=10.28.28.31:6789/0,ceph-mon3=10.28.28.50:6789/0 >>>> <http://10.28.28.18:6789/0,ceph-mon2=10.28.28.31:6789/0,ceph-mon3=10.28.28.50:6789/0>}, >>>> election epoch 4, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3 >>>> osdmap e25: 6 osds: 6 up, 6 in >>>> pgmap v56: 192 pgs, 3 pools, 0 bytes data, 0 objects >>>> 197 MB used, 30455 MB / 30653 MB avail >>>> 192 creating+incomplete >>>> >>>> I'm running on Ubuntu 14.04.1 LTS Server. I did try to get it >>>> running on CentOS 6.5 too (CentOS 6.5 is my actual distro of >>>> choice, but Ceph has more affinity with Ubuntu, so I tried that >>>> too), but I got exactly the same results. >>>> >>>> But because this is my first install of Ceph I don't know the >>>> exact debug commands and stuff. I'm willing to get this >>>> working, but I just don't know how :). Any help is appreciated :). >>>> >>>> >>>> Did you use ceph-deploy? (the link to the quick start guide makes >>>> me think you did) >>>> >>>> If that was the case, did you get any warnings/errors at all? >>>> >>>> ceph-deploy is very verbose because some of these things are hard >>>> to debug. Mind sharing that output? >>>> >>>> >>>> Met vriendelijke groet/With kind regards, >>>> >>>> Tijn Buijs >>>> >>>> Cloud.nl logo >>>> >>>> tijn at cloud.nl <mailto:tijn at cloud.nl> | T. 0800-CLOUDNL / +31 >>>> (0)162 820 000 <tel:%2B31%20%280%29162%20820%20000> | F. +31 >>>> (0)162 820 001 <tel:%2B31%20%280%29162%20820%20001> >>>> Cloud.nl B.V. | Minervum 7092D | 4817 ZK Breda | www.cloud.nl >>>> <http://www.cloud.nl> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>> >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users at lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140805/49308262/attachment.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 14056 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140805/49308262/attachment.jpeg> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 14056 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140805/49308262/attachment-0001.jpeg> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 14056 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140805/49308262/attachment-0002.jpeg>
 |