Hi , I think I figuered it out .. I had to create folder /var/lib/ceph/osd/ceph-$id and there might be a bug in ceph-disk-prepare I had to run --snip-- [root at cc01 ceph]# ceph-deploy osd --zap-disk create cc02:sdb [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.2): /usr/bin/ceph-deploy osd --zap-disk create cc02:sdb [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc02:/dev/sdb: [cc02][DEBUG ] connected to host: cc02 [cc02][DEBUG ] detect platform information from remote host [cc02][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Deploying osd to cc02 [cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [cc02][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add [ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None activate True [cc02][INFO ] Running command: ceph-disk-prepare --zap-disk --fs-type xfs --cluster ceph -- /dev/sdb [cc02][WARNIN] Caution: invalid backup GPT header, but valid main header; regenerating [cc02][WARNIN] backup header from main header. [cc02][WARNIN] [cc02][WARNIN] Warning! Main and backup partition tables differ! Use the 'c' and 'e' options [cc02][WARNIN] on the recovery & transformation menu to examine the two tables. [cc02][WARNIN] [cc02][WARNIN] Warning! One or more CRCs don't match. You should repair the disk! [cc02][WARNIN] [cc02][WARNIN] Invalid partition data! [cc02][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk', '--zap-all', '--clear', '--mbrtogpt', '--', '/dev/sdb']' returned non-zero exit status 3 [cc02][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or [cc02][DEBUG ] other utilities. [cc02][DEBUG ] Information: Creating fresh partition table; will override earlier problems! [cc02][DEBUG ] Non-GPT disk; not saving changes. Use -g to override. [cc02][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --zap-disk --fs-type xfs --cluster ceph -- /dev/sdb [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs [root at cc01 ceph]# ceph-deploy osd --zap-disk create cc02:sdb [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.2): /usr/bin/ceph-deploy osd --zap-disk create cc02:sdb [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc02:/dev/sdb: [cc02][DEBUG ] connected to host: cc02 [cc02][DEBUG ] detect platform information from remote host [cc02][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Deploying osd to cc02 [cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [cc02][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add [ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None activate True [cc02][INFO ] Running command: ceph-disk-prepare --zap-disk --fs-type xfs --cluster ceph -- /dev/sdb [cc02][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb [cc02][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb [cc02][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors [cc02][WARNIN] BLKPG: Device or resource busy [cc02][WARNIN] error adding partition 2 [cc02][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb [cc02][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors [cc02][WARNIN] BLKPG: Device or resource busy [cc02][WARNIN] error adding partition 1 [cc02][WARNIN] BLKPG: Device or resource busy [cc02][WARNIN] error adding partition 2 [cc02][DEBUG ] Creating new GPT entries. [cc02][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or [cc02][DEBUG ] other utilities. [cc02][DEBUG ] The operation has completed successfully. [cc02][DEBUG ] Setting name! [cc02][DEBUG ] partNum is 1 [cc02][DEBUG ] REALLY setting name! [cc02][DEBUG ] The operation has completed successfully. [cc02][DEBUG ] Setting name! [cc02][DEBUG ] partNum is 0 [cc02][DEBUG ] REALLY setting name! [cc02][DEBUG ] The operation has completed successfully. [cc02][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=30458561 blks [cc02][DEBUG ] = sectsz=512 attr=2, projid32bit=0 [cc02][DEBUG ] data = bsize=4096 blocks=121834241, imaxpct=25 [cc02][DEBUG ] = sunit=0 swidth=0 blks [cc02][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 [cc02][DEBUG ] log =internal log bsize=4096 blocks=59489, version=2 [cc02][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1 [cc02][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0 [cc02][DEBUG ] The operation has completed successfully. [cc02][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add [cc02][INFO ] checking OSD status... [cc02][INFO ] Running command: ceph --cluster=ceph osd stat --format=json [ceph_deploy.osd][DEBUG ] Host cc02 is now ready for osd use. --snip-- To get the ceph health status Ok . I had to readjust pool size for i in $(rados lspools);do ceph osd pool set $i size 1;done set pool 0 size to 1 set pool 1 size to 1 set pool 2 size to 1 set pool 3 size to 1 set pool 4 size to 1 set pool 5 size to 1 [root at cc01 ceph]# for i in $(rados lspools);do ceph osd pool set $i size 2;done set pool 0 size to 2 set pool 1 size to 2 set pool 2 size to 2 set pool 3 size to 2 set pool 4 size to 2 set pool 5 size to 2 [root at cc01 ceph]# ceph -s cluster 9f951603-0c31-4942-aefd-96f85b5ea908 health HEALTH_OK monmap e1: 3 mons at {cc01= 172.18.1.31:6789/0,cc02=172.18.1.32:6789/0,cc03=172.18.1.33:6789/0}, election epoch 26, quorum 0,1,2 cc01,cc02,cc03 osdmap e64: 2 osds: 2 up, 2 in pgmap v121: 492 pgs, 6 pools, 0 bytes data, 0 objects 80636 kB used, 928 GB / 928 GB avail 492 active+clean --snip-- Can I pass these values via ceph.conf ? On Wed, May 21, 2014 at 4:05 PM, 10 minus <t10tennn at gmail.com> wrote: > Hi, > > I have just started to dabble with ceph - went thru the docs > http://ceph.com/howto/deploying-ceph-with-ceph-deploy/ > > > I have a 3 node setup with 2 nodes for OSD > > I use ceph-deploy mechanism. > > The ceph init scripts expects that cluster.conf to be ceph.conf . If I > give any other name the init scripts dont work. So for test purpose Im > using ceph.conf > > > --ceph.conf-- > [global] > auth_service_required = cephx > filestore_xattr_use_omap = true > auth_client_required = cephx > auth_cluster_required = cephx > mon_host = 172.18.1.31,172.18.1.32,172.18.1.33 > mon_initial_members = cc01, cc02, cc03 > fsid = b58e50f1-13a3-4b14-9cff-32b6edd851c9 > --snip-- > > I managed to get mon deployed but ceph -s returns health error > > --snip-- > ceph -s > cluster b58e50f1-13a3-4b14-9cff-32b6edd851c9 > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no > osds > monmap e1: 3 mons at {cc01= > 172.18.1.31:6789/0,cc02=172.18.1.32:6789/0,cc03=172.18.1.33:6789/0}, > election epoch 4, quorum 0,1,2 cc01,cc02,cc03 > osdmap e1: 0 osds: 0 up, 0 in > pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects > 0 kB used, 0 kB / 0 kB avail > 192 creating > --snip-- > > I tried creating two osds. Well they fail too probably has to do with > health error message. > > --snip-- > ceph-deploy osd create cc01:/dev/sdb cc02:/dev/sdb > [ceph_deploy.conf][DEBUG ] found configuration file at: > /root/.cephdeploy.conf > [ceph_deploy.cli][INFO ] Invoked (1.5.2): /usr/bin/ceph-deploy osd create > cc01:/dev/sdb cc02:/dev/sdb > [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc01:/dev/sdb: > cc02:/dev/sdb: > [cc01][DEBUG ] connected to host: cc01 > [cc01][DEBUG ] detect platform information from remote host > [cc01][DEBUG ] detect machine type > [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final > [ceph_deploy.osd][DEBUG ] Deploying osd to cc01 > [cc01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf > [cc01][INFO ] Running command: udevadm trigger --subsystem-match=block > --action=add > [ceph_deploy.osd][DEBUG ] Preparing host cc01 disk /dev/sdb journal None > activate True > [cc01][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster > ceph -- /dev/sdb > [cc01][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb > [cc01][WARNIN] Could not create partition 2 from 10485761 to 10485760 > [cc01][WARNIN] Error encountered; not saving changes. > [cc01][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk', > '--new=2:0:5120M', '--change-name=2:ceph journal', > '--partition-guid=2:d882631c-0069-4238-86df-9762ad478daa', > '--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', > '/dev/sdb']' returned non-zero exit status 4 > [cc01][DEBUG ] Setting name! > [cc01][DEBUG ] partNum is 1 > [cc01][DEBUG ] REALLY setting name! > [cc01][ERROR ] RuntimeError: command returned non-zero exit status: 1 > [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare > --fs-type xfs --cluster ceph -- /dev/sdb > [cc02][DEBUG ] connected to host: cc02 > [cc02][DEBUG ] detect platform information from remote host > [cc02][DEBUG ] detect machine type > [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final > [ceph_deploy.osd][DEBUG ] Deploying osd to cc02 > [cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf > [cc02][INFO ] Running command: udevadm trigger --subsystem-match=block > --action=add > [ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None > activate True > [cc02][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster > ceph -- /dev/sdb > [cc02][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb > [cc02][WARNIN] Could not create partition 2 from 10485761 to 10485760 > [cc02][WARNIN] Error encountered; not saving changes. > [cc02][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk', > '--new=2:0:5120M', '--change-name=2:ceph journal', > '--partition-guid=2:486c9081-a73c-4906-b97a-c03458feba26', > '--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', > '/dev/sdb']' returned non-zero exit status 4 > [cc02][DEBUG ] Found valid GPT with corrupt MBR; using GPT and will write > new > [cc02][DEBUG ] protective MBR on save. > [cc02][DEBUG ] Setting name! > [cc02][DEBUG ] partNum is 1 > [cc02][DEBUG ] REALLY setting name! > [cc02][ERROR ] RuntimeError: command returned non-zero exit status: 1 > [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare > --fs-type xfs --cluster ceph -- /dev/sdb > [ceph_deploy][ERROR ] GenericError: Failed to create 2 OSDs > --snip-- > > Any pointers to fix the issue. > > Cheers > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140524/70e1f710/attachment.htm>