Also one more thing, If I want to use BlueStore how do I let it to know that I have more space? Do i need to specify a size at any point? -Gencer. -----Original Message----- From: gencer@xxxxxxxxxxxxx [mailto:gencer@xxxxxxxxxxxxx] Sent: Monday, July 17, 2017 6:04 PM To: 'Wido den Hollander' <wido@xxxxxxxx>; 'ceph-users@xxxxxxxxxxxxxx' <ceph-users@xxxxxxxxxxxxxx> Subject: RE: Ceph (Luminous) shows total_space wrong I used this methods: $ ceph-deploy osd prepare sr-09-01-18:/dev/sdb1 sr-10-01-18:/dev/sdb1 .... (one from 09th server one from 10th server..) and then; $ ceph-deploy osd activate sr-09-01-18:/dev/sdb1 sr-10-01-18:/dev/sdb1 ... This is my second creation for ceph cluster. At first I used bluestore. This time i did not use bluestore (also removed from conf file). Still seen as 200GB. How can I make sure BlueStore is disabled (even if i not put any command). -Gencer. -----Original Message----- From: Wido den Hollander [mailto:wido@xxxxxxxx] Sent: Monday, July 17, 2017 5:57 PM To: ceph-users@xxxxxxxxxxxxxx; gencer@xxxxxxxxxxxxx Subject: RE: Ceph (Luminous) shows total_space wrong > Op 17 juli 2017 om 16:41 schreef gencer@xxxxxxxxxxxxx: > > > Hi Wido, > > Each disk is 3TB SATA (2.8TB seen) but what I got is this: > > First let me gave you df -h: > > /dev/sdb1 2.8T 754M 2.8T 1% /var/lib/ceph/osd/ceph-0 > /dev/sdc1 2.8T 753M 2.8T 1% /var/lib/ceph/osd/ceph-2 > /dev/sdd1 2.8T 752M 2.8T 1% /var/lib/ceph/osd/ceph-4 > /dev/sde1 2.8T 752M 2.8T 1% /var/lib/ceph/osd/ceph-6 > /dev/sdf1 2.8T 753M 2.8T 1% /var/lib/ceph/osd/ceph-8 > /dev/sdg1 2.8T 752M 2.8T 1% /var/lib/ceph/osd/ceph-10 > /dev/sdh1 2.8T 751M 2.8T 1% /var/lib/ceph/osd/ceph-12 > /dev/sdi1 2.8T 751M 2.8T 1% /var/lib/ceph/osd/ceph-14 > /dev/sdj1 2.8T 751M 2.8T 1% /var/lib/ceph/osd/ceph-16 > /dev/sdk1 2.8T 751M 2.8T 1% /var/lib/ceph/osd/ceph-18 > > > Then here is my results from ceph df commands: > > ceph df > > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 200G 179G 21381M 10.44 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > rbd 0 0 0 86579M 0 > cephfs_data 1 0 0 86579M 0 > cephfs_metadata 2 2488 0 86579M 21 > Ok, that's odd. But I think these disks are using BlueStore since that's what Luminous defaults to. The partitions seem to be mixed up, so can you check on how you created the OSDs? Was that with ceph-disk? If so, what additional arguments did you use? Wido > ceph osd df > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 0 0.00980 1.00000 10240M 1070M 9170M 10.45 1.00 173 > 2 0.00980 1.00000 10240M 1069M 9170M 10.45 1.00 150 > 4 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 148 > 6 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 167 > 8 0.00980 1.00000 10240M 1069M 9171M 10.44 1.00 166 > 10 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 171 > 12 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 160 > 14 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 179 > 16 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 182 > 18 0.00980 1.00000 10240M 1069M 9170M 10.44 1.00 168 > 1 0.00980 1.00000 10240M 1069M 9170M 10.45 1.00 167 > 3 0.00980 1.00000 10240M 1069M 9170M 10.45 1.00 156 > 5 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 152 > 7 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 158 > 9 0.00980 1.00000 10240M 1069M 9170M 10.44 1.00 174 > 11 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 153 > 13 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 179 > 15 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 186 > 17 0.00980 1.00000 10240M 1068M 9171M 10.44 1.00 185 > 19 0.00980 1.00000 10240M 1067M 9172M 10.43 1.00 154 > TOTAL 200G 21381M 179G 10.44 > MIN/MAX VAR: 1.00/1.00 STDDEV: 0.00 > > > -Gencer. > > -----Original Message----- > From: Wido den Hollander [mailto:wido@xxxxxxxx] > Sent: Monday, July 17, 2017 4:57 PM > To: ceph-users@xxxxxxxxxxxxxx; gencer@xxxxxxxxxxxxx > Subject: Re: Ceph (Luminous) shows total_space wrong > > > > Op 17 juli 2017 om 15:49 schreef gencer@xxxxxxxxxxxxx: > > > > > > Hi, > > > > > > > > I successfully managed to work with ceph jewel. Want to try luminous. > > > > > > > > I also set experimental bluestore while creating osds. Problem is, I > > have 20x3TB hdd in two nodes and i would expect 55TB usable (as on > > jewel) on luminous but i see 200GB. Ceph thinks I have only 200GB > > space available in total. I see all osds are up and in. > > > > > > > > 20 osd up; 20 osd in. 0 down. > > > > > > > > Ceph -s shows HEALTH_OK. I have only one monitor and one mds. > > (1/1/1) and it is up:active. > > > > > > > > ceph osd tree gave me all OSDs in nodes are up and results are > > 1.0000... I checked via df -h but all disks ahows 2.7TB. Basically something is wrong. > > Same settings and followed schema on jewel is successful except luminous. > > > > What do these commands show: > > - ceph df > - ceph osd df > > Might be that you are looking at the wrong numbers. > > Wido > > > > > > > What might it be? > > > > > > > > What do you need to know to solve this problem? Why ceph thinks I > > have 200GB space only? > > > > > > > > Thanks, > > > > Gencer. > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |