|
O.k. thank you!
I removed the osd just in case after the fact but I will re-add
it back in and update the thread if things still don't look right.
Shain
On 2/1/19 6:35 PM, Vladimir Prokofev
wrote:
Your output looks a bit weird, but still, this is
normal for bluestore. It creates small separate data partition
that is presented as XFS mounted in /var/lib/ceph/osd, while
real data partition is hidden as raw(bluestore) block device.
It's no longer possible to check disk utilisation with df
using bluestore.
To check your osd capacity use 'ceph osd df'
Hi,
I went to replace a disk today (which I had not had to do in a
while)
and after I added it the results looked rather odd compared to
times past:
I was attempting to replace /dev/sdk on one of our osd nodes:
#ceph-deploy disk zap hqosd7 /dev/sdk
#ceph-deploy osd create --data /dev/sdk hqosd7
[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1):
/usr/local/bin/ceph-deploy
osd create --data /dev/sdk hqosd7
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose :
False
[ceph_deploy.cli][INFO ] bluestore :
None
[ceph_deploy.cli][INFO ] cd_conf :
<ceph_deploy.conf.cephdeploy.Conf instance at
0x7fa3b1065a70>
[ceph_deploy.cli][INFO ] cluster :
ceph
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] block_wal :
None
[ceph_deploy.cli][INFO ] default_release :
False
[ceph_deploy.cli][INFO ] username :
None
[ceph_deploy.cli][INFO ] journal :
None
[ceph_deploy.cli][INFO ] subcommand :
create
[ceph_deploy.cli][INFO ] host :
hqosd7
[ceph_deploy.cli][INFO ] filestore :
None
[ceph_deploy.cli][INFO ] func :
<function osd
at 0x7fa3b14b3398>
[ceph_deploy.cli][INFO ] ceph_conf :
None
[ceph_deploy.cli][INFO ] zap_disk :
False
[ceph_deploy.cli][INFO ] data :
/dev/sdk
[ceph_deploy.cli][INFO ] block_db :
None
[ceph_deploy.cli][INFO ] dmcrypt :
False
[ceph_deploy.cli][INFO ] overwrite_conf :
False
[ceph_deploy.cli][INFO ] dmcrypt_key_dir :
/etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet :
False
[ceph_deploy.cli][INFO ] debug :
False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with
data device
/dev/sdk
[hqosd7][DEBUG ] connected to host: hqosd7
[hqosd7][DEBUG ] detect platform information from remote host
[hqosd7][DEBUG ] detect machine type
[hqosd7][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 16.04 xenial
[ceph_deploy.osd][DEBUG ] Deploying osd to hqosd7
[hqosd7][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf
[hqosd7][DEBUG ] find the location of an executable
[hqosd7][INFO ] Running command: /usr/sbin/ceph-volume
--cluster ceph
lvm create --bluestore --data /dev/sdk
[hqosd7][DEBUG ] Running command: /usr/bin/ceph-authtool
--gen-print-key
[hqosd7][DEBUG ] Running command: /usr/bin/ceph --cluster ceph
--name
client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring
-i - osd new c98a11d1-9b7f-487e-8c69-72fc662927d4
[hqosd7][DEBUG ] Running command: vgcreate --force --yes
ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1 /dev/sdk
[hqosd7][DEBUG ] stdout: Physical volume "/dev/sdk"
successfully created
[hqosd7][DEBUG ] stdout: Volume group
"ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1" successfully
created
[hqosd7][DEBUG ] Running command: lvcreate --yes -l 100%FREE
-n
osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1
[hqosd7][DEBUG ] stdout: Logical volume
"osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4" created.
[hqosd7][DEBUG ] Running command: /usr/bin/ceph-authtool
--gen-print-key
[hqosd7][DEBUG ] Running command: mount -t tmpfs tmpfs
/var/lib/ceph/osd/ceph-81
[hqosd7][DEBUG ] Running command: chown -R ceph:ceph /dev/dm-0
[hqosd7][DEBUG ] Running command: ln -s
/dev/ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1/osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
/var/lib/ceph/osd/ceph-81/block
[hqosd7][DEBUG ] Running command: ceph --cluster ceph --name
client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring
mon getmap -o /var/lib/ceph/osd/ceph-81/activate.monmap
[hqosd7][DEBUG ] stderr: got monmap epoch 2
[hqosd7][DEBUG ] Running command: ceph-authtool
/var/lib/ceph/osd/ceph-81/keyring --create-keyring --name
osd.81
--add-key AQCyyFRcSwWqGBAAKZR8rcWIEknj/o3rsehOdA==
[hqosd7][DEBUG ] stdout: creating
/var/lib/ceph/osd/ceph-81/keyring
[hqosd7][DEBUG ] stdout: added entity osd.81 auth auth(auid =
18446744073709551615
key=AQCyyFRcSwWqGBAAKZR8rcWIEknj/o3rsehOdA== with 0
caps)
[hqosd7][DEBUG ] Running command: chown -R ceph:ceph
/var/lib/ceph/osd/ceph-81/keyring
[hqosd7][DEBUG ] Running command: chown -R ceph:ceph
/var/lib/ceph/osd/ceph-81/
[hqosd7][DEBUG ] Running command: /usr/bin/ceph-osd --cluster
ceph
--osd-objectstore bluestore --mkfs -i 81 --monmap
/var/lib/ceph/osd/ceph-81/activate.monmap --keyfile -
--osd-data
/var/lib/ceph/osd/ceph-81/ --osd-uuid
c98a11d1-9b7f-487e-8c69-72fc662927d4 --setuser ceph --setgroup
ceph
[hqosd7][DEBUG ] --> ceph-volume lvm prepare successful
for: /dev/sdk
[hqosd7][DEBUG ] Running command: ceph-bluestore-tool
--cluster=ceph
prime-osd-dir --dev
/dev/ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1/osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
--path /var/lib/ceph/osd/ceph-81
[hqosd7][DEBUG ] Running command: ln -snf
/dev/ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1/osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
/var/lib/ceph/osd/ceph-81/block
[hqosd7][DEBUG ] Running command: chown -R ceph:ceph /dev/dm-0
[hqosd7][DEBUG ] Running command: chown -R ceph:ceph
/var/lib/ceph/osd/ceph-81
[hqosd7][DEBUG ] Running command: systemctl enable
ceph-volume@lvm-81-c98a11d1-9b7f-487e-8c69-72fc662927d4
[hqosd7][DEBUG ] stderr: Created symlink from
/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-81-c98a11d1-9b7f-487e-8c69-72fc662927d4.service
to /lib/systemd/system/ceph-volume@.service.
[hqosd7][DEBUG ] Running command: systemctl start ceph-osd@81
[hqosd7][DEBUG ] --> ceph-volume lvm activate successful
for osd ID: 81
[hqosd7][DEBUG ] --> ceph-volume lvm create successful for:
/dev/sdk
[hqosd7][INFO ] checking OSD status...
[hqosd7][DEBUG ] find the location of an executable
[hqosd7][INFO ] Running command: /usr/bin/ceph --cluster=ceph
osd stat
--format=json
[hqosd7][WARNIN] there are 2 OSDs down
[hqosd7][WARNIN] there are 2 OSDs out
[ceph_deploy.osd][DEBUG ] Host hqosd7 is now ready for osd
use.
_________________________________________________________
However when I listed out the partitions on the server...this
is what I
found (osd 81 was showing up as 32G as opposed to the 3.7T
that the
drive is):
/dev/sdm1 3.7T 2.9T 756G 80%
/var/lib/ceph/osd/ceph-77
tmpfs 32G 48K 32G 1%
/var/lib/ceph/osd/ceph-81
__________________________________________________________
Here is some output from fdisk as well:
Disk /dev/sdm: 3.7 TiB, 4000225165312 bytes, 7812939776
sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: CD3A35E7-CF85-4E79-9911-B80099349C85
Device Start End Sectors Size Type
/dev/sdm1 20973568 7812939742 7791966175 3.6T Ceph OSD
/dev/sdm2 2048 20971520 20969473 10G Ceph Journal
Partition table entries are not in disk order.
Disk /dev/sdk: 3.7 TiB, 4000225165312 bytes, 7812939776
sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk
/dev/mapper/ceph--bbe0e44e--afc9--4cf1--9f1a--ed7d20f796c1-osd--block--c98a11d1--9b7f--487e--8c69--72fc662927d4:
3.7 TiB, 4000220971008 bytes, 7812931584 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
__________________________________________________________
I would normally spend more time looking around for an answer
however
our cluster is a little tight on space and I really need to
replace 2 or
3 drives ASAP in order to resolve some of this 'backfillfull'
errors I
am seeing.
I am assuming this isn't normal...however this would be the
first
bluestore osd added to this cluster...so I am not really sure.
Thanks in advance,
Shain
--
NPR | Shain Miley | Manager of Infrastructure, Digital Media |
smiley@xxxxxxx | 202.513.3649
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com&d=DwICAg&c=E2nBno7hEddFhl23N5nD1Q&r=cqFccwnwHGRorPuRWs36Dw&m=pfB6jmrh1mNDkHEJQD2WAo1zfgb_kOWj-CVReKz9SJU&s=Dmc_tJZr_Rtff9x_Xd3avq6jOXZFRqHAQRxavxG-cOw&e=
--
NPR | Shain Miley | Manager of Infrastructure, Digital Media | smiley@xxxxxxx | 202.513.3649
|
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
