On Fri, Feb 1, 2019 at 6:07 PM Shain Miley <smiley@xxxxxxx> wrote:
>
> Hi,
>
> I went to replace a disk today (which I had not had to do in a while)
> and after I added it the results looked rather odd compared to times past:
>
> I was attempting to replace /dev/sdk on one of our osd nodes:
>
> #ceph-deploy disk zap hqosd7 /dev/sdk
> #ceph-deploy osd create --data /dev/sdk hqosd7
>
> [ceph_deploy.conf][DEBUG ] found configuration file at:
> /root/.cephdeploy.conf
> [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/local/bin/ceph-deploy
> osd create --data /dev/sdk hqosd7
> [ceph_deploy.cli][INFO ] ceph-deploy options:
> [ceph_deploy.cli][INFO ] verbose : False
> [ceph_deploy.cli][INFO ] bluestore : None
> [ceph_deploy.cli][INFO ] cd_conf :
> <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fa3b1065a70>
> [ceph_deploy.cli][INFO ] cluster : ceph
> [ceph_deploy.cli][INFO ] fs_type : xfs
> [ceph_deploy.cli][INFO ] block_wal : None
> [ceph_deploy.cli][INFO ] default_release : False
> [ceph_deploy.cli][INFO ] username : None
> [ceph_deploy.cli][INFO ] journal : None
> [ceph_deploy.cli][INFO ] subcommand : create
> [ceph_deploy.cli][INFO ] host : hqosd7
> [ceph_deploy.cli][INFO ] filestore : None
> [ceph_deploy.cli][INFO ] func : <function osd
> at 0x7fa3b14b3398>
> [ceph_deploy.cli][INFO ] ceph_conf : None
> [ceph_deploy.cli][INFO ] zap_disk : False
> [ceph_deploy.cli][INFO ] data : /dev/sdk
> [ceph_deploy.cli][INFO ] block_db : None
> [ceph_deploy.cli][INFO ] dmcrypt : False
> [ceph_deploy.cli][INFO ] overwrite_conf : False
> [ceph_deploy.cli][INFO ] dmcrypt_key_dir :
> /etc/ceph/dmcrypt-keys
> [ceph_deploy.cli][INFO ] quiet : False
> [ceph_deploy.cli][INFO ] debug : False
> [ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device
> /dev/sdk
> [hqosd7][DEBUG ] connected to host: hqosd7
> [hqosd7][DEBUG ] detect platform information from remote host
> [hqosd7][DEBUG ] detect machine type
> [hqosd7][DEBUG ] find the location of an executable
> [ceph_deploy.osd][INFO ] Distro info: Ubuntu 16.04 xenial
> [ceph_deploy.osd][DEBUG ] Deploying osd to hqosd7
> [hqosd7][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [hqosd7][DEBUG ] find the location of an executable
> [hqosd7][INFO ] Running command: /usr/sbin/ceph-volume --cluster ceph
> lvm create --bluestore --data /dev/sdk
> [hqosd7][DEBUG ] Running command: /usr/bin/ceph-authtool --gen-print-key
> [hqosd7][DEBUG ] Running command: /usr/bin/ceph --cluster ceph --name
> client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
> -i - osd new c98a11d1-9b7f-487e-8c69-72fc662927d4
> [hqosd7][DEBUG ] Running command: vgcreate --force --yes
> ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1 /dev/sdk
> [hqosd7][DEBUG ] stdout: Physical volume "/dev/sdk" successfully created
> [hqosd7][DEBUG ] stdout: Volume group
> "ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1" successfully created
> [hqosd7][DEBUG ] Running command: lvcreate --yes -l 100%FREE -n
> osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
> ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1
> [hqosd7][DEBUG ] stdout: Logical volume
> "osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4" created.
> [hqosd7][DEBUG ] Running command: /usr/bin/ceph-authtool --gen-print-key
> [hqosd7][DEBUG ] Running command: mount -t tmpfs tmpfs
> /var/lib/ceph/osd/ceph-81
> [hqosd7][DEBUG ] Running command: chown -R ceph:ceph /dev/dm-0
> [hqosd7][DEBUG ] Running command: ln -s
> /dev/ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1/osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
> /var/lib/ceph/osd/ceph-81/block
> [hqosd7][DEBUG ] Running command: ceph --cluster ceph --name
> client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
> mon getmap -o /var/lib/ceph/osd/ceph-81/activate.monmap
> [hqosd7][DEBUG ] stderr: got monmap epoch 2
> [hqosd7][DEBUG ] Running command: ceph-authtool
> /var/lib/ceph/osd/ceph-81/keyring --create-keyring --name osd.81
> --add-key AQCyyFRcSwWqGBAAKZR8rcWIEknj/o3rsehOdA==
> [hqosd7][DEBUG ] stdout: creating /var/lib/ceph/osd/ceph-81/keyring
> [hqosd7][DEBUG ] stdout: added entity osd.81 auth auth(auid =
> 18446744073709551615 key=AQCyyFRcSwWqGBAAKZR8rcWIEknj/o3rsehOdA== with 0
> caps)
> [hqosd7][DEBUG ] Running command: chown -R ceph:ceph
> /var/lib/ceph/osd/ceph-81/keyring
> [hqosd7][DEBUG ] Running command: chown -R ceph:ceph
> /var/lib/ceph/osd/ceph-81/
> [hqosd7][DEBUG ] Running command: /usr/bin/ceph-osd --cluster ceph
> --osd-objectstore bluestore --mkfs -i 81 --monmap
> /var/lib/ceph/osd/ceph-81/activate.monmap --keyfile - --osd-data
> /var/lib/ceph/osd/ceph-81/ --osd-uuid
> c98a11d1-9b7f-487e-8c69-72fc662927d4 --setuser ceph --setgroup ceph
> [hqosd7][DEBUG ] --> ceph-volume lvm prepare successful for: /dev/sdk
> [hqosd7][DEBUG ] Running command: ceph-bluestore-tool --cluster=ceph
> prime-osd-dir --dev
> /dev/ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1/osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
> --path /var/lib/ceph/osd/ceph-81
> [hqosd7][DEBUG ] Running command: ln -snf
> /dev/ceph-bbe0e44e-afc9-4cf1-9f1a-ed7d20f796c1/osd-block-c98a11d1-9b7f-487e-8c69-72fc662927d4
> /var/lib/ceph/osd/ceph-81/block
> [hqosd7][DEBUG ] Running command: chown -R ceph:ceph /dev/dm-0
> [hqosd7][DEBUG ] Running command: chown -R ceph:ceph
> /var/lib/ceph/osd/ceph-81
> [hqosd7][DEBUG ] Running command: systemctl enable
> ceph-volume@lvm-81-c98a11d1-9b7f-487e-8c69-72fc662927d4
> [hqosd7][DEBUG ] stderr: Created symlink from
> /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-81-c98a11d1-9b7f-487e-8c69-72fc662927d4.service
> to /lib/systemd/system/ceph-volume@.service.
> [hqosd7][DEBUG ] Running command: systemctl start ceph-osd@81
> [hqosd7][DEBUG ] --> ceph-volume lvm activate successful for osd ID: 81
> [hqosd7][DEBUG ] --> ceph-volume lvm create successful for: /dev/sdk
> [hqosd7][INFO ] checking OSD status...
> [hqosd7][DEBUG ] find the location of an executable
> [hqosd7][INFO ] Running command: /usr/bin/ceph --cluster=ceph osd stat
> --format=json
> [hqosd7][WARNIN] there are 2 OSDs down
> [hqosd7][WARNIN] there are 2 OSDs out
> [ceph_deploy.osd][DEBUG ] Host hqosd7 is now ready for osd use.
>
> _________________________________________________________
>
>
> However when I listed out the partitions on the server...this is what I
> found (osd 81 was showing up as 32G as opposed to the 3.7T that the
> drive is):
>
> /dev/sdm1 3.7T 2.9T 756G 80% /var/lib/ceph/osd/ceph-77
> tmpfs 32G 48K 32G 1% /var/lib/ceph/osd/ceph-81
>
> __________________________________________________________
>
> Here is some output from fdisk as well:
>
> Disk /dev/sdm: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: CD3A35E7-CF85-4E79-9911-B80099349C85
>
> Device Start End Sectors Size Type
> /dev/sdm1 20973568 7812939742 7791966175 3.6T Ceph OSD
> /dev/sdm2 2048 20971520 20969473 10G Ceph Journal
>
> Partition table entries are not in disk order.
I am guessing that /dev/sdm* was deployed using ceph-disk (hence you
end up with partitions) but you are using ceph-volume for /dev/sdk
which uses LVM.
When you say osd.81 is showing up as 32gb, you mean the space
available with `df` ? The report will look odd for bluestore + LVM,
because it is
using a tmpfs dir, and bluestore no longer mounts devices, they are
just symlinked inside that tmpfs.
That is why you do see 3.7tb for
/dev/mapper/ceph--bbe0e44e--afc9--4cf1--9f1a--ed7d20f796c1-osd--block--c98a11d1--9b7f--487e--8c69--72fc662927d4
which should be mapped as /var/lib/ceph/osd/ceph-81/block
In short: you did deployed correctly and you do have 3.7TiB, not 32gb
>
>
> Disk /dev/sdk: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
>
>
> Disk
> /dev/mapper/ceph--bbe0e44e--afc9--4cf1--9f1a--ed7d20f796c1-osd--block--c98a11d1--9b7f--487e--8c69--72fc662927d4:
> 3.7 TiB, 4000220971008 bytes, 7812931584 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
>
> __________________________________________________________
>
>
> I would normally spend more time looking around for an answer however
> our cluster is a little tight on space and I really need to replace 2 or
> 3 drives ASAP in order to resolve some of this 'backfillfull' errors I
> am seeing.
>
> I am assuming this isn't normal...however this would be the first
> bluestore osd added to this cluster...so I am not really sure.
>
> Thanks in advance,
>
> Shain
>
> --
> NPR | Shain Miley | Manager of Infrastructure, Digital Media | smiley@xxxxxxx | 202.513.3649
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
