So, currently this is what /var/lib/ceph/osd/ceph-60 shows. Is it not correct? I don't know what I should expect to see.
root@osd1:~# ls -l /var/lib/ceph/osd/ceph-60
total 86252
-rw-r--r-- 1 ceph ceph 384 Nov 2 16:20 activate.monmap
-rw-r--r-- 1 ceph ceph 10737418240 Nov 5 16:32 block
lrwxrwxrwx 1 ceph ceph 14 Nov 2 16:20 block.db -> /dev/ssd0/db60
-rw-r--r-- 1 ceph ceph 2 Nov 2 16:20 bluefs
-rw-r--r-- 1 ceph ceph 37 Nov 2 16:20 ceph_fsid
-rw-r--r-- 1 ceph ceph 37 Nov 2 16:20 fsid
-rw------- 1 ceph ceph 57 Nov 2 16:20 keyring
-rw-r--r-- 1 ceph ceph 8 Nov 2 16:20 kv_backend
-rw-r--r-- 1 ceph ceph 21 Nov 2 16:20 magic
-rw-r--r-- 1 ceph ceph 4 Nov 2 16:20 mkfs_done
-rw-r--r-- 1 ceph ceph 41 Nov 2 16:20 osd_key
-rw-r--r-- 1 ceph ceph 6 Nov 2 16:20 ready
-rw-r--r-- 1 ceph ceph 10 Nov 2 16:20 type
-rw-r--r-- 1 ceph ceph 3 Nov 2 16:20 whoami
The disk I am using for this osd (osd.60) is a 3.7TB hdd.
lsblk shows
sdh 8:112 0 3.7T 0 disk
└─hdd60-data60 252:1 0 3.7T 0 lvm
and "ceph osd tree" shows
60 hdd 3.63689 osd.60 up 1.00000 1.00000
On Mon, Nov 5, 2018 at 11:23 PM, Hector Martin <hector@xxxxxxxxxxxxxx> wrote:
On 11/6/18 6:03 AM, Hayashida, Mami wrote:
> WOW. With you two guiding me through every step, the 10 OSDs in
> question are now added back to the cluster as Bluestore disks!!! Here
> are my responses to the last email from Hector:
>
> 1. I first checked the permissions and they looked like this
>
> root@osd1:/var/lib/ceph/osd/ceph-60# ls -l Wait.
> total 56
> -rw-r--r-- 1 ceph ceph 384 Nov 2 16:20 activate.monmap
> -rw-r--r-- 1 ceph ceph 10737418240 Nov 2 16:20 block
> lrwxrwxrwx 1 ceph ceph 14 Nov 2 16:20 block.db -> /dev/ssd0/db60
That's not right, is it? That's a 10GB raw file being used as a
BlueStore block device. It should be a symlink to an LVM volume, not a file.
Can you check the `block` files/symlinks again for all the OSDs and also
what is mounted on the OSD directories? It should be tmpfs directories
with symlinks to block devices. I'm not sure what happened there.
Mami Hayashida
Research Computing Associate
Research Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida@xxxxxxx
(859)323-7521
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida@xxxxxxx
(859)323-7521
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
