bugs ceph-volume scripting

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have been creating lvm osd's with:
ceph-volume lvm zap --destroy /dev/sdf && ceph-volume lvm create --data 
/dev/sdf --dmcrypt

Because this procedure failed:
ceph-volume lvm zap --destroy /dev/sdf  
(waiting on slow human typing)
ceph-volume lvm create --data /dev/sdf --dmcrypt

However when I was looking at the /var/lib/ceph/osd I expected these lvm 
mounts to list as world writable[1]

So I decided to compare:
[@osd]# cp -a ceph-33 ceph-33.bak2

[@osd]# service ceph-osd@33 stop

[@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 
start

[BUG1]
With the zap && create, I seem to be running the osd without the tmpfs 
mounted?! So that means that if I reboot this node and the content is 
different from the tmpfs. I have a serious problem.

So I am trying to unmount ceph-33
[BUG2]
Wtf ceph-osd@33 is running! Something started (ceph-volume?) the osd, 
without even having a chance to inspect the difference of these folders.
If ceph-volume 'by design' starts the osd, then design is bad, nobody is 
expecting this behaviour and I have no idea what can go wrong if this 
startup data in tmpfs ceph-33 mount is different from the lvm create 
files on the os disk.

Stopping again osd-33
[@osd]# service ceph-osd@33 stop

Trying to unmount again ceph-33
[@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 
stop

[BUG3]
service ceph-volume just does not unmount tmpfs. I have to unmount with
umount /var/lib/ceph/osd/ceph-33


Inspecting differences of both,
[@osd]# ls -l ceph-33.bak2 ceph-33.new
ceph-33.new:
total 28
lrwxrwxrwx 1 ceph ceph  50 Oct  1 10:06 block -> 
/dev/mapper/1K8AX3-D3Gv-VKdY-0wTW-qjgd-txAu-JbNJHo
-rw------- 1 ceph ceph  37 Oct  1 10:06 ceph_fsid
-rw------- 1 ceph ceph  37 Oct  1 10:06 fsid
-rw------- 1 ceph ceph  56 Oct  1 10:06 keyring
-rw------- 1 ceph ceph 106 Oct  1 10:06 lockbox.keyring
-rw------- 1 ceph ceph   6 Oct  1 10:06 ready
-rw------- 1 ceph ceph  10 Oct  1 10:06 type
-rw------- 1 ceph ceph   3 Oct  1 10:06 whoami

ceph-33.bak2:
total 56
-rw-r----- 1 ceph ceph 373 Sep 30 21:23 activate.monmap
lrwxrwxrwx 1 ceph ceph  50 Sep 30 21:23 block -> 
/dev/mapper/1K8AX3-D3Gv-VKdY-0wTW-qjgd-txAu-JbNJHo
-rw------- 1 ceph ceph   2 Sep 30 21:23 bluefs
-rw------- 1 ceph ceph  37 Sep 30 21:23 ceph_fsid
-rw-r----- 1 ceph ceph  37 Sep 30 21:23 fsid
-rw------- 1 ceph ceph  56 Sep 30 21:23 keyring
-rw------- 1 ceph ceph   8 Sep 30 21:23 kv_backend
-rw------- 1 ceph ceph 106 Sep 30 21:23 lockbox.keyring
-rw------- 1 ceph ceph  21 Sep 30 21:23 magic
-rw------- 1 ceph ceph   4 Sep 30 21:23 mkfs_done
-rw------- 1 ceph ceph  41 Sep 30 21:23 osd_key
-rw------- 1 ceph ceph   6 Sep 30 21:23 ready
-rw------- 1 ceph ceph   3 Sep 30 21:23 require_osd_release
-rw------- 1 ceph ceph  10 Sep 30 21:23 type
-rw------- 1 ceph ceph   3 Sep 30 21:23 whoami

The contents of the files in new (tmpfs) luckily are the same as in bak2 
(ceph-volume create?). However as you can see I miss quite a few files 
in the tmpfs

So I am giving it a try and start this osd.33 with the tmpfs mounted.

[@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 
start^C
[@osd]# ps -ef | grep ceph-osd | grep 33
[@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 
start
Redirecting to /bin/systemctl start 
ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670.service

And indeed, again ceph-osd is started.
[@osd]# ps -ef | grep ceph-osd | grep 33
ceph     1651105       1 48 11:29 ?        00:00:00 /usr/bin/ceph-osd -f 
--cluster ceph --id 33 --setuser ceph --setgroup ceph


[QUESTION1]
Should I just copy these files like kv_backend, mkfs_done to the tmpfs 
mount? I seem to have these files on other ceph-volume created osd's.

[QUESTION2]
Is there a reasonable explanation for running into such issues, before I 
start thinking this is shitty scripting?


[1]
https://tracker.ceph.com/issues/47549

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux