Hi there,
I have ceph cluster created by ceph-volume - bluestore, in every
node is 12 HDD and 1 NVMe, which is divided to 24 LVM partition
for DB and WAL.
I've turned this cluster to 'ceph orch' management, then I've
moved to quincy release (now I'm using a 17.2.5 version).
I had to move whole cluster to another addreses and another
hostnames.
MON, MGR and MDS goes without problem, but OSD was really pain
process :-(
Now I have cluster with this problem:
# ceph orch ps
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
mds.cephfs.mon1.ulytsa mon1 running (11w) 3m ago 11w 5609M - 17.2.5 cc65afd6173a db1aa336263a
mds.cephfs.mon2.zxhxqk mon2 running (11w) 3m ago 11w 33.1M - 17.2.5 cc65afd6173a 5b9ced4a4b71
mds.cephfs.mon3.rpkvlt mon3 running (11w) 3m ago 11w 32.4M - 17.2.5 cc65afd6173a 045e23f124aa
mgr.mon1.buqyga mon1 *:8080 running (11w) 3m ago 11w 2300M - 17.2.5 cc65afd6173a 9577239527b5
mgr.mon2.goghws mon2 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 4fb1ae26765e
mgr.mon3.slpgay mon3 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 06e491084a5e
mon.mon1 mon1 running (11w) 3m ago 11w 1576M 2048M 17.2.5 cc65afd6173a 2f18c737faa9
mon.mon2 mon2 running (11w) 3m ago 11w 1598M 2048M 17.2.5 cc65afd6173a 31091cbbfb8e
mon.mon3 mon3 running (11w) 3m ago 11w 1463M 2048M 17.2.5 cc65afd6173a 4d0b094c9ca1
osd.0 osd1 running (9w) 3m ago 10w 5133M 3745M 17.2.5 cc65afd6173a 3b28e48d3630
osd.1 osd1 running (7w) 3m ago 10w 5425M 3745M 17.2.5 cc65afd6173a 3336ccdfd232
osd.2 osd1 running (9w) 3m ago 10w 5223M 3745M 17.2.5 cc65afd6173a e8fc077aef59
osd.3 osd1 running (9w) 3m ago 10w 5050M 3745M 17.2.5 cc65afd6173a 4fbf34450237
osd.4 osd1 running (9w) 3m ago 10w 7526M 3745M 17.2.5 cc65afd6173a a4875c354540
osd.5 osd1 running (9w) 3m ago 10w 4854M 3745M 17.2.5 cc65afd6173a b006526228ae
osd.6 osd1 running (9w) 3m ago 10w 6498M 3745M 17.2.5 cc65afd6173a 4c326271e188
osd.7 osd1 running (9w) 3m ago 10w 4410M 3745M 17.2.5 cc65afd6173a ca0f3ce31031
osd.8 osd1 running (9w) 3m ago 10w 7337M 3745M 17.2.5 cc65afd6173a 99269a832819
osd.9 osd1 running (9w) 3m ago 10w 4717M 3745M 17.2.5 cc65afd6173a f39ce0bb5316
osd.10 osd1 running (9w) 3m ago 10w 4295M 3745M 17.2.5 cc65afd6173a 0871793fa261
osd.11 osd1 running (9w) 3m ago 10w 5552M 3745M 17.2.5 cc65afd6173a 32a8b589b3bd
osd.24 osd3 running (109m) 3m ago 6M 3306M 3745M 17.2.5 cc65afd6173a 466d80a55d96
osd.25 osd3 running (109m) 3m ago 6M 3145M 3745M 17.2.5 cc65afd6173a b1705621116a
osd.26 osd3 running (109m) 3m ago 6M 3063M 3745M 17.2.5 cc65afd6173a c30253a1a83f
osd.27 osd3 running (109m) 3m ago 6M 3257M 3745M 17.2.5 cc65afd6173a aa0a647d93f1
osd.28 osd3 running (109m) 3m ago 6M 2244M 3745M 17.2.5 cc65afd6173a d3c68ed6572b
osd.29 osd3 running (109m) 3m ago 6M 3509M 3745M 17.2.5 cc65afd6173a 2c425b17abf7
osd.30 osd3 running (109m) 3m ago 6M 3814M 3745M 17.2.5 cc65afd6173a 44747256b34a
osd.31 osd3 running (109m) 3m ago 6M 2958M 3745M 17.2.5 cc65afd6173a b7b7946fa24e
osd.32 osd3 running (109m) 3m ago 6M 3016M 3745M 17.2.5 cc65afd6173a fc9c024fed4f
osd.33 osd3 running (109m) 3m ago 6M 5366M 3745M 17.2.5 cc65afd6173a edc2dbd9c556
osd.34 osd3 running (109m) 3m ago 6M 4577M 3745M 17.2.5 cc65afd6173a 46d7668742cf
osd.35 osd3 running (109m) 3m ago 6M 2538M 3745M 17.2.5 cc65afd6173a 96a15a9ad3d7
osd.36 osd4 running (103m) 3m ago 8w 2707M 3745M 17.2.5 cc65afd6173a adf884af609b
osd.37 osd4 running (103m) 3m ago 6M 3347M 3745M 17.2.5 cc65afd6173a 8f824026c6ae
osd.38 osd4 running (103m) 3m ago 6M 3377M 3745M 17.2.5 cc65afd6173a 2a70c0b860ff
osd.39 osd4 running (103m) 3m ago 6M 2814M 3745M 17.2.5 cc65afd6173a 4d5833f1faaf
osd.40 osd4 running (103m) 3m ago 6M 2633M 3745M 17.2.5 cc65afd6173a e2e492c4f4a6
osd.41 osd4 running (103m) 3m ago 6M 2141M 3745M 17.2.5 cc65afd6173a 80f67faf1238
osd.42 osd4 running (103m) 3m ago 6M 1527M 3745M 17.2.5 cc65afd6173a c1aef5891ad5
osd.43 osd4 running (103m) 3m ago 6M 1927M 3745M 17.2.5 cc65afd6173a b092b03f211d
osd.44 osd4 running (103m) 3m ago 6M 3332M 3745M 17.2.5 cc65afd6173a 9309d93c80da
osd.45 osd4 running (103m) 3m ago 6M 3172M 3745M 17.2.5 cc65afd6173a 8a47d7c155cd
osd.46 osd4 running (103m) 3m ago 6M 2545M 3745M 17.2.5 cc65afd6173a 0550b66a837f
osd.47 osd4 running (103m) 3m ago 6M 3010M 3745M 17.2.5 cc65afd6173a 52328564e503
osd.48 osd5 running (8w) 8m ago 8w 6054M 3745M 17.2.5 cc65afd6173a c5494c95566e
osd.49 osd5 running (8w) 8m ago 8w 5648M 3745M 17.2.5 cc65afd6173a cebc3405950e
osd.50 osd5 running (8w) 8m ago 8w 4898M 3745M 17.2.5 cc65afd6173a 44bd2f018476
osd.51 osd5 running (8w) 8m ago 8w 5066M 3745M 17.2.5 cc65afd6173a 008cb9d89e68
osd.52 osd5 running (8w) 8m ago 8w 6325M 3745M 17.2.5 cc65afd6173a db3d01bbfce7
osd.53 osd5 running (8w) 8m ago 8w 6040M 3745M 17.2.5 cc65afd6173a 215c70cf7ca3
osd.54 osd5 running (8w) 8m ago 8w 5147M 3745M 17.2.5 cc65afd6173a 92a4bc3eae20
osd.55 osd5 running (8w) 8m ago 8w 6590M 3745M 17.2.5 cc65afd6173a ce07fe507b47
osd.56 osd5 running (8w) 8m ago 8w 3838M 3745M 17.2.5 cc65afd6173a 540cf969419d
osd.57 osd5 running (8w) 8m ago 8w 4820M 3745M 17.2.5 cc65afd6173a 89653e57e85c
osd.58 osd5 running (8w) 8m ago 8w 5016M 3745M 17.2.5 cc65afd6173a c19d06b09533
osd.59 osd5 running (8w) 8m ago 8w 6547M 3745M 17.2.5 cc65afd6173a 8c387d3d06a6
osd.60 osd6 running (99m) 3m ago 6M 2050M 3745M 17.2.5 cc65afd6173a 90c1d1745480
osd.61 osd6 running (99m) 3m ago 6M 2694M 3745M 17.2.5 cc65afd6173a 32da4b4cec45
osd.62 osd6 running (99m) 3m ago 6M 2608M 3745M 17.2.5 cc65afd6173a 43788b932105
osd.63 osd6 running (99m) 3m ago 6M 4048M 3745M 17.2.5 cc65afd6173a 4a6863748bc5
osd.64 osd6 running (99m) 3m ago 6M 2542M 3745M 17.2.5 cc65afd6173a c49dd095aa64
osd.65 osd6 running (99m) 3m ago 6M 2796M 3745M 17.2.5 cc65afd6173a cdb2bed58cf3
osd.66 osd6 running (99m) 3m ago 6M 2129M 3745M 17.2.5 cc65afd6173a c17db58e047f
osd.67 osd6 running (99m) 3m ago 6M 3559M 3745M 17.2.5 cc65afd6173a 46f2527c6112
osd.68 osd6 running (99m) 3m ago 6M 2552M 3745M 17.2.5 cc65afd6173a c4491c1b8de9
osd.69 osd6 running (99m) 3m ago 6M 2750M 3745M 17.2.5 cc65afd6173a 9e03c1d5351e
osd.70 osd6 running (99m) 3m ago 6M 1849M 3745M 17.2.5 cc65afd6173a 3454e09aab70
osd.72 osd6 running (99m) 3m ago 8w 5368M 3745M 17.2.5 cc65afd6173a 0c1ca9a8677d
# ceph orch host ls
HOST ADDR LABELS STATUS
iscsi1 192.168.6.166
iscsi2 192.168.6.167
mon1 192.168.7.208
mon2 192.168.7.209
mon3 192.168.7.210
osd1 192.168.7.214
osd2 192.168.7.215
osd3 192.168.7.216
osd4 192.168.7.217
osd5 192.168.7.218
osd6 192.168.7.219
11 hosts in cluster
# ceph health detail
HEALTH_WARN 12 stray daemon(s) not managed by cephadm; 3 stray host(s) with 35 daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 12 stray daemon(s) not managed by cephadm
stray daemon osd.12 on host osd2 not managed by cephadm
stray daemon osd.13 on host osd2 not managed by cephadm
stray daemon osd.14 on host osd2 not managed by cephadm
stray daemon osd.15 on host osd2 not managed by cephadm
stray daemon osd.16 on host osd2 not managed by cephadm
stray daemon osd.17 on host osd2 not managed by cephadm
stray daemon osd.18 on host osd2 not managed by cephadm
stray daemon osd.19 on host osd2 not managed by cephadm
stray daemon osd.20 on host osd2 not managed by cephadm
stray daemon osd.21 on host osd2 not managed by cephadm
stray daemon osd.22 on host osd2 not managed by cephadm
stray daemon osd.23 on host osd2 not managed by cephadm
[WRN] CEPHADM_STRAY_HOST: 3 stray host(s) with 35 daemon(s) not managed by cephadm
stray host c-osd3 has 12 stray daemons: ['osd.24', 'osd.25', 'osd.26', 'osd.27', 'osd.28', 'osd.29', 'osd.30', 'osd.31', 'osd.32', 'osd.33', 'osd.34', 'osd.35']
stray host c-osd4 has 12 stray daemons: ['osd.36', 'osd.37', 'osd.38', 'osd.39', 'osd.40', 'osd.41', 'osd.42', 'osd.43', 'osd.44', 'osd.45', 'osd.46', 'osd.47']
stray host c-osd6 has 11 stray daemons: ['osd.60', 'osd.61', 'osd.62', 'osd.63', 'osd.64', 'osd.65', 'osd.66', 'osd.67', 'osd.68', 'osd.69', 'osd.70']
Comments:
1) c-osd3, c-osd4, c-osd6 are old names (with old addresses) which
are not in cluster now, as you can see from 'ceph orch ps'
listing.
2) On the osd1 machine I've tried create osd daemons bych 'ceph
orch', but I cannot create with one NVMe for DB and WAL together,
thus I have OSD without cached DB and WAL.
3) On the osd2, there is a OSD daemons created by ceph-volume
4) On the other hosts I've renamed host, moved to another IP
address, 'ceph orch rm c-osdx', 'ceph orch add osdx', in LVM
changed all c-osdx tags to osdx tags, now I've changed in
/var/lib/ceph/uuid/osd.x/unit.run and unit.poststop every c-osdx
to osdx.
And now questions:
Is there a way to definitelly remove c-osd3, c-osd4 and c-osd6
from 'ceph health detail' and manage this host via 'ceph orch'?
On the osd2, there is a problem to adopt OSD created by
ceph-volume to ceph orchestrator:
cephadm adopt --style legacy -n osd.12
Pulling container image quay.io/ceph/ceph:v17...
Found online OSD at //var/lib/ceph/osd/ceph-12/fsid
objectstore_type is bluestore
Stopping old systemd unit ceph-osd@12...
Disabling old systemd unit ceph-osd@12...
Moving data...
Traceback (most recent call last):
File "/usr/sbin/cephadm", line 9468, in <module>
main()
File "/usr/sbin/cephadm", line 9456, in main
r = ctx.func(ctx)
File "/usr/sbin/cephadm", line 2135, in _default_image
return func(ctx)
File "/usr/sbin/cephadm", line 6582, in command_adopt
command_adopt_ceph(ctx, daemon_type, daemon_id, fsid)
File "/usr/sbin/cephadm", line 6755, in command_adopt_ceph
os.rmdir(data_dir_src)
OSError: [Errno 39] Directory not empty: '//var/lib/ceph/osd/ceph-12'
Thanks a lot.
Sincerely
Jan Marek
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
