On my OSD node that I built with ceph-ansible, the OSDs are failing to
start after a reboot.
Errors on boot:
~# systemctl status ceph-disk@dev-nvme0n1p16.service
● ceph-disk@dev-nvme0n1p16.service - Ceph disk activation:
/dev/nvme0n1p16
Loaded: loaded (/lib/systemd/system/ceph-disk@.service; static;
vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-01-29 10:14:21
EST; 5min ago
Process: 2819 ExecStart=/bin/sh -c timeout $CEPH_DISK_TIMEOUT flock
/var/lock/ceph-disk-$(basename %f) /usr/sbin/ceph-disk --verbose
--log-stdout trigger --sync %f (code=exited, status=1/FAILURE)
Main PID: 2819 (code=exited, status=1/FAILURE)
Jan 29 10:14:21 osd-08 sh[2819]: main(sys.argv[1:])
Jan 29 10:14:21 osd-08 sh[2819]: File
"/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5674, in main
Jan 29 10:14:21 osd-08 sh[2819]: args.func(args)
Jan 29 10:14:21 osd-08 sh[2819]: File
"/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4874, in
main_trigger
Jan 29 10:14:21 osd-08 sh[2819]: raise Error('return code ' +
str(ret))
Jan 29 10:14:21 osd-08 sh[2819]: ceph_disk.main.Error: Error: return
code 1
Jan 29 10:14:21 osd-08 systemd[1]: ceph-disk@dev-nvme0n1p16.service:
Main process exited, code=exited, status=1/FAILURE
Jan 29 10:14:21 osd-08 systemd[1]: Failed to start Ceph disk
activation: /dev/nvme0n1p16.
Jan 29 10:14:21 osd-08 systemd[1]: ceph-disk@dev-nvme0n1p16.service:
Unit entered failed state.
Jan 29 10:14:21 osd-08 systemd[1]: ceph-disk@dev-nvme0n1p16.service:
Failed with result 'exit-code'.
When I try to run "systemctl start ceph-disk@dev-nvme0n1p16.service"
manually, the command appears to just hang:
Jan 29 10:33:47 osd-08 systemd[1]: Starting Ceph disk activation:
/dev/nvme0n1p16...
Jan 29 10:33:47 osd-08 sh[8093]:
/usr/lib/python2.7/dist-packages/ceph_disk/main.py:5653: UserWarning:
Jan 29 10:33:47 osd-08 sh[8093]:
*******************************************************************************
Jan 29 10:33:47 osd-08 sh[8093]: This tool is now deprecated in
favor of ceph-volume.
Jan 29 10:33:47 osd-08 sh[8093]: It is recommended to use
ceph-volume for OSD deployments. For details see:
Jan 29 10:33:47 osd-08 sh[8093]:
http://docs.ceph.com/docs/master/ceph-volume/#migrating
Jan 29 10:33:47 osd-08 sh[8093]:
*******************************************************************************
Jan 29 10:33:47 osd-08 sh[8093]: warnings.warn(DEPRECATION_WARNING)
Jan 29 10:33:47 osd-08 sh[8093]: main_trigger: main_trigger:
Namespace(cluster='ceph', dev='/dev/nvme0n1p16', dmcrypt=None,
dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at
0x7efdbf275a28>, log_stdout=True, prepend_to_path='/usr/bin',
prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph',
sync=True, sysconfdir='/etc/ceph', verbose=True)
Jan 29 10:33:47 osd-08 sh[8093]: command: Running command:
/sbin/init --version
Jan 29 10:33:47 osd-08 sh[8093]: command_check_call: Running
command: /bin/chown ceph:ceph /dev/nvme0n1p16
Jan 29 10:33:47 osd-08 sh[8093]: command: Running command:
/sbin/blkid -o udev -p /dev/nvme0n1p16
Jan 29 10:33:47 osd-08 sh[8093]: command: Running command:
/sbin/blkid -o udev -p /dev/nvme0n1p16
Jan 29 10:33:47 osd-08 sh[8093]: main_trigger: trigger
/dev/nvme0n1p16 parttype 86a32090-3647-40b9-bbbd-38d8c573aa86 uuid
4bc8ecb2-8c83-4006-9d15-09d213952077
Jan 29 10:33:47 osd-08 sh[8093]: command: Running command:
/usr/sbin/ceph-disk --verbose activate-block --dmcrypt /dev/nvme0n1p16
I've searched around and found that perhaps my permissions are off, but
all appears correct on the disks controlled by ceph:
brw-rw---- 1 root disk 8, 0 Jan 29 10:14 /dev/sda
brw-rw---- 1 ceph ceph 8, 1 Jan 29 10:14 /dev/sda1
brw-rw---- 1 ceph ceph 8, 2 Jan 29 10:14 /dev/sda2
brw-rw---- 1 ceph ceph 8, 5 Jan 29 10:14 /dev/sda5
brw-rw---- 1 root disk 8, 16 Jan 29 10:14 /dev/sdb
brw-rw---- 1 ceph ceph 8, 17 Jan 29 10:14 /dev/sdb1
brw-rw---- 1 ceph ceph 8, 18 Jan 29 10:14 /dev/sdb2
brw-rw---- 1 ceph ceph 8, 21 Jan 29 10:14 /dev/sdb5
brw-rw---- 1 root disk 8, 32 Jan 29 10:14 /dev/sdc
brw-rw---- 1 ceph ceph 8, 33 Jan 29 10:14 /dev/sdc1
brw-rw---- 1 ceph ceph 8, 34 Jan 29 10:14 /dev/sdc2
brw-rw---- 1 ceph ceph 8, 37 Jan 29 10:14 /dev/sdc5
brw-rw---- 1 root disk 8, 48 Jan 29 10:14 /dev/sdd
brw-rw---- 1 ceph ceph 8, 49 Jan 29 10:14 /dev/sdd1
brw-rw---- 1 ceph ceph 8, 50 Jan 29 10:14 /dev/sdd2
brw-rw---- 1 ceph ceph 8, 53 Jan 29 10:14 /dev/sdd5
brw-rw---- 1 root disk 8, 64 Jan 29 10:14 /dev/sde
brw-rw---- 1 ceph ceph 8, 65 Jan 29 10:14 /dev/sde1
brw-rw---- 1 ceph ceph 8, 66 Jan 29 10:14 /dev/sde2
brw-rw---- 1 ceph ceph 8, 69 Jan 29 10:14 /dev/sde5
brw-rw---- 1 root disk 8, 80 Jan 29 10:14 /dev/sdf
brw-rw---- 1 ceph ceph 8, 81 Jan 29 10:14 /dev/sdf1
brw-rw---- 1 ceph ceph 8, 82 Jan 29 10:14 /dev/sdf2
brw-rw---- 1 ceph ceph 8, 85 Jan 29 10:14 /dev/sdf5
brw-rw---- 1 root disk 8, 96 Jan 29 10:14 /dev/sdg
brw-rw---- 1 ceph ceph 8, 97 Jan 29 10:14 /dev/sdg1
brw-rw---- 1 ceph ceph 8, 98 Jan 29 10:14 /dev/sdg2
brw-rw---- 1 ceph ceph 8, 101 Jan 29 10:14 /dev/sdg5
brw-rw---- 1 root disk 8, 112 Jan 29 10:14 /dev/sdh
brw-rw---- 1 ceph ceph 8, 113 Jan 29 10:14 /dev/sdh1
brw-rw---- 1 ceph ceph 8, 114 Jan 29 10:14 /dev/sdh2
brw-rw---- 1 ceph ceph 8, 117 Jan 29 10:14 /dev/sdh5
brw-rw---- 1 root disk 8, 128 Jan 29 10:14 /dev/sdi
brw-rw---- 1 root disk 8, 129 Jan 29 10:14 /dev/sdi1
brw-rw---- 1 root disk 8, 130 Jan 29 10:14 /dev/sdi2
brw-rw---- 1 root disk 8, 144 Jan 29 10:14 /dev/sdj
brw-rw---- 1 root disk 8, 145 Jan 29 10:14 /dev/sdj1
brw-rw---- 1 root disk 8, 146 Jan 29 10:14 /dev/sdj2
brw-rw---- 1 root disk 8, 160 Jan 29 10:14 /dev/sdk
brw-rw---- 1 ceph ceph 8, 161 Jan 29 10:14 /dev/sdk1
brw-rw---- 1 ceph ceph 8, 162 Jan 29 10:14 /dev/sdk2
brw-rw---- 1 ceph ceph 8, 165 Jan 29 10:14 /dev/sdk5
brw-rw---- 1 root disk 8, 176 Jan 29 10:14 /dev/sdl
brw-rw---- 1 ceph ceph 8, 177 Jan 29 10:14 /dev/sdl1
brw-rw---- 1 ceph ceph 8, 178 Jan 29 10:14 /dev/sdl2
brw-rw---- 1 ceph ceph 8, 181 Jan 29 10:14 /dev/sdl5
brw-rw---- 1 root disk 8, 192 Jan 29 10:14 /dev/sdm
brw-rw---- 1 ceph ceph 8, 193 Jan 29 10:14 /dev/sdm1
brw-rw---- 1 ceph ceph 8, 194 Jan 29 10:14 /dev/sdm2
brw-rw---- 1 ceph ceph 8, 197 Jan 29 10:14 /dev/sdm5
brw-rw---- 1 root disk 8, 208 Jan 29 10:14 /dev/sdn
brw-rw---- 1 ceph ceph 8, 209 Jan 29 10:14 /dev/sdn1
brw-rw---- 1 ceph ceph 8, 210 Jan 29 10:14 /dev/sdn2
brw-rw---- 1 ceph ceph 8, 213 Jan 29 10:14 /dev/sdn5
crw------- 1 root root 247, 0 Jan 29 10:14 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Jan 29 10:14 /dev/nvme0n1
brw-rw---- 1 ceph ceph 259, 1 Jan 29 10:14 /dev/nvme0n1p1
brw-rw---- 1 ceph ceph 259, 10 Jan 29 10:14 /dev/nvme0n1p10
brw-rw---- 1 ceph ceph 259, 11 Jan 29 10:14 /dev/nvme0n1p11
brw-rw---- 1 ceph ceph 259, 12 Jan 29 10:14 /dev/nvme0n1p12
brw-rw---- 1 ceph ceph 259, 13 Jan 29 10:14 /dev/nvme0n1p13
brw-rw---- 1 ceph ceph 259, 14 Jan 29 10:14 /dev/nvme0n1p14
brw-rw---- 1 ceph ceph 259, 15 Jan 29 10:14 /dev/nvme0n1p15
brw-rw---- 1 ceph ceph 259, 16 Jan 29 10:14 /dev/nvme0n1p16
brw-rw---- 1 ceph ceph 259, 17 Jan 29 10:14 /dev/nvme0n1p17
brw-rw---- 1 ceph ceph 259, 18 Jan 29 10:14 /dev/nvme0n1p18
brw-rw---- 1 ceph ceph 259, 19 Jan 29 10:14 /dev/nvme0n1p19
brw-rw---- 1 ceph ceph 259, 2 Jan 29 10:14 /dev/nvme0n1p2
brw-rw---- 1 ceph ceph 259, 20 Jan 29 10:14 /dev/nvme0n1p20
brw-rw---- 1 ceph ceph 259, 21 Jan 29 10:14 /dev/nvme0n1p21
brw-rw---- 1 ceph ceph 259, 22 Jan 29 10:14 /dev/nvme0n1p22
brw-rw---- 1 ceph ceph 259, 23 Jan 29 10:14 /dev/nvme0n1p23
brw-rw---- 1 ceph ceph 259, 24 Jan 29 10:14 /dev/nvme0n1p24
brw-rw---- 1 ceph ceph 259, 3 Jan 29 10:14 /dev/nvme0n1p3
brw-rw---- 1 ceph ceph 259, 4 Jan 29 10:14 /dev/nvme0n1p4
brw-rw---- 1 ceph ceph 259, 5 Jan 29 10:14 /dev/nvme0n1p5
brw-rw---- 1 ceph ceph 259, 6 Jan 29 10:14 /dev/nvme0n1p6
brw-rw---- 1 ceph ceph 259, 7 Jan 29 10:14 /dev/nvme0n1p7
brw-rw---- 1 ceph ceph 259, 8 Jan 29 10:14 /dev/nvme0n1p8
brw-rw---- 1 ceph ceph 259, 9 Jan 29 10:14 /dev/nvme0n1p9
Any ideas? I can provide more logs if necessary.
--
Andre Goree
-=-=-=-=-=-
Email - andre at drenet.net
Website - http://blog.drenet.net
PGP key - http://www.drenet.net/pubkey.html
-=-=-=-=-=-
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
