On the MDS host you can see all cephadm daemon with 'cephadm ls', with
'cephadm logs --name mds.<DAEMON>' you get the logs.
Zitat von kreept.sama@xxxxxxxxx:
Sorry, dont know where to find mds logs. I just found some logs in
/var/log/ceph/ceph-volume.log from mds pod and here it is(just a
piece):
...
[2023-02-15 12:09:07,460][ceph_volume.main][INFO ] Running command:
ceph-volume inventory --format json /dev/sda3
[2023-02-15 12:09:07,461][ceph_volume.process][INFO ] Running
command: /usr/bin/lsblk -plno KNAME,NAME,TYPE
[2023-02-15 12:09:07,471][ceph_volume.process][INFO ] stdout
/dev/sda /dev/sda
disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout
/dev/sda1 /dev/sda1
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout
/dev/sda2 /dev/sda2
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout
/dev/sda3 /dev/sda3
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout
/dev/sda4 /dev/sda4
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout
/dev/sdb /dev/sdb
disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout
/dev/sr0 /dev/sr0
rom
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout
/dev/dm-0
/dev/mapper/ceph--893e62b0--dc66--4db6--9487--404cad5907d6-osd--block--19ef4373--d828--4259--9159--86f2e89f4397
lvm
[2023-02-15 12:09:07,479][ceph_volume.process][INFO ] Running
command: /usr/sbin/lvs --noheadings --readonly --separator=";" -a
--units=b --nosuffix -S lv_path=/dev/sda3 -o
lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
[2023-02-15 12:09:07,525][ceph_volume.process][INFO ] stderr Udev
is running and DM_DISABLE_UDEV environment variable is set.
Bypassing udev, LVM will manage logical volume symlinks in device
directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] stderr Udev
is running and DM_DISABLE_UDEV environment variable is set.
Bypassing udev, LVM will obtain device list by scanning device
directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] Running
command: /usr/bin/lsblk --nodeps -P -o
NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
/dev/sda3
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] stdout
NAME="sda3" KNAME="sda3" MAJ:MIN="8:3" FSTYPE="" MOUNTPOINT=""
LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="1M" STATE=""
OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0"
PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="bfq" TYPE="part"
DISC-ALN="0" DISC-GRAN="4K" DISC-MAX="1G" DISC-ZERO="0" PKNAME="sda"
PARTLABEL="BIOS-BOOT"
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] Running
command: /usr/sbin/blkid -c /dev/null -p /dev/sda3
[2023-02-15 12:09:07,553][ceph_volume.process][INFO ] stdout
/dev/sda3: PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="BIOS-BOOT"
PART_ENTRY_UUID="860433f4-8158-432c-88aa-e42901086299"
PART_ENTRY_TYPE="21686148-6449-6e6f-744e-656564454649"
PART_ENTRY_NUMBER="3" PART_ENTRY_OFFSET="1048576"
PART_ENTRY_SIZE="2048" PART_ENTRY_DISK="8:0"
[2023-02-15 12:09:07,554][ceph_volume.process][INFO ] Running
command: /usr/sbin/pvs --noheadings --readonly --units=b --nosuffix
--separator=";" -o
vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size
/dev/sda3
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev
is running and DM_DISABLE_UDEV environment variable is set.
Bypassing udev, LVM will manage logical volume symlinks in device
directory.
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev
is running and DM_DISABLE_UDEV environment variable is set.
Bypassing udev, LVM will obtain device list by scanning device
directory.
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] stderr Failed
to find device for physical volume "/dev/sda3".
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] Running
command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] stderr unable
to read label for /dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] Running
command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,672][ceph_volume.process][INFO ] stderr unable
to read label for /dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,673][ceph_volume.process][INFO ] Running
command: /usr/sbin/udevadm info --query=property /dev/sda3
[2023-02-15 12:09:07,687][ceph_volume.process][INFO ] stdout
DEVLINKS=/dev/disk/by-partuuid/860433f4-8158-432c-88aa-e42901086299
/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3 /dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:0-part3
/dev/disk/by-partlabel/BIOS-BOOT
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
DEVNAME=/dev/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
DEVPATH=/devices/pci0000:00/0000:00:06.0/virtio2/host2/target2:0:0/2:0:0:0/block/sda/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
DEVTYPE=partition
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_BUS=scsi
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_MODEL=QEMU_HARDDISK
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_PART_ENTRY_DISK=8:0
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_PART_ENTRY_NAME=BIOS-BOOT
....
Also after some manipulation our filesystem now fine(few steps later
we create new metadata pool and restore current from new). But mds
still read only. Our ceph shows only this issue :
cluster:
id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119
health: HEALTH_WARN
1 MDSs are read only
21 daemons have recently crashed
services:
mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 8h)
mgr: a(active, since 102m)
mds: 1/1 daemons up, 1 hot standby
osd: 9 osds: 9 up (since 8h), 9 in (since 7d)
rgw: 1 daemon active (1 hosts, 1 zones)
Thank you
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
