Sorry, dont know where to find mds logs. I just found some logs in /var/log/ceph/ceph-volume.log from mds pod and here it is(just a piece):
...
[2023-02-15 12:09:07,460][ceph_volume.main][INFO ] Running command: ceph-volume inventory --format json /dev/sda3
[2023-02-15 12:09:07,461][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE
[2023-02-15 12:09:07,471][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda2 /dev/sda2 part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda3 /dev/sda3 part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda4 /dev/sda4 part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sdb /dev/sdb disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sr0 /dev/sr0 rom
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/ceph--893e62b0--dc66--4db6--9487--404cad5907d6-osd--block--19ef4373--d828--4259--9159--86f2e89f4397 lvm
[2023-02-15 12:09:07,479][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/sda3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
[2023-02-15 12:09:07,525][ceph_volume.process][INFO ] stderr Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical volume symlinks in device directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] stderr Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list by scanning device directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/sda3
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] stdout NAME="sda3" KNAME="sda3" MAJ:MIN="8:3" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="1M" STATE="" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="bfq" TYPE="part" DISC-ALN="0" DISC-GRAN="4K" DISC-MAX="1G" DISC-ZERO="0" PKNAME="sda" PARTLABEL="BIOS-BOOT"
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/sda3
[2023-02-15 12:09:07,553][ceph_volume.process][INFO ] stdout /dev/sda3: PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="BIOS-BOOT" PART_ENTRY_UUID="860433f4-8158-432c-88aa-e42901086299" PART_ENTRY_TYPE="21686148-6449-6e6f-744e-656564454649" PART_ENTRY_NUMBER="3" PART_ENTRY_OFFSET="1048576" PART_ENTRY_SIZE="2048" PART_ENTRY_DISK="8:0"
[2023-02-15 12:09:07,554][ceph_volume.process][INFO ] Running command: /usr/sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size /dev/sda3
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical volume symlinks in device directory.
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list by scanning device directory.
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] stderr Failed to find device for physical volume "/dev/sda3".
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] stderr unable to read label for /dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,672][ceph_volume.process][INFO ] stderr unable to read label for /dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,673][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/sda3
[2023-02-15 12:09:07,687][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/disk/by-partuuid/860433f4-8158-432c-88aa-e42901086299 /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3 /dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:0-part3 /dev/disk/by-partlabel/BIOS-BOOT
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout DEVNAME=/dev/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout DEVPATH=/devices/pci0000:00/0000:00:06.0/virtio2/host2/target2:0:0/2:0:0:0/block/sda/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout DEVTYPE=partition
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_BUS=scsi
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_MODEL=QEMU_HARDDISK
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_PART_ENTRY_DISK=8:0
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_PART_ENTRY_NAME=BIOS-BOOT
....
Also after some manipulation our filesystem now fine(few steps later we create new metadata pool and restore current from new). But mds still read only. Our ceph shows only this issue :
cluster:
id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119
health: HEALTH_WARN
1 MDSs are read only
21 daemons have recently crashed
services:
mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 8h)
mgr: a(active, since 102m)
mds: 1/1 daemons up, 1 hot standby
osd: 9 osds: 9 up (since 8h), 9 in (since 7d)
rgw: 1 daemon active (1 hosts, 1 zones)
Thank you
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
