-----Message d'origine----- De : Loic Dachary [mailto:loic@xxxxxxxxxxx] The failure journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 244973de-7472-421c-bb25-4b09d3f8d441 and the udev logs DEBUG:ceph-disk:Journal /dev/sdc2 has OSD UUID 00000000-0000-0000-0000-000000000000 means /usr/bin/ceph-osd -i 0 --get-journal-uuid --osd-journal /dev/sdc2 fails to read the OSD UUID from /dev/sdc2 which means something went wrong when preparing the journal. It would be great if you could send the command you used to prepare the disk and the output (verbose if possible). I think you can reproduce the problem by zapping the disk with ceph-disk zap /dev/sdc and running partx -u if the corresponding entries in /dev/disk/by-partuuid have not been removed. That would also help me fix zap in the context of https://github.com/ceph/ceph/pull/2648 ... or have confirmation that it does not need fixing because it updates correctly on RHEL ;-) Cheers -- Loïc Dachary, Artisan Logiciel Libre [>- FS : -<] Hi Loic, At first, some notes : - I noticed I have to wait at least 1sec before I run partx -u on a prepared disk, otherwise even with that disks won't get properly handled by udev. Maybe some caching somewhere ? - the '-u' option does not seem to exist under RHEL6... so maybe the RHEL6 behaviour was just to include the kernel updating in the -a option, and not anymore ? - this is CentOS 7, i.e RHEL like, but not "pure" RHEL7 even if very close. I Zapped the disk (it seems to work as expected) : [root@ceph1 ~]# ll /dev/disk/by-partuuid/ total 0 lrwxrwxrwx 1 root root 10 Oct 9 15:57 668f92f5-df46-4052-92ba-e8b8f7efd2d9 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Oct 9 15:57 feb09ba1-30a2-44a8-a338-fef39ae6626a -> ../../sdb2 [root@ceph1 ~] partx -u : [root@ceph1 ~]# partx -u /dev/sdc partx: specified range <1:0> does not make sense Right after that, I re-prepared the disk : [root@ceph1 ~]# parted -s /dev/sdc mklabel gpt [root@ceph1 ~]# ceph-disk -v prepare /dev/sdc INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_type INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_type INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size INFO:ceph-disk:Will colocate journal with data on /dev/sdc DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on /dev/sdc INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:5120M --change-name=2:ceph journal --partition-guid=2:46bf261f-7ec3-485e-98c9-3c185de5efb8 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc Information: Moved requested sector from 34 to 2048 in order to align on 2048-sector boundaries. The operation has completed successfully. INFO:ceph-disk:calling partx on prepared device /dev/sdc INFO:ceph-disk:re-reading known partitions will display errors INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdc partx: /dev/sdc: error adding partition 2 INFO:ceph-disk:Running command: /usr/bin/udevadm settle DEBUG:ceph-disk:Journal is GPT partition /dev/disk/by-partuuid/46bf261f-7ec3-485e-98c9-3c185de5efb8 DEBUG:ceph-disk:Creating osd partition on /dev/sdc INFO:ceph-disk:Running command: /usr/sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:436ac41b-8800-466e-98f5-098aa2c64ba9 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be -- /dev/sdc Information: Moved requested sector from 10485761 to 10487808 in order to align on 2048-sector boundaries. The operation has completed successfully. INFO:ceph-disk:Running command: /usr/sbin/partprobe /dev/sdc INFO:ceph-disk:Running command: /usr/bin/udevadm settle DEBUG:ceph-disk:Creating xfs fs on /dev/sdc1 INFO:ceph-disk:Running command: /usr/sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdc1 meta-data=/dev/sdc1 isize=2048 agcount=4, agsize=60686271 blks = sectsz=512 attr=2, projid32bit=1 = crc=0 data = bsize=4096 blocks=242745083, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=0 log =internal log bsize=4096 blocks=118527, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 DEBUG:ceph-disk:Mounting /dev/sdc1 on /var/lib/ceph/tmp/mnt.Jtg9rD with options noatime INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o noatime -- /dev/sdc1 /var/lib/ceph/tmp/mnt.Jtg9rD DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.Jtg9rD DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.Jtg9rD/journal -> /dev/disk/by-partuuid/46bf261f-7ec3-485e-98c9-3c185de5efb8 DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.Jtg9rD INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.Jtg9rD INFO:ceph-disk:Running command: /usr/sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdc Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. The operation has completed successfully. INFO:ceph-disk:calling partx on prepared device /dev/sdc INFO:ceph-disk:re-reading known partitions will display errors INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdc partx: /dev/sdc: error adding partitions 1-2 And it remained prepared/unmounted : [root@ceph1 ~]# ceph-disk list|grep sdc WARNING:ceph-disk:Old blkid does not support ID_PART_ENTRY_* fields, trying sgdisk; may not correctly identify ceph volumes with dmcrypt /dev/sdc : /dev/sdc1 ceph data, prepared, cluster ceph, journal /dev/sdc2 /dev/sdc2 ceph journal, for /dev/sdc1 If I mount the sdc1 partition I see this : [root@ceph1 ~]# ll /mnt/ total 16 -rw-r--r-- 1 root root 37 Oct 10 10:49 ceph_fsid -rw-r--r-- 1 root root 37 Oct 10 10:49 fsid lrwxrwxrwx 1 root root 58 Oct 10 10:49 journal -> /dev/disk/by-partuuid/46bf261f-7ec3-485e-98c9-3c185de5efb8 -rw-r--r-- 1 root root 37 Oct 10 10:49 journal_uuid -rw-r--r-- 1 root root 21 Oct 10 10:49 magic [root@ceph1 ~]# cat /mnt/journal_uuid 46bf261f-7ec3-485e-98c9-3c185de5efb8 And this is the content of /dev/by-partuuid : [root@ceph1 ~]# ll /dev/disk/by-partuuid/ total 0 lrwxrwxrwx 1 root root 10 Oct 10 10:49 436ac41b-8800-466e-98f5-098aa2c64ba9 -> ../../sdc1 lrwxrwxrwx 1 root root 10 Oct 10 10:49 46bf261f-7ec3-485e-98c9-3c185de5efb8 -> ../../sdc2 lrwxrwxrwx 1 root root 10 Oct 9 15:57 668f92f5-df46-4052-92ba-e8b8f7efd2d9 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Oct 9 15:57 feb09ba1-30a2-44a8-a338-fef39ae6626a -> ../../sdb2 Regards Frederic [>- FS : -<] _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |