Hi Loic, Patched, and still not working (sorry)... I'm attaching the prepare output, and also a different a "real " udev debug output I captured using " udevadm monitor --environment " (udev.log file) I added a "sync" command in ceph-disk-udev (this did not change a thing), and I noticed that udev script is called 3 times when adding one disk, and that the debug output was captured and then mixed all into one file. This may lead to log mis-interpretation (race conditions ?)... I changed a bit the logging in order to get one file per call and attached those logs to this mail. File timestamps are as follows : File: '/var/log/udev_ceph.log.out.22706' Change: 2014-10-10 15:48:09.136386306 +0200 File: '/var/log/udev_ceph.log.out.22749' Change: 2014-10-10 15:48:11.502425395 +0200 File: '/var/log/udev_ceph.log.out.22750' Change: 2014-10-10 15:48:11.606427113 +0200 Actually, I can reproduce the UUID=0 thing with this command : [root@ceph1 ~]# /usr/sbin/ceph-disk -v activate-journal /dev/sdc2 INFO:ceph-disk:Running command: /usr/bin/ceph-osd -i 0 --get-journal-uuid --osd-journal /dev/sdc2 SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0b 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 DEBUG:ceph-disk:Journal /dev/sdc2 has OSD UUID 00000000-0000-0000-0000-000000000000 INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/disk/by-partuuid/00000000-0000-0000-0000-000000000000 error: /dev/disk/by-partuuid/00000000-0000-0000-0000-000000000000: No such file or directory ceph-disk: Cannot discover filesystem type: device /dev/disk/by-partuuid/00000000-0000-0000-0000-000000000000: Command '/sbin/blkid' returned non-zero exit status 2 Ah - to answer previous mails : - I tried to manually create the gpt partition table to see if things would improve, but this was not the case (I also tried to zero out the start and end of disks, and also to add random data) - running ceph-disk prepare twice does not work, it's just that once every 20 (?) times it "surprisingly does not fail" on this hardware/os combination ;) Regards -----Message d'origine----- De : Loic Dachary [mailto:loic@xxxxxxxxxxx] Envoyé : vendredi 10 octobre 2014 14:37 À : SCHAER Frederic; ceph-users@xxxxxxxxxxxxxx Objet : Re: ceph-dis prepare : UUID=00000000-0000-0000-0000-000000000000 Hi Frederic, To be 100% sure it would be great if you could manually patch your local ceph-disk script and change 'partprobe', into 'partx', '-a', in https://github.com/ceph/ceph/blob/v0.80.6/src/ceph-disk#L1284 ceph-disk zap ceph-disk prepare and hopefully it will show up as it should. It works for me on centos7 but ... Cheers On 10/10/2014 14:33, Loic Dachary wrote: > Hi Frederic, > > It looks like this is just because https://github.com/ceph/ceph/blob/v0.80.6/src/ceph-disk#L1284 should call partx instead of partprobe. The udev debug output makes this quite clear http://tracker.ceph.com/issues/9721 > > I think https://github.com/dachary/ceph/commit/8d914001420e5bfc1e12df2d4882bfe2e1719a5c#diff-788c3cea6213c27f5fdb22f8337096d5R1285 fixes it > > Cheers > > On 09/10/2014 16:29, SCHAER Frederic wrote: >> >> >> -----Message d'origine----- >> De : Loic Dachary [mailto:loic@xxxxxxxxxxx] >> Envoyé : jeudi 9 octobre 2014 16:20 >> À : SCHAER Frederic; ceph-users@xxxxxxxxxxxxxx >> Objet : Re: ceph-dis prepare : UUID=00000000-0000-0000-0000-000000000000 >> >> >> >> On 09/10/2014 16:04, SCHAER Frederic wrote: >>> Hi Loic, >>> >>> Back on sdb, as the sde output was from another machine on which I ran partx -u afterwards. >>> To reply your last question first : I think the SG_IO error comes from the fact that disks are exported as a single disks RAID0 on a PERC 6/E, which does not support JBOD - this is decommissioned hardware on which I'd like to test and validate we can use ceph for our use case... >>> >>> So back on the UUID. >>> It's funny : I retried and ceph-disk prepare worked this time. I tried on another disk, and it failed. >>> There is a difference in the output from ceph-disk : on the failing disk, I have these extra lines after disks are prepared : >>> >>> (...) >>> realtime =none extsz=4096 blocks=0, rtextents=0 >>> Warning: The kernel is still using the old partition table. >>> The new table will be used at the next reboot. >>> The operation has completed successfully. >>> partx: /dev/sdc: error adding partitions 1-2 >>> >>> I didn't have the warning about the old partition tables on the disk that worked. >>> So on this new disk, I have : >>> >>> [root@ceph1 ~]# mount /dev/sdc1 /mnt >>> [root@ceph1 ~]# ll /mnt/ >>> total 16 >>> -rw-r--r-- 1 root root 37 Oct 9 15:58 ceph_fsid >>> -rw-r--r-- 1 root root 37 Oct 9 15:58 fsid >>> lrwxrwxrwx 1 root root 58 Oct 9 15:58 journal -> /dev/disk/by-partuuid/5e50bb8b-0b99-455f-af71-10815a32bfbc >>> -rw-r--r-- 1 root root 37 Oct 9 15:58 journal_uuid >>> -rw-r--r-- 1 root root 21 Oct 9 15:58 magic >>> >>> [root@ceph1 ~]# cat /mnt/journal_uuid >>> 5e50bb8b-0b99-455f-af71-10815a32bfbc >>> >>> [root@ceph1 ~]# sgdisk --info=1 /dev/sdc >>> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown) >>> Partition unique GUID: 244973DE-7472-421C-BB25-4B09D3F8D441 >>> First sector: 10487808 (at 5.0 GiB) >>> Last sector: 1952448478 (at 931.0 GiB) >>> Partition size: 1941960671 sectors (926.0 GiB) >>> Attribute flags: 0000000000000000 >>> Partition name: 'ceph data' >>> >>> [root@ceph1 ~]# sgdisk --info=2 /dev/sdc >>> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown) >>> Partition unique GUID: 5E50BB8B-0B99-455F-AF71-10815A32BFBC >>> First sector: 2048 (at 1024.0 KiB) >>> Last sector: 10485760 (at 5.0 GiB) >>> Partition size: 10483713 sectors (5.0 GiB) >>> Attribute flags: 0000000000000000 >>> Partition name: 'ceph journal' >>> >>> Puzzling, isn't it ? >>> >>> >> >> Yes :-) Just to be 100% sure, when you try to activate this /dev/sdc it shows an error and complains that the journal uuid is 0000-000* etc ? If so could you copy your udev debug output ? >> >> Cheers >> >> [>- FS : -<] >> >> No, when I manually activate the disk instead of attempting to go the udev way, it seems to work : >> [root@ceph1 ~]# ceph-disk activate /dev/sdc1 >> got monmap epoch 1 >> SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0b 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> 2014-10-09 16:21:43.286288 7f2be6a027c0 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 244973de-7472-421c-bb25-4b09d3f8d441, invalid (someone else's?) journal >> SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0b 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0b 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0b 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> 2014-10-09 16:21:43.301957 7f2be6a027c0 -1 filestore(/var/lib/ceph/tmp/mnt.4lJlzP) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory >> 2014-10-09 16:21:43.305941 7f2be6a027c0 -1 created object store /var/lib/ceph/tmp/mnt.4lJlzP journal /var/lib/ceph/tmp/mnt.4lJlzP/journal for osd.47 fsid 70ac4a78-46c0-45e6-8ff9-878b37f50fa1 >> 2014-10-09 16:21:43.305992 7f2be6a027c0 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.4lJlzP/keyring: can't open /var/lib/ceph/tmp/mnt.4lJlzP/keyring: (2) No such file or directory >> 2014-10-09 16:21:43.306099 7f2be6a027c0 -1 created new key in keyring /var/lib/ceph/tmp/mnt.4lJlzP/keyring >> added key for osd.47 >> === osd.47 === >> create-or-move updating item name 'osd.47' weight 0.9 at location {host=ceph1,root=default} to crush map >> Starting Ceph osd.47 on ceph1... >> Running as unit run-12392.service. >> >> The osd then appeared in the osd tree... >> I attached the logs to this email (I just added a set -x in the script called by udev, and redirected the output) >> >> Regards >> > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Loïc Dachary, Artisan Logiciel Libre
Attachment:
udev_ceph.log.out.22706
Description: udev_ceph.log.out.22706
Attachment:
udev_ceph.log.out.22749
Description: udev_ceph.log.out.22749
Attachment:
udev_ceph.log.out.22750
Description: udev_ceph.log.out.22750
[root@ceph1 ~]# ceph-disk zap /dev/sdc
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.
****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
The operation has completed successfully.
[root@ceph1 ~]# ceph-disk -v prepare /dev/sdc
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
INFO:ceph-disk:Will colocate journal with data on /dev/sdc
DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on /dev/sdc
INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:5120M --change-name=2:ceph journal --partition-guid=2:8a19148a-ddae-4bcb-b36a-8b556403ad3d --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc
Information: Moved requested sector from 34 to 2048 in
order to align on 2048-sector boundaries.
The operation has completed successfully.
INFO:ceph-disk:calling partx on prepared device /dev/sdc
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdc
partx: /dev/sdc: error adding partition 2
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
DEBUG:ceph-disk:Journal is GPT partition /dev/disk/by-partuuid/8a19148a-ddae-4bcb-b36a-8b556403ad3d
DEBUG:ceph-disk:Creating osd partition on /dev/sdc
INFO:ceph-disk:Running command: /usr/sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:80506fc1-3c45-4926-a8aa-22cffcc6760d --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be -- /dev/sdc
Information: Moved requested sector from 10485761 to 10487808 in
order to align on 2048-sector boundaries.
The operation has completed successfully.
INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdc
partx: /dev/sdc: error adding partitions 1-2
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
DEBUG:ceph-disk:Creating xfs fs on /dev/sdc1
INFO:ceph-disk:Running command: /usr/sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdc1
meta-data=/dev/sdc1 isize=2048 agcount=4, agsize=60686271 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0
data = bsize=4096 blocks=242745083, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal log bsize=4096 blocks=118527, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
DEBUG:ceph-disk:Mounting /dev/sdc1 on /var/lib/ceph/tmp/mnt.aLNCAP with options noatime
INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o noatime -- /dev/sdc1 /var/lib/ceph/tmp/mnt.aLNCAP
DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.aLNCAP
DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.aLNCAP/journal -> /dev/disk/by-partuuid/8a19148a-ddae-4bcb-b36a-8b556403ad3d
DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.aLNCAP
INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.aLNCAP
INFO:ceph-disk:Running command: /usr/sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdc
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
INFO:ceph-disk:calling partx on prepared device /dev/sdc
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdc
partx: /dev/sdc: error adding partitions 1-2
Attachment:
udev.log
Description: udev.log
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
