Hi Loic,
Yep, 95-ceph-osd.rules contains exactly that...
***********
And 95-ceph-osd.rules contains the following ?
# Check gpt partion for ceph tags and activate
ACTION="" SUBSYSTEM=="block", \
ENV{DEVTYPE}=="partition", \
ENV{ID_PART_TABLE_TYPE}=="gpt", \
RUN+="/usr/sbin/ceph-disk-udev $number $name $parent"
On 17/12/2015 08:29, Jesper Thorhauge wrote:
> Hi Loic,
>
> osd's are on /dev/sda and /dev/sdb, journal's is on /dev/sdc (sdc3 / sdc4).
>
> sgdisk for sda shows;
>
> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
> Partition unique GUID: E85F4D92-C8F1-4591-BD2A-AA43B80F58F6
> First sector: 2048 (at 1024.0 KiB)
> Last sector: 1953525134 (at 931.5 GiB)
> Partition size: 1953523087 sectors (931.5 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph data'
>
> for sdb
>
> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
> Partition unique GUID: C83B5AA5-FE77-42F6-9415-25CA0266FB7F
> First sector: 2048 (at 1024.0 KiB)
> Last sector: 1953525134 (at 931.5 GiB)
> Partition size: 1953523087 sectors (931.5 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph data'
>
> for /dev/sdc3
>
> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
> Partition unique GUID: C34D4694-B486-450D-B57F-DA24255F0072
> First sector: 935813120 (at 446.2 GiB)
> Last sector: 956293119 (at 456.0 GiB)
> Partition size: 20480000 sectors (9.8 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph journal'
>
> for /dev/sdc4
>
> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
> Partition unique GUID: 1E9D527F-0866-4284-B77C-C1CB04C5A168
> First sector: 956293120 (at 456.0 GiB)
> Last sector: 976773119 (at 465.8 GiB)
> Partition size: 20480000 sectors (9.8 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph journal'
>
> 60-ceph-partuuid-workaround.rules is located in /lib/udev/rules.d, so it seems correct to me.
>
> after a reboot, /dev/disk/by-partuuid is;
>
> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168
> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072
> lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1
> lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1
>
> i dont know how to verify the symlink of the journal file - can you guide me on that one?
>
> Thank :-) !
>
> /Jesper
>
> **************
>
> Hi,
>
> On 17/12/2015 07:53, Jesper Thorhauge wrote:
>> Hi,
>>
>> Some more information showing in the boot.log;
>>
>> 2015-12-16 07:35:33.289830 7f1b990ad800 -1 filestore(/var/lib/ceph/tmp/mnt.aWZTcE) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.aWZTcE/journal: (22) Invalid argument
>> 2015-12-16 07:35:33.289842 7f1b990ad800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22
>> 2015-12-16 07:35:33.289883 7f1b990ad800 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.aWZTcE: (22) Invalid argument
>> ERROR:ceph-disk:Failed to activate
>> ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '7', '--monmap', '/var/lib/ceph/tmp/mnt.aWZTcE/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.aWZTcE', '--osd-journal', '/var/lib/ceph/tmp/mnt.aWZTcE/journal', '--osd-uuid', 'c83b5aa5-fe77-42f6-9415-25ca0266fb7f', '--keyring', '/var/lib/ceph/tmp/mnt.aWZTcE/keyring']' returned non-zero exit status 1
>> ceph-disk: Error: One or more partitions failed to activate
>>
>> Maybe related to the "(22) Invalid argument" part..?
>
> After a reboot the symlinks are reconstructed and if they are still incorrect, it means there is an inconsistency somewhere else. To debug the problem, could you mount /dev/sda1 and verify the symlink of the journal file ? Then verify the content of /dev/disk/by-partuuid. And also display the partition information with sgdisk -i 1 /dev/sda and sgdisk -i 2 /dev/sda. Are you collocating your journal with the data, on the same disk ? Or are they on two different disks ?
>
> git log --no-merges --oneline tags/v0.94.3..tags/v0.94.5 udev
>
> shows nothing, meaning there has been no change to udev rules. There is one change related to the installation of the udev rules https://github.com/ceph/ceph/commit/4eb58ad2027148561d94bb43346b464b55d041a6. Could you double check 60-ceph-partuuid-workaround.rules is installed where it should ?
>
> Cheers
>
>>
>> /Jesper
>>
>> *********************
>>
>> Hi,
>>
>> I have done several reboots, and it did not lead to healthy symlinks :-(
>>
>> /Jesper
>>
>> ************
>>
>> Hi,
>>
>> On 16/12/2015 07:39, Jesper Thorhauge wrote:
>>> Hi,
>>>
>>> A fresh server install on one of my nodes (and yum update) left me with CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2.
>>>
>>> "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but "ceph-disk activate / dev/sda1" fails. I have traced the problem to "/dev/disk/by-partuuid", where the journal symlinks are broken;
>>>
>>> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168
>>> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072
>>> lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1
>>> lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1
>>>
>>> Re-creating them manually wont survive a reboot. Is this a problem with the udev rules in Ceph 0.94.3+?
>>
>> This usually is a symptom of something else going wrong (i.e. it is possible to confuse the kernel into creating the wrong symbolic links). The correct symlinks should be set when you reboot.
>>
>>> Hope that somebody can help me :-)
>>
>> Please let us know if rebooting leads to healthy symlinks.
>>
>> Cheers
>>>
>>> Thanks!
>>>
>>> Best regards,
>>> Jesper
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Loïc Dachary, Artisan Logiciel Libre
# Check gpt partion for ceph tags and activate
ACTION="" SUBSYSTEM=="block", \
ENV{DEVTYPE}=="partition", \
ENV{ID_PART_TABLE_TYPE}=="gpt", \
RUN+="/usr/sbin/ceph-disk-udev $number $name $parent"
On 17/12/2015 08:29, Jesper Thorhauge wrote:
> Hi Loic,
>
> osd's are on /dev/sda and /dev/sdb, journal's is on /dev/sdc (sdc3 / sdc4).
>
> sgdisk for sda shows;
>
> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
> Partition unique GUID: E85F4D92-C8F1-4591-BD2A-AA43B80F58F6
> First sector: 2048 (at 1024.0 KiB)
> Last sector: 1953525134 (at 931.5 GiB)
> Partition size: 1953523087 sectors (931.5 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph data'
>
> for sdb
>
> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
> Partition unique GUID: C83B5AA5-FE77-42F6-9415-25CA0266FB7F
> First sector: 2048 (at 1024.0 KiB)
> Last sector: 1953525134 (at 931.5 GiB)
> Partition size: 1953523087 sectors (931.5 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph data'
>
> for /dev/sdc3
>
> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
> Partition unique GUID: C34D4694-B486-450D-B57F-DA24255F0072
> First sector: 935813120 (at 446.2 GiB)
> Last sector: 956293119 (at 456.0 GiB)
> Partition size: 20480000 sectors (9.8 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph journal'
>
> for /dev/sdc4
>
> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
> Partition unique GUID: 1E9D527F-0866-4284-B77C-C1CB04C5A168
> First sector: 956293120 (at 456.0 GiB)
> Last sector: 976773119 (at 465.8 GiB)
> Partition size: 20480000 sectors (9.8 GiB)
> Attribute flags: 0000000000000000
> Partition name: 'ceph journal'
>
> 60-ceph-partuuid-workaround.rules is located in /lib/udev/rules.d, so it seems correct to me.
>
> after a reboot, /dev/disk/by-partuuid is;
>
> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168
> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072
> lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1
> lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1
>
> i dont know how to verify the symlink of the journal file - can you guide me on that one?
>
> Thank :-) !
>
> /Jesper
>
> **************
>
> Hi,
>
> On 17/12/2015 07:53, Jesper Thorhauge wrote:
>> Hi,
>>
>> Some more information showing in the boot.log;
>>
>> 2015-12-16 07:35:33.289830 7f1b990ad800 -1 filestore(/var/lib/ceph/tmp/mnt.aWZTcE) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.aWZTcE/journal: (22) Invalid argument
>> 2015-12-16 07:35:33.289842 7f1b990ad800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22
>> 2015-12-16 07:35:33.289883 7f1b990ad800 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.aWZTcE: (22) Invalid argument
>> ERROR:ceph-disk:Failed to activate
>> ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '7', '--monmap', '/var/lib/ceph/tmp/mnt.aWZTcE/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.aWZTcE', '--osd-journal', '/var/lib/ceph/tmp/mnt.aWZTcE/journal', '--osd-uuid', 'c83b5aa5-fe77-42f6-9415-25ca0266fb7f', '--keyring', '/var/lib/ceph/tmp/mnt.aWZTcE/keyring']' returned non-zero exit status 1
>> ceph-disk: Error: One or more partitions failed to activate
>>
>> Maybe related to the "(22) Invalid argument" part..?
>
> After a reboot the symlinks are reconstructed and if they are still incorrect, it means there is an inconsistency somewhere else. To debug the problem, could you mount /dev/sda1 and verify the symlink of the journal file ? Then verify the content of /dev/disk/by-partuuid. And also display the partition information with sgdisk -i 1 /dev/sda and sgdisk -i 2 /dev/sda. Are you collocating your journal with the data, on the same disk ? Or are they on two different disks ?
>
> git log --no-merges --oneline tags/v0.94.3..tags/v0.94.5 udev
>
> shows nothing, meaning there has been no change to udev rules. There is one change related to the installation of the udev rules https://github.com/ceph/ceph/commit/4eb58ad2027148561d94bb43346b464b55d041a6. Could you double check 60-ceph-partuuid-workaround.rules is installed where it should ?
>
> Cheers
>
>>
>> /Jesper
>>
>> *********************
>>
>> Hi,
>>
>> I have done several reboots, and it did not lead to healthy symlinks :-(
>>
>> /Jesper
>>
>> ************
>>
>> Hi,
>>
>> On 16/12/2015 07:39, Jesper Thorhauge wrote:
>>> Hi,
>>>
>>> A fresh server install on one of my nodes (and yum update) left me with CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2.
>>>
>>> "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but "ceph-disk activate / dev/sda1" fails. I have traced the problem to "/dev/disk/by-partuuid", where the journal symlinks are broken;
>>>
>>> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168
>>> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072
>>> lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f -> ../../sdb1
>>> lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 -> ../../sda1
>>>
>>> Re-creating them manually wont survive a reboot. Is this a problem with the udev rules in Ceph 0.94.3+?
>>
>> This usually is a symptom of something else going wrong (i.e. it is possible to confuse the kernel into creating the wrong symbolic links). The correct symlinks should be set when you reboot.
>>
>>> Hope that somebody can help me :-)
>>
>> Please let us know if rebooting leads to healthy symlinks.
>>
>> Cheers
>>>
>>> Thanks!
>>>
>>> Best regards,
>>> Jesper
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Loïc Dachary, Artisan Logiciel Libre
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
