Re: CentOS7 Mounting Problem

[Xavier Villaneau <xvillaneau+ceph@xxxxxxxxx>]

Hi Georgios,

I've had a few issues with automatic mounting on CentOS two months ago, and here are a few tips to how we got automatic mount running with no entries in the fstab. The versions for my test are CentOS 7.1 with Ceph Hammer, kernel 3.10.0-229 and udev/systemd 208.

First, I strongly recommend using `ceph-disk list` as a first test. If all goes well the output should look like this:

[root@ceph-test ~]# ceph-disk list
/dev/sda :
 /dev/sda1 other, xfs, mounted on /boot
 /dev/sda2 other, LVM2_member
/dev/sdb :
 /dev/sdb1 ceph journal, for /dev/sdd1
 /dev/sdb2 ceph journal, for /dev/sde1
 /dev/sdb3 ceph journal, for /dev/sdc1
/dev/sdc :
 /dev/sdc1 ceph data, active, cluster ceph, osd.2, journal /dev/sdb3
/dev/sdd :
 /dev/sdd1 ceph data, active, cluster ceph, osd.1, journal /dev/sdb1
/dev/sde :
 /dev/sde1 ceph data, active, cluster ceph, osd.0, journal /dev/sdb2

If the partitions are not detected as ceph data/journal, then your partitions type UUIDs are not set properly; this is important for the Ceph udev rules to work. An if the data-journal associations are not displayed, you might want to check that the "journal" symlink and "journal_uuid" files in the OSD directory are correct and pointing to the right device. That's if you're using separate partitions as journals, of course.

Then `udevadm`can help you see what exactly is going on in the udev rule when it's run. Try:
    udevadm test -h $(udevadm info -q path /dev/sdc)

(or any other device that's used as data for OSDs)

This command should show you a full log of the events. In our case, the failure was due to a missing keyring file that made the `ceph-disk-activate` call from 95-ceph-osd.rules fail.

Finally, you might also want to try using 60-ceph-partuuid-workaround.rules instead of 60-ceph-by-parttypeuuid.rules if it's the later that is used in your system. The `udevadm test` log should give good clues to whether that's the issue or not.

Kind Regards,
--

Xavier Villaneau

Software Engineer, Concurrent Computer Corporation

On Sat, Apr 1, 2017 at 4:47 AM Georgios Dimitrakakis <giorgis@xxxxxxxxxxxx> wrote:

 Hi,

 just to provide some more feedback on this one and what I 've done to
 solve it, although not sure if this is the most "elegant" solution.

 I have add manually to /etc/fstab on all systems the respective mount
 points for Ceph OSDs, e.g. entries like this:

 UUID=9d2e7674-f143-48a2-bb7a-1c55b99da1f7 /var/lib/ceph/osd/ceph-0 xfs
 defaults               0 0


 Then I 've checked and seen that the "ceph-osd@.service" was "disabled"
 which means that wasn't starting by default.

 Therefore I did modify all respective services on all nodes, with
 commands like:

 systemctl enable ceph-osd@0.service


 Did a reboot on the nodes and all CEPH OSDs were mounted and the
 service was starting by default, so the problem was solved.

 As said I don't know if this is the correct way to do it but for me it
 works.
 I guess that something still goes wrong when the root volume is on LVM
 and all the above that should happen automatically don't happen and
 require manual intervention.

 Looking forward for any comments on this procedure or things that I
 might have missed.

 Regards,

 G.


> Hi Tom and thanks a lot for the feedback.
>
> Indeed my root filesystem is on an LVM volume and I am currently
> running CentOS 7.3.1611 with kernel 3.10.0-514.10.2.el7.x86_64 and
> the
> ceph version is 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)
>
> The 60-ceph-by-parttypeuuid.rules on the system is the same is the
> one on the bug you 've mentioned but unfortunately it still doesn't
> work.
>
> So, are there any more ideas on how to further debug it?
>
> Best,
>
> G.
>
>
>> Are you running the CentOS filesystem as LVM? This
>> (http://tracker.ceph.com/issues/16351 [1]) still seems to be an
>> issue
>> on CentOS 7 that I've seen myself too. After migrating to a standard
>> filesystem layout (i.e. no LVM) the issue disappeared.
>>
>> Regards,
>>
>>  Tom
>>
>> -------------------------
>>
>> FROM: ceph-users  on behalf of Georgios Dimitrakakis
>>  SENT: Thursday, March 23, 2017 10:21:34 PM
>>  TO: ceph-users@xxxxxxxx
>>  SUBJECT: CentOS7 Mounting Problem
>>
>> Hello Ceph community!
>>
>>  I would like some help with a new CEPH installation.
>>
>>  I have install Jewel on CentOS7 and after the reboot my OSDs are
>> not
>>  mount automatically and as a consequence ceph is not operating
>>  normally...
>>
>>  What can I do?
>>
>>  Could you please help me solve the problem?
>>
>>  Regards,
>>
>>  G.
>>  _______________________________________________
>>  ceph-users mailing list
>>  ceph-users@xxxxxxxxxxxxxx
>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [2]
>>
>>
>> Links:
>> ------
>> [1] http://tracker.ceph.com/issues/16351
>> [2] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com