Re: Ceph 12.2.4 MGR spams syslog with "mon failed to return metadata for mds"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I see. Do you need any log or debug output, John?

Kind regards,

Charles Alva
Sent from Gmail Mobile


On Thu, Apr 26, 2018 at 7:46 PM John Spray <jspray@xxxxxxxxxx> wrote:
On Wed, Apr 25, 2018 at 1:42 PM, Charles Alva <charlesalva@xxxxxxxxx> wrote:
> Hi John,
>
> The "ceph mds metadata mds1" produced "Error ENOENT:".

Hmm.  So that points to something going wrong between the MDS and the
monitor -- ceph-mgr doesn't get involved until the metadata is there.

John


> Querying mds metadata
> to mds2 and mds3 worked as expected. It seemed, only the active MDS could
> not be queried by Ceph MGR.
>
> I also stated wrong that Ceph MGR spamming the syslog, it should be the
> ceph-mgr log itself, sorry for the confusion.
>
>
> # ceph -s
>   cluster:
>     id:     b63f4ca1-f5e1-4ac1-a6fc-5ab70c65864a
>     health: HEALTH_OK
>
>   services:
>     mon: 3 daemons, quorum mon1,mon2,mon3
>     mgr: mon1(active), standbys: mon2, mon3
>     mds: cephfs-1/1/1 up  {0=mds1=up:active}, 2 up:standby
>     osd: 14 osds: 14 up, 14 in
>     rgw: 3 daemons active
>
>   data:
>     pools:   10 pools, 248 pgs
>     objects: 583k objects, 2265 GB
>     usage:   6816 GB used, 6223 GB / 13039 GB avail
>     pgs:     247 active+clean
>              1   active+clean+scrubbing+deep
>
>   io:
>     client:   115 kB/s rd, 759 kB/s wr, 22 op/s rd, 24 op/s wr
>
> # ceph mds metadata mds1
> Error ENOENT:
>
> # ceph mds metadata mds2
> {
>     "addr": "10.100.100.115:6800/1861195236",
>     "arch": "x86_64",
>     "ceph_version": "ceph version 12.2.4
> (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)",
>     "cpu": "Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz",
>     "distro": "ubuntu",
>     "distro_description": "Ubuntu 16.04.4 LTS",
>     "distro_version": "16.04",
>     "hostname": "mds2",
>     "kernel_description": "#1 SMP PVE 4.13.13-40 (Fri, 16 Feb 2018 09:51:20
> +0100)",
>     "kernel_version": "4.13.13-6-pve",
>     "mem_swap_kb": "2048000",
>     "mem_total_kb": "2048000",
>     "os": "Linux"
> }
>
> # ceph mds metadata mds3
> {
>     "addr": "10.100.100.116:6800/4180418633",
>     "arch": "x86_64",
>     "ceph_version": "ceph version 12.2.4
> (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)",
>     "cpu": "Intel(R) Xeon(R) CPU E31240 @ 3.30GHz",
>     "distro": "ubuntu",
>     "distro_description": "Ubuntu 16.04.4 LTS",
>     "distro_version": "16.04",
>     "hostname": "mds3",
>     "kernel_description": "#1 SMP PVE 4.13.16-47 (Mon, 9 Apr 2018 09:58:12
> +0200)",
>     "kernel_version": "4.13.16-2-pve",
>     "mem_swap_kb": "4096000",
>     "mem_total_kb": "2048000",
>     "os": "Linux"
> }
>
>
>
>
>
> Kind regards,
>
> Charles Alva
> Sent from Gmail Mobile
>
> On Tue, Apr 24, 2018 at 4:29 PM, John Spray <jspray@xxxxxxxxxx> wrote:
>>
>> On Fri, Apr 20, 2018 at 11:29 AM, Charles Alva <charlesalva@xxxxxxxxx>
>> wrote:
>> > Marc,
>> >
>> > Thanks.
>> >
>> > The mgr log spam occurs even without dashboard module enabled. I never
>> > checked the ceph mgr log before because the ceph cluster is always
>> > healthy.
>> > Based on the ceph mgr logs in syslog, the spam occurred long before and
>> > after I enabled the dashboard module.
>> >
>> >> # ceph -s
>> >>   cluster:
>> >>     id:     xxx
>> >>     health: HEALTH_OK
>> >>
>> >>   services:
>> >>     mon: 3 daemons, quorum mon1,mon2,mon3
>> >>     mgr: mon1(active), standbys: mon2, mon3
>> >>     mds: cephfs-1/1/1 up  {0=mds1=up:active}, 2 up:standby
>> >>     osd: 14 osds: 14 up, 14 in
>> >>     rgw: 3 daemons active
>> >>
>> >>   data:
>> >>     pools:   10 pools, 248 pgs
>> >>     objects: 546k objects, 2119 GB
>> >>     usage:   6377 GB used, 6661 GB / 13039 GB avail
>> >>     pgs:     248 active+clean
>> >>
>> >>   io:
>> >>     client:   25233 B/s rd, 1409 kB/s wr, 6 op/s rd, 59 op/s wr
>> >
>> >
>> >
>> > My ceph mgr log is spam with following log every second. This happens on
>> > 2
>> > separate Ceph 12.2.4 clusters.
>>
>> (I assume that the mon, mgr and mds are all 12.2.4)
>>
>> The "failed to return metadata" part is kind of mysterious.  Do you
>> also get an error if you try to do "ceph mds metadata mds1" by hand?
>> (that's what the mgr is trying to do).
>>
>> If the metadata works when using the CLI by hand, you may have an
>> issue with the mgr's auth caps, check that its mon caps are set to
>> "allow profile mgr".
>>
>> The "unhandled message" part is from a path where the mgr code is
>> ignoring messages from services that don't have any metadata (I think
>> this is actually a bug, as we should be considering these messages as
>> handled even if we're ignoring them).
>>
>> John
>>
>> >> # less +F /var/log/ceph/ceph-mgr.mon1.log
>> >>
>> >>  ...
>> >>
>> >> 2018-04-20 06:21:18.782861 7fca238ff700  1 mgr send_beacon active
>> >> 2018-04-20 06:21:19.050671 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf897d1c00 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:19.051047 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> 2018-04-20 06:21:20.050889 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf897eac00 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:20.051351 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> 2018-04-20 06:21:20.784455 7fca238ff700  1 mgr send_beacon active
>> >> 2018-04-20 06:21:21.050968 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf897d0d00 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:21.051441 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> 2018-04-20 06:21:22.051254 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf897ec100 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:22.051704 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> 2018-04-20 06:21:22.786656 7fca238ff700  1 mgr send_beacon active
>> >> 2018-04-20 06:21:23.051235 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf897d0400 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:23.051712 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> 2018-04-20 06:21:24.051353 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf897e6000 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:24.051971 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> 2018-04-20 06:21:24.788228 7fca238ff700  1 mgr send_beacon active
>> >> 2018-04-20 06:21:25.051642 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf897d1900 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:25.052182 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> 2018-04-20 06:21:26.051641 7fca14809700  0 ms_deliver_dispatch:
>> >> unhandled
>> >> message 0x55bf89835600 mgrreport(mds.mds1 +24-0 packed 214) v5 from
>> >> mds.0
>> >> 10.100.100.114:6800/4132681434
>> >> 2018-04-20 06:21:26.052169 7fca25102700  1 mgr finish mon failed to
>> >> return
>> >> metadata for mds.mds1: (2) No such file or directory
>> >> ...
>> >
>> >
>> > Kind regards,
>> >
>> > Charles Alva
>> > Sent from Gmail Mobile
>> >
>> > On Fri, Apr 20, 2018 at 10:57 AM, Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx>
>> > wrote:
>> >>
>> >>
>> >> Hi Charles,
>> >>
>> >> I am more or less responding to your syslog issue. I don’t have the
>> >> experience on cephfs to give you a reliable advice. So lets wait for
>> >> the
>> >> experts to reply. But I guess you have to give a little more background
>> >> info, like
>> >>
>> >> This happened to running cluster, you didn’t apply any changes to?
>> >> Looks like your dashboard issue is not related to "1 mgr finish mon
>> >> failed to return metadata for mds.mds1"
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Charles Alva [mailto:charlesalva@xxxxxxxxx]
>> >> Sent: vrijdag 20 april 2018 10:33
>> >> To: Marc Roos
>> >> Cc: ceph-users
>> >> Subject: Re: Ceph 12.2.4 MGR spams syslog with "mon failed
>> >> to return metadata for mds"
>> >>
>> >> Hi Marc,
>> >>
>> >> I'm using CephFS and mgr could not get the metadata of the mds. I
>> >> enabled the dashboard module and everytime I visit the ceph filesystem
>> >> page, it got internal error 500.
>> >>
>> >> Kind regards,
>> >>
>> >> Charles Alva
>> >> Sent from Gmail Mobile
>> >>
>> >>
>> >> On Fri, Apr 20, 2018 at 9:24 AM, Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx>
>> >> wrote:
>> >>
>> >>
>> >>
>> >>         Remote syslog server, and buffering writes to the log?
>> >>
>> >>
>> >>         Actually this is another argument to fix logging to syslog a
>> >> bit,
>> >>         because the default syslog is also be set to throttle and group
>> >> the
>> >>
>> >>         messages like:
>> >>
>> >>         Mar 9 17:59:35 db1 influxd: last message repeated 132 times
>> >>
>> >>
>> >>
>> >> https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg45025.htm
>> >> l
>> >> <https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg45025.html>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>         -----Original Message-----
>> >>         From: Charles Alva [mailto:charlesalva@xxxxxxxxx]
>> >>         Sent: vrijdag 20 april 2018 8:08
>> >>         To: ceph-users@xxxxxxxxxxxxxx
>> >>         Subject: Ceph 12.2.4 MGR spams syslog with "mon
>> >> failed
>> >> to
>> >>         return metadata for mds"
>> >>
>> >>         Hi All,
>> >>
>> >>         Just noticed on 2 Ceph Luminous 12.2.4 clusters, Ceph mgr spams
>> >> the
>> >>
>> >>         syslog with lots of "mon failed to return metadata for mds"
>> >> every
>> >>         second.
>> >>
>> >>         ```
>> >>         2018-04-20 06:06:03.951412 7fca238ff700  1 mgr send_beacon
>> >> active
>> >>         2018-04-20 06:06:04.934477 7fca14809700  0 ms_deliver_dispatch:
>> >>         unhandled message 0x55bf897f0a00 mgrreport(mds.mds1 +24-0
>> >> packed
>> >> 214) v5
>> >>         from mds.0 10.100.100.114:6800/4132681434 2018-04-20
>> >> 06:06:04.934937
>> >>         7fca25102700  1 mgr finish mon failed to return metadata for
>> >> mds.mds1:
>> >>         (2) No such file or directory ```
>> >>
>> >>         How to fix this issue? or disable it completely to reduce disk
>> >> IO
>> >> and
>> >>         increase SSD life span?
>> >>
>> >>
>> >>
>> >>         Kind regards,
>> >>
>> >>         Charles Alva
>> >>         Sent from Gmail Mobile
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux