Re: Mon scrub errors

[Elias Abacioglu <elias.abacioglu@xxxxxxxxxxxxxxxxx>]

Hi,

Sorry for bumping an old thread, but I have the same issue.

Is it the mon.0 that I should reset?

2018-07-02 11:39:51.223731 [ERR]  mon.2 ScrubResult(keys {monmap=48,osd_metadata=52} crc {monmap=3659766006,osd_metadata=1928984927})
2018-07-02 11:39:51.223701 [ERR]  mon.0 ScrubResult(keys {monmap=45,osd_metadata=55} crc {monmap=1049715966,osd_metadata=409731371})
2018-07-02 11:39:51.223666 [ERR]  scrub mismatch
2018-07-02 11:39:51.223632 [ERR]  mon.1 ScrubResult(keys {monmap=48,osd_metadata=52} crc {monmap=672606225,osd_metadata=1928984927})
2018-07-02 11:39:51.223598 [ERR]  mon.0 ScrubResult(keys {monmap=45,osd_metadata=55} crc {monmap=1049715966,osd_metadata=409731371})
2018-07-02 11:39:51.223551 [ERR]  scrub mismatch
2018-07-02 11:39:51.211198 [ERR]  mon.2 ScrubResult(keys {logm=33,mds_health=10,mds_metadata=1,mdsmap=56} crc {logm=3564099269,mds_health=692648600,mds_metadata=2226414273,mdsmap=643765345})
2018-07-02 11:39:51.211148 [ERR]  mon.0 ScrubResult(keys {logm=33,mds_health=6,mds_metadata=1,mdsmap=60} crc {logm=3564099269,mds_health=4085298890,mds_metadata=2226414273,mdsmap=758287940})
2018-07-02 11:39:51.211100 [ERR]  scrub mismatch
2018-07-02 11:39:51.211052 [ERR]  mon.1 ScrubResult(keys {logm=33,mds_health=9,mds_metadata=1,mdsmap=57} crc {logm=3564099269,mds_health=4264808124,mds_metadata=2226414273,mdsmap=704105513})
2018-07-02 11:39:51.211000 [ERR]  mon.0 ScrubResult(keys {logm=33,mds_health=6,mds_metadata=1,mdsmap=60} crc {logm=3564099269,mds_health=4085298890,mds_metadata=2226414273,mdsmap=758287940})
2018-07-02 11:39:51.210944 [ERR]  scrub mismatch

2018-07-03 12:42:21.674471 [ERR]  mon.2 ScrubResult(keys {monmap=24,osd_metadata=76} crc {monmap=578360729,osd_metadata=2641573038})
2018-07-03 12:42:21.674447 [ERR]  mon.0 ScrubResult(keys {monmap=24,osd_metadata=76} crc {monmap=962305203,osd_metadata=2641573038})
2018-07-03 12:42:21.674422 [ERR]  scrub mismatch
2018-07-03 12:42:21.674399 [ERR]  mon.1 ScrubResult(keys {monmap=24,osd_metadata=76} crc {monmap=3891180386,osd_metadata=2641573038})
2018-07-03 12:42:21.674359 [ERR]  mon.0 ScrubResult(keys {monmap=24,osd_metadata=76} crc {monmap=962305203,osd_metadata=2641573038})
2018-07-03 12:42:21.674319 [ERR]  scrub mismatch
2018-07-03 12:42:21.664602 [ERR]  mon.2 ScrubResult(keys {mgrstat=73,monmap=27} crc {mgrstat=199345228,monmap=2433199488})
2018-07-03 12:42:21.664523 [ERR]  mon.0 ScrubResult(keys {mgrstat=73,monmap=27} crc {mgrstat=199345228,monmap=880923984})

2018-07-16 12:42:32.413856 [ERR]  mon.2 ScrubResult(keys {monmap=17,osd_metadata=83} crc {monmap=3927456148,osd_metadata=4212506330})
2018-07-16 12:42:32.413828 [ERR]  mon.0 ScrubResult(keys {monmap=17,osd_metadata=83} crc {monmap=4047155390,osd_metadata=4212506330})
2018-07-16 12:42:32.413800 [ERR]  scrub mismatch
2018-07-16 12:42:32.413769 [ERR]  mon.1 ScrubResult(keys {monmap=17,osd_metadata=83} crc {monmap=797941615,osd_metadata=4212506330})
2018-07-16 12:42:32.413740 [ERR]  mon.0 ScrubResult(keys {monmap=17,osd_metadata=83} crc {monmap=4047155390,osd_metadata=4212506330})
2018-07-16 12:42:32.413689 [ERR]  scrub mismatch
2018-07-16 12:42:32.412795 [ERR]  mon.2 ScrubResult(keys {mgrstat=66,monmap=34} crc {mgrstat=1700428385,monmap=4227348033})
2018-07-16 12:42:32.412768 [ERR]  mon.0 ScrubResult(keys {mgrstat=66,monmap=34} crc {mgrstat=1700428385,monmap=3150674595})
2018-07-16 12:42:32.412741 [ERR]  scrub mismatch 

Thanks,

Elias

On Thu, Apr 5, 2018 at 4:40 PM, kefu chai <tchaikov@xxxxxxxxx> wrote:

On Thu, Apr 5, 2018 at 3:10 PM, Rickard Nilsson
<rickardnilsson88@xxxxxxxxx> wrote:
> Hi,
>
> Im' having a cluster with three moitors, two mds and nine osd. Lately I've
> been getting scrub errors from the monitors;
>
> 2018-04-05 07:26:52.147185 [ERR]  mon.2 ScrubResult(keys
> {osd_pg_creating=1,osdmap=99} crc
> {osd_pg_creating=1404726104,osdmap=3323124730})
> 2018-04-05 07:26:52.147167 [ERR]  mon.0 ScrubResult(keys
> {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc
> {osd_metadata=477302505,osd_pg_creating=1404726104,osdmap=2387598890})
> 2018-04-05 07:26:52.147139 [ERR]  scrub mismatch
> 2018-04-05 07:26:52.144378 [ERR]  mon.2 ScrubResult(keys
> {mgrstat=92,monmap=5,osd_pg_creating=1,osdmap=2} crc
> {mgrstat=2630742218,monmap=4118007020,osd_pg_creating=1404726104,osdmap=940126788})
> 2018-04-05 07:26:52.144360 [ERR]  mon.0 ScrubResult(keys
> {mgrstat=92,monmap=5,osd_metadata=3} crc
> {mgrstat=2630742218,monmap=4118007020,osd_metadata=3256871745})
> 2018-04-05 07:26:52.144334 [ERR]  scrub mismatch
> 2018-04-05 07:26:52.140213 [ERR]  mon.2 ScrubResult(keys {mgr=67,mgrstat=33}
> crc {mgr=1823433831,mgrstat=2050325555})
> 2018-04-05 07:26:52.140193 [ERR]  mon.0 ScrubResult(keys
> {mgr=67,mgr_command_descs=1,mgr_metadata=2,mgrstat=30} crc
> {mgr=1823433831,mgr_command_descs=2758154725,mgr_metadata=2776211204,mgrstat=779157107})
> 2018-04-05 07:26:52.140165 [ERR]  scrub mismatch
> 2018-04-05 07:26:52.120025 [ERR]  mon.2 ScrubResult(keys {logm=23,mdsmap=77}
> crc {logm=1176883332,mdsmap=2557424777})
> 2018-04-05 07:26:52.120008 [ERR]  mon.0 ScrubResult(keys
> {logm=23,mds_health=4,mds_metadata=1,mdsmap=72} crc
> {logm=1176883332,mds_health=984298828,mds_metadata=395047004,mdsmap=809526521})
> 2018-04-05 07:26:52.119982 [ERR]  scrub mismatch
>
> What can I do to correct the errors?

Rickard, probably you can nuke mon.2, and re-add it ? so its monstore
can be sync'ed from other two monitor (for instance, mon.0)? for how
to remove/add monitors , see
http://docs.ceph.com/docs/kraken/rados/operations/add-or-rm-mons/ .

>
> Ceph version 12.2.3
>
> Best regards
> //Rickard
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



--
Regards
Kefu Chai
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com