Since upgrade to Ceph Luminous (12.2.2) from Jewel we get scrub mismatch
errors every day at the same time (19:25), how can we fix them? Seems to
be the same problem as described at
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-December/023202.html
(can't reply to archived messages), but we don't use MDS.
We can force the error by running ceph scrub. Ceph health shows health
OK all the time.
============================================================================================================================
/var/log/ceph/ceph.log:
============================================================================================================================
2018-01-24 19:25:17.627962 mon.stor01 mon.0 192.168.110.131:6789/0 7015
: cluster [ERR] scrub mismatch
2018-01-24 19:25:17.627981 mon.stor01 mon.0 192.168.110.131:6789/0 7016
: cluster [ERR] mon.0 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-24 19:25:17.628001 mon.stor01 mon.0 192.168.110.131:6789/0 7017
: cluster [ERR] mon.1 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-24 19:25:17.628014 mon.stor01 mon.0 192.168.110.131:6789/0 7018
: cluster [ERR] scrub mismatch
2018-01-24 19:25:17.628029 mon.stor01 mon.0 192.168.110.131:6789/0 7019
: cluster [ERR] mon.0 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-24 19:25:17.628040 mon.stor01 mon.0 192.168.110.131:6789/0 7020
: cluster [ERR] mon.4 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-23 19:25:17.348731 mon.stor01 mon.0 192.168.110.131:6789/0 5199
: cluster [ERR] scrub mismatch
2018-01-23 19:25:17.348751 mon.stor01 mon.0 192.168.110.131:6789/0 5200
: cluster [ERR] mon.0 ScrubResult(keys
{mgrstat=43,monmap=46,osd_metadata=11} crc
{mgrstat=3495389473,monmap=2803006433,osd_metadata=1919422177})
2018-01-23 19:25:17.348764 mon.stor01 mon.0 192.168.110.131:6789/0 5201
: cluster [ERR] mon.1 ScrubResult(keys
{mgrstat=43,monmap=46,osd_metadata=11} crc
{mgrstat=3495389473,monmap=463219445,osd_metadata=1919422177})
2018-01-23 19:25:17.348772 mon.stor01 mon.0 192.168.110.131:6789/0 5202
: cluster [ERR] scrub mismatch
2018-01-23 19:25:17.348780 mon.stor01 mon.0 192.168.110.131:6789/0 5203
: cluster [ERR] mon.0 ScrubResult(keys
{mgrstat=43,monmap=46,osd_metadata=11} crc
{mgrstat=3495389473,monmap=2803006433,osd_metadata=1919422177})
2018-01-23 19:25:17.348795 mon.stor01 mon.0 192.168.110.131:6789/0 5204
: cluster [ERR] mon.4 ScrubResult(keys
{mgrstat=43,monmap=46,osd_metadata=11} crc
{mgrstat=3495389473,monmap=463219445,osd_metadata=1919422177})
2018-01-22 19:25:17.197627 mon.stor01 mon.0 192.168.110.131:6789/0 3406
: cluster [ERR] scrub mismatch
2018-01-22 19:25:17.197649 mon.stor01 mon.0 192.168.110.131:6789/0 3407
: cluster [ERR] mon.0 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
2018-01-22 19:25:17.197663 mon.stor01 mon.0 192.168.110.131:6789/0 3408
: cluster [ERR] mon.1 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
2018-01-22 19:25:17.197678 mon.stor01 mon.0 192.168.110.131:6789/0 3409
: cluster [ERR] scrub mismatch
2018-01-22 19:25:17.197691 mon.stor01 mon.0 192.168.110.131:6789/0 3410
: cluster [ERR] mon.0 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
2018-01-22 19:25:17.197707 mon.stor01 mon.0 192.168.110.131:6789/0 3411
: cluster [ERR] mon.4 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
============================================================================================================================
/var/log/ceph/ceph-mon.log:
============================================================================================================================
2018-01-24 19:25:17.627955 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : scrub mismatch
2018-01-24 19:25:17.627979 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.0 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-24 19:25:17.627998 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.1 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-24 19:25:17.628012 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : scrub mismatch
2018-01-24 19:25:17.628028 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.0 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-24 19:25:17.628039 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.4 ScrubResult(keys
{mgrstat=23,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=5} crc
{mgrstat=947634900,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=2944932770,osdmap=77513567})
2018-01-23 19:25:17.348723 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : scrub mismatch
2018-01-23 19:25:17.348750 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.0 ScrubResult(keys {mgrstat=43,monmap=46,osd_metadata=11}
crc {mgrstat=3495389473,monmap=2803006433,osd_metadata=1919422177})
2018-01-23 19:25:17.348763 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.1 ScrubResult(keys {mgrstat=43,monmap=46,osd_metadata=11}
crc {mgrstat=3495389473,monmap=463219445,osd_metadata=1919422177})
2018-01-23 19:25:17.348771 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : scrub mismatch
2018-01-23 19:25:17.348779 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.0 ScrubResult(keys {mgrstat=43,monmap=46,osd_metadata=11}
crc {mgrstat=3495389473,monmap=2803006433,osd_metadata=1919422177})
2018-01-23 19:25:17.348793 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.4 ScrubResult(keys {mgrstat=43,monmap=46,osd_metadata=11}
crc {mgrstat=3495389473,monmap=463219445,osd_metadata=1919422177})
2018-01-22 19:25:17.197618 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : scrub mismatch
2018-01-22 19:25:17.197646 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.0 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
2018-01-22 19:25:17.197661 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.1 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
2018-01-22 19:25:17.197676 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : scrub mismatch
2018-01-22 19:25:17.197690 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.0 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=2803006433,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
2018-01-22 19:25:17.197706 7f4bd191e700 -1 log_channel(cluster) log
[ERR] : mon.4 ScrubResult(keys
{mgrstat=24,monmap=46,osd_metadata=25,osd_pg_creating=1,osdmap=4} crc
{mgrstat=1702457410,monmap=463219445,osd_metadata=1876741065,osd_pg_creating=1008471957,osdmap=4220436819})
============================================================================================================================
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
