On 05/28/2018 05:43 PM, kefu chai wrote:
On Mon, May 28, 2018 at 7:40 PM, Joao Eduardo Luis <joao@xxxxxxx> wrote:
Hello all,
I've bumped into log messages on the monitors showing epochs as hexadecimal.
And being inconsistent at that, mind you, as consecutive messages will end
up having different representations of decimal numbers; e.g.,
2018-05-25 23:08:15.483 7f5df53dc700 20 mon.b@3(peon).osd e10
send_pg_creates not using stale creating_pgs@f
2018-05-25 23:08:15.483 7f5df53dc700 20 mon.b@3(peon).osd e10
send_pg_creates not using stale creating_pgs@f
2018-05-25 23:08:15.483 7f5df53dc700 20 mon.b@3(peon).osd e10
send_pg_creates not using stale creating_pgs@f
2018-05-25 23:08:15.487 7f5df5bdd700 20 mon.b@3(peon).osd e16
send_pg_creates osd.1 from 16 : epoch 16 4 pgs
Funny enough, in a teuthology cluster this only affect a couple of different
monitors. As far as I can see, 7 out of 9 monitors are not showing symptoms
of this.
Given not all the monitors are affected, and that I didn't see any specific
change as of late in our PRs, I'm tempted to rule out a code change in Ceph
as being the culprit. But... what could it be? Different stl versions due to
different distros? Compilers? Cosmic rays?
as discussed over IRC. we are caching the "t_os.streambuf", so the
fmtflag persists across log entries. and there is chance we have
unbalanced std::hex and std::dec, for instance,
https://github.com/ceph/ceph/blob/master/src/osd/PrimaryLogPG.cc#L1302
. i will try to identify them and put the missing std::dec(s) back.
Fwiw, a few more instances that popped up while going through logs:
2018-05-25 23:14:20.701 7f5df53dc700 1 -- 172.21.15.172:6790/0 -->
172.21.15.163:6809/2ff4 -- osd_map(ad..ae src has 1..ae) v4 --
0x56507dad6000 con 0
2018-05-25 23:14:21.705 7f5df53dc700 1 -- 172.21.15.172:6790/0 -->
172.21.15.163:6809/2ff4 -- osd_map(ae..af src has 1..af) v4 --
0x56507dad7080 con 0
2018-05-25 23:14:22.717 7f5df53dc700 1 -- 172.21.15.172:6790/0 -->
172.21.15.163:6809/2ff4 -- osd_map(af..b0 src has 1..b0) v4 --
0x56507dad62c0 con 0
Let me know if you want a hand going through the code.
-Joao
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
