Re: fedora messaging

Michal Novotny <clime@xxxxxxxxxx> · Wed, 15 Aug 2018 15:17:51 +0200

Oh yeah, protocol buffers are useful probably in strongly typed languages with
lots of types to maintain that type information and being able to parse out the
serialized content based on that type information. But in case we are transferring
json, we send the type information in the content itself ({}, [], "", <int>) so we
don't need to know anything else except the content, we just need to decide
how we are going to transform the content into language-native structures. 
It is just an additional thing that I've just realized but I am little bit guessing here.
It might not be relevant. Forgive me if it is not.

On Wed, Aug 15, 2018 at 2:53 PM Michal Novotny <clime@xxxxxxxxxx> wrote:
> Anyway, to summarize, I really really want this to be super easy to use

and just work. I hope we can improve it further and I'd love to hear

your thoughts. Do you think my problem statements and design goals are

reasonable? Given those, do you still feel like sending the schema along

is worthwhile?

I actually no longer think it is worthwhile.

> As a consumer, I can validate the JSON in a message matches the JSON schema

in the same message, but what does that get me? It doesn't seem any

different (on the consumer side) than just parsing the JSON outright and

trying to access whatever deserialized object I get.

I completely agree with this.

Let's go through the problems you mentioned:

1. Make catching accidental schema changes as a publisher easy.

So we can just solve this by registering the scheme with the publisher
first before any content gets published and based on the scheme, the publisher
instance may check if the content intended to be sent conforms to
the scheme, which could catch some bugs before the content
is actually sent. If we require this to be done on publisher side, then
there is actually no reason to send the schema alongside the content
because the check has already been done so consumer already knows
the message is alright when it is received. What should be sent, however,
is a scheme ID, e.g. just a natural number. The scheme ID may be then
used to version the scheme, which would be available somewhere publicly
e.g. in the service docs the same way Github/Gitlab/etc publishes structures
of their webhook messages. It would be basically part of public API of
a service.

2. Make catching mis-behaving publishers on the consuming side easy.

By checking against the scheme on the publisher side, this
shouldn't be necessary. If someone somehow bypasses the
publisher check, at worst the message won't be parsable,
depending on how the message is being parsed. If someone
wants to really make sure the message is what it is supposed
to be, he/she can integrate the schema published on the service
site into the parsing logic but I don't think that's necessary
thing to do (I personally wouldn't do it in my code).

3. Make changing the schema a painless process for publishers and

   consumers.

I think, the only way to do this is to send both content types simultaneously
for some time, each message being marked with its scheme ID. It would be
good if consumer always specified what scheme ID it wants to consume.
If there is a higher scheme ID available in the message, a warning could be printed
maybe even to syslog even so that consumers get the information. At the same time it should
be communicated on the service site or by other means available. I don't think it is possible
to make it any better than this.

I fail to see what's the point of packaging the schemas.
If the message content is in json, then after receiving the message,
I would like to be able to just call json.loads(msg) and work with the resulting structure
as I am used to.

Actually, what I would do in python is that I would make it a munch and then work
with it. Needing to install some additional package and instantiate some high-level
objects just seems clumsy to me in comparison.

In other programming languages, this procedure would be pretty much the same,
I believe as they all probably provide some json implementation.

You mentioned:

> In the current proposal, consumers don't interact with the JSON at all,

but with a higher-level Python API that gives publishers flexibility

when altering their on-the-wire format.

Yes, but with the current proposal if I change the on-the-wire API, I need
to make a new version of the schema, package it and somehow get it to
consumers and make them use the correct version that correctly parses
the new on-the-wire format and translates it correctly to what the consumers
are used to consume? That's seems like something very difficult to get
done.

And also I don't quite see the point. I wouldn't alter the on-the-wire
format if it is not actually what users work with and if I needed to go
through all those steps described above.

If I need to alter the on-the-wire format because application logic
has been somehow changed, then I would like to make the changes
in the high-level API as well so again there is no gain there except
more work with packaging new schemas.

My main point here is that trying to package the schemas to provide
some high-level objects seems to be redundant. I think lots of people would
just welcome to work something really simple, which is already provided in
the language standard library.

For python, If I had to install and import just a single messaging library,
say to what hub, topic, and scheme ID I want to listen and then consume
the incoming messages immediately as munches, I would be super happy.

Actually, it might be the case the scheme ID is redundant as well and
it can be just made part of the topic somehow, in which case the producer
would probably just produce the content twice on a scheme change at
least for some time. "Deprecated by <topic>" flag on an incoming message
would be nice then. Of course, the producer would need to register the two
schemas and mark one of them as deprecated. The framework would then
send two messages simultaneously for him. This might be even easier
solution to the problem. The exact publisher (producer) interface would
need to be thought through.

> The big problem is that right now the majority of messages are not

formatted in a way that makes sense and really need to be changed to be

simple, flat structures that contain the information services need and

nothing they don't. I'd like to get those fixed in a way that doesn't

require massive coordinated changes in apps.

In Copr, for example, we take this as an opportunity to change our
format. If the messaging framework will support format deprecation,
we might go that way as well to avoid sudden change. But we don't
currently have many (or maybe any) consumers so I am not sure it is
necessary for us.

I am not familiar with protocol buffers but to me that thing
seems rather useful, if you want to send the content in a compact
binary form to save as much space as possible. If we will send content,
which can be interpreted as json already, then to make some
higher-level classes and objects on that seems already unnecessary.

I think we could really just take that already existing generic framework
you were talking about (RabbitMQ?) and just make sure we can
check the content against message schemas on producer side (which is
great for catching little bugs) and that we know how a message format can
get deprecated (e.g. by adding "deprecated_by: <topic>" field into each message
by the messaging framework, which should somehow log warnings on
consumer side), also the framework could automatically
transform the messages into some language-native structures:
in python, the munches would probably be the most sexy ones.

The whole "let's package schemas" thing seems like something
we would typically do (because we are packagers) but not as something
that would solve the actual problems you have mentioned. Rather it
makes them more difficult to deal with if I am correct.

I think what you are doing is good but I think most people will
welcome less dependencies and simpler language-native structures.
So if we could make the framework go more into that direction,
that would be great.

clime

On Tue, Aug 14, 2018 at 10:55 AM Jeremy Cline <jeremy@xxxxxxxxxx> wrote:
On 08/13/2018 10:20 PM, Michal Novotny wrote:

> So I got to know on the flock that fedmsg is going to be replaced?

> 

> Anyway, it seems that there is an idea to create schemas for the messages

> and distribute them in packages? And those python packages need to be

> present on producer as well as consumer?

>> JSON schemas

> 

>> Message bodies are JSON objects, that adhere to a schema. Message schemas

> live in their own Python package, so they can be installed on the producer

> and on the consumer.

> 

> Could we instead just send the message schemas together with the message

> content always?

I considered this early on, but it seemed to me it didn't solve all the

problems I wanted solved. Those problems are:

1. Make catching accidental schema changes as a publisher easy.

2. Make catching mis-behaving publishers on the consuming side easy.

3. Make changing the schema a painless process for publishers and

   consumers.

Doing this would solve #1, but #2 and #3 are still a problem. As a

consumer, I can validate the JSON in a message matches the JSON schema

in the same message, but what does that get me? It doesn't seem any

different (on the consumer side) than just parsing the JSON outright and

trying to access whatever deserialized object I get.

In the current proposal, consumers don't interact with the JSON at all,

but with a higher-level Python API that gives publishers flexibility

when altering their on-the-wire format.

> 

> I would like to be able to parse any message I receive without some

> additional packages installed. If I am about to start listening to a new

> message type, I don't want to spend time to be looking up what i should

> install to make it work. It should just work. Requiring to have some

> packages with schemas installed on consumer and having to maintain them by

> the producer does not seem that great idea. Mainly because one of the

> raising requirements for fedmsg was that it should be made a generic

> messaging framework easily usable outside of Fedora Infrastructure. We

> should make it easy for anyone outside to be able to listen and understand

> our messages so that they can react to them. Needing to have some python

> packages installed (how are they going to be distributed PyPI + fedora ?)

> seems to be just an unnecessary hassle. So can we send a schema with each

> message as documentation and validation of the message itself?

You can parse any message you receive without anything beyond a JSON

parsing library. You can do that now and you'll be able to do that after

the move. The problem with that is the JSON format might change. The

schema alone doesn't solve the problem of changing formats, it just

clearly documents what the message used to be and what it is now.

I'd love for this to just work and I'm up for any suggestions to make it

easier, but I do think we need to make sure any solution covers the

three problems stated above.

Finally, I do not want to create a generic messaging framework. I want

something small that makes a generic messaging framework very easy to

use for Fedora infrastructure specifically. I'm happy to help develop a

generic framework (like Pika) when necessary, but I don't want to be in

the business of authoring and maintaining a generic framework.

> 

> a) it will make our life easier

> 

> b) it will allow people outside of Fedora (that e.g. also don't tend to use

> python) to consume our messages easily

> 

> c) what if I am doing a ruby app, not python app, do I need then provide

> ruby schema as well as python schema? What if a consumer is a ruby app? We

> should only need to write a consumer and producer parts in different

> languages. The message schemes should not be bound to a particular

> language, otherwise we are just adding us more work when somebody wants to

> use the messaging system in another language than python.

I agree, and that's why I chose json-schema. A different language just

needs to wrap the schema in accessor functions. An alternative (and

something I wanted to propose longer term after the AMQP->ZMQ

transition) is to use something like protocol buffers rather than JSON.

The advantage there is a simplified schema format, it generally pushes

into a pattern of backwards compatibility (thus reducing the need for

a higher level API), and it auto-generates an object wrapper in many

languages. You still need to potentially implement wrappers for access

if you change the schema in a way that isn't additive, though.

You may notice (and it's not an accident) that the recommended

implementation of a Message produces an API that is very similar to the

one produced by a Python object generated by protocol buffers. This

makes it possible to quietly change to protocol buffers without breaking

consumers, assuming they're not digging into the JSON. I'm not saying

we'll definitely do that, but it is still on the table and a transition

_should_ be easy.

The big problem is that right now the majority of messages are not

formatted in a way that makes sense and really need to be changed to be

simple, flat structures that contain the information services need and