Signed-off-by: Tony Ibbs <tibs@xxxxxxxxxxxxxx>
---
Documentation/Kbus.txt | 1222 ++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 1222 insertions(+), 0 deletions(-)
create mode 100644 Documentation/Kbus.txt
diff --git a/Documentation/Kbus.txt b/Documentation/Kbus.txt
new file mode 100644
index 0000000..7cf723fd6
--- /dev/null
+++ b/Documentation/Kbus.txt
@@ -0,0 +1,1222 @@
+=============================================
+KBUS -- Lightweight kernel-mediated messaging
+=============================================
+
+Summary
+=======
+KBUS provides lightweight kernel-mediated messaging for Linux.
+
+* "lightweight" means that there is no intent to provide complex or
+ sophisticated mechanisms - if you need something more, consider DBUS or
+ other alternatives.
+
+* "kernel-mediated" means that the actual business of message passing and
+ message synchronisation is handled by a kernel module.
+
+* "for Linux" means what it says, since the Linux kernel is required.
+
+Initial use is expected to be in embedded systems.
+
+There is (at least initially) no intent to aim for a "fast" system - this is
+not aimed at real-time systems.
+
+Although the implementation is kernel-mediated, there is a mechanism
+("Limpets") for commnicating KBUS messages between buses and/or systems.
+
+Intentions
+==========
+KBUS is intended:
+
+* To be simple to use and simple to understand.
+* To have a small codebase, written in C.
+* To provide predictable message delivery.
+* To give deterministic message ordering.
+* To guarantee a reply to every request.
+
+It needs to be simple to use and understand because the expected users are
+typically busy with other matters, and do not have time to spend learning
+a complex messaging system.
+
+It needs to have a small codebase, written in C, because embedded systems
+often lack resources, and may not have enough space for C++ libraries, or
+messaging systems supporting more complex protocol stacks.
+
+Our own experience on embedded systems of various sizes indicates that
+the last three points are especially important.
+
+Predictable message delivery means the user can know whether they can tell in
+what circumstances messages will or will not be received.
+
+Deterministic message ordering means that all recipients of a given set of
+messages will receive them in the same order as all other recpients (and this
+will be the order in which the messages were sent). This is important when
+several part of (for instance) an audio/video stack are interoperating.
+
+Guaranteeing that a request will always result in a reply means that the user
+will be told if the intended replier has (for instance) crashed. This again
+allows for simpler use of the system.
+
+The basics
+==========
+Python and C
+------------
+Although the KBUS kernel module is written in C, the module tests are written
+in Python, and there is a Python module providing useful interfaces, which is
+expected to be the normal way of using KBUS from Python.
+
+There is also a C library (libkbus) which provides a similar level of
+abstraction, so that C programmers can use KBUS without having to handle the
+low level details of sockets and message datastructures. Note that the C
+programer using KBUS does need to have some awareness of how KBUS messages
+work in order to get memory management right.
+
+Messages
+========
+Message names
+-------------
+All messages have names - for instance "$.Sensors.Kitchen".
+
+All message names start with "$.", followed by one or more alphanumeric words
+separated by dots. There are two wildcard characters, "*" and "%", which can
+be the last word of a name.
+
+Thus (in some notation or other)::
+
+ name := '$.' [ word '.' ]+ ( word | '*' | '%' )
+ word := alphanumerics
+
+Case is significant. There is probably a limit on the maximum size of a
+subname, and also on the maximum length of a message name.
+
+Names form a name hierarchy or tree - so "$.Sensors" might have children
+"$.Sensors.Kitchen" and "$.Sensors.Bedroom".
+
+If the last word of a name is "*", then this is a wildcard name that also
+includes all the child names at that level and below -- i.e., all the names
+that start with the name up to the "*". So "$.Sensors.*" includes
+"$.Sensors.Kitchen", "$.Sensors.Bedroom", "$.Sensors.Kitchen.FireAlarm",
+"$.Sensors.Kitchen.Toaster", "$.Sensors.Bedroom.FireAlarm", and so on.
+
+If the last word of a name is "%", then this is a wildcard name that also
+includes all the child names at that level -- i.e., all the names obtained by
+replacing the "%" by another word. So "$.Sensors.%" includes
+"$.Sensors.Kitchen" and "$.Sensors.Bedroom", but not
+"$.Sensors.Kitchen.Toaster".
+
+Message ids
+-----------
+Every message is expected to have a unique id.
+
+A message id is made up of two parts, a network id and a serial number.
+
+The network id is used to carry useful information when a message is
+transferred from one KBUS system to another (for instance, over a bridge). By
+default (for local messages) it is 0.
+
+A serial number is used to identify the particular message within a network.
+
+If a message is sent via KBUS with a network id of 0, then KBUS itself will
+assign a new message id to the message, with the network id (still) 0, and
+with the serial number one more than the last serial number assigned. Thus for
+local messages, message ids ascend, and their order is deterministic.
+
+If a message is sent via KBUS with a non-zero network id, then KBUS does not
+touch its message id.
+
+Network ids are represented textually as ``{n,s}``, where ``n`` is the
+network id and ``s`` is the serial number.
+
+ Message id {0,0} is reserved for use as an invalid message id. Both
+ network id and serial number are unsigned 32-bit integers. Note that this
+ means that local serial numbers will eventually wrap.
+
+Message content
+---------------
+Messages are made of the following parts:
+
+:start and end guards:
+
+ These are unsigned 32-bit words. 'start_guard' is notionally "Kbus",
+ and 'end_guard' (the 32 bit word after the rest of the message) is
+ notionally "subK". Obviously that depends on how one looks at the 32-bit
+ word. Every message shall start with a start guard and end with an end
+ guard (but see `Message implementation`_ for details).
+
+ These provide some help in checking that a message is well formed, and in
+ particular the end guard helps to check for broken length fields.
+
+ If the message layout changes in an incompatible manner (this has happened
+ once, and is strongly discouraged), then the start and end guards change.
+
+Unset
+~~~~~
+Unset values are 0, or have zero length (as appropriate).
+
+It is not possible for a message name to be unset.
+
+The message header
+~~~~~~~~~~~~~~~~~~
+:message id: identifies this particular message. This is made up of a network
+ id and a serial number, and is discussed in `Message ids`_.
+
+ When replying to a message, copy this value into the 'In reply to' field.
+
+:in_reply_to: is the message id of the message that this is a reply to.
+
+ This shall be set to 0 unless this message *is* a reply to a previous
+ message. In other words, if this value is non-0, then the message *is* a
+ reply.
+
+:to: is the Ksock id identifying who the message is to be sent to.
+
+ When writing a new message, this should normally be set to 0, meaning
+ "anyone listening" (but see below if "state" is being maintained).
+
+ When replying to a message, it shall be set to the 'from' value of the
+ orginal message.
+
+ When constructing a request message (a message wanting a reply), then it can
+ be set to a specific replier's Ksock id. When such a message is sent, if the
+ replier bound (at that time) does not have that specific Ksock id, then the
+ send will fail.
+
+:from: indicates the Ksock id of the message's sender.
+
+ When writing a new message, set this to 0, since KBUS will set it.
+
+ When reading a message, this will have been set by KBUS.
+
+:orig_from: this indicates the original sender of a message, when being
+ transported via Limpet. This will be documented in more detail in the future.
+
+:final_to: this indicates the final target of a message, when being
+ transported via Limpet. This will be documented in more detail in the future.
+
+:extra: this is a zero field, for future expansion. KBUS will always set this
+ field to zero.
+
+:flags: indicates extra information about the message. See `Message Flags`_
+ for detailed information.
+
+ When writing a message, typical uses include:
+
+ * the message is URGENT
+ * a reply is wanted
+
+ When reading a message, typical uses include:
+
+ * the message is URGENT
+ * a reply is wanted
+ * a reply is wanted from the specific reader
+
+ The top 16 bits of the flags field is reserved for use by the user - KBUS
+ will not touch it.
+
+:name_length: is the length of the message name in bytes. This will always be
+ non-zero, as a message name must always be given.
+
+:data_length: is the length of the message data in bytes. It may be zero
+ if there is no data associated with this message.
+
+:name: identifies the message. It must be terminated with a
+ zero byte (as is normal for C - in the Python binding a normal Python string
+ can be used, and the this will be done for you). Byte ordering is according
+ to that of the platform.
+
+ In an "entire" message (see `Message implementation`_ below) the name shall
+ be padded out to a multiple of 4 bytes. Neither the terminating zero byte
+ nor the padding are included in the name length. Padding should be with
+ zero bytes.
+
+:data: is optional. KBUS does not touch the content of the
+ data, but just copies it. Byte ordering is according to that of the
+ platform.
+
+ In an "entire" message (see `Message implementation`_ below) the data shall,
+ if present, be padded out to a multiple of 4 bytes. This padding is not
+ included in the data length, and the padding bytes may be whatever byte
+ values are convenient to the user. KBUS does not guarantee to copy the exact
+ given padding bytes (in fact, current implementations just ignore them).
+
+Message implementation
+~~~~~~~~~~~~~~~~~~~~~~
+There are two ways in which a message may be constructed, "pointy" and
+"entire".
+See the ``kbus_defns.h`` header file for details.
+
+.. note:: The Python binding hides most of the detail of the message
+ implementation from the user, so if you are using Python you may be able to
+ skip this section.
+
+In a "pointy" message, the ``name`` and ``data`` fields in the message header
+are C pointers to the actual name and data. If there is no data, then the
+``data`` field is NULL. This is probably the simplest form of message for a C
+programmer to create. This might be represented as::
+
+ start_guard: 'Kbus'
+ id: (0,0)
+ in_reply_to: (0,0)
+ to: 0
+ from: 0
+ name_len: 6
+ data_len: 0
+ name: ---------------------------> "$.Fred"
+ data: NULL
+ end_guard: 'subK'
+
+or (with data)::
+
+ start_guard: 'Kbus'
+ id: (0,0)
+ in_reply_to: (0,0)
+ to: 0
+ from: 0
+ name_len: 6
+ data_len: 7
+ name: ---------------------------> "$.Fred"
+ data: ---------------------------> "abc1234"
+ end_guard: 'subK'
+
+.. warning:: When writing a "pointy" message in C, be very careful not to
+ free the name and data between the ``write`` and the SEND, as it is
+ only when the message is sent that KBUS actually follows the ``name`` and
+ ``data`` pointers.
+
+ *After* the SEND, KBUS will have taken its own copies of the name and
+ (any) data.
+
+In an "entire" message, both ``name`` and ``data`` fields are required to be
+NULL. The message header is followed by the message name (padded as described
+above), any message data (also padded), and another end guard. This might be
+represented as::
+
+ start_guard: 'Kbus'
+ id: (0,0)
+ in_reply_to: (0,0)
+ to: 0
+ from: 0
+ name_len: 6
+ data_len: 0
+ name: NULL
+ data: NULL
+ end_guard: 'subK'
+ name_data: '$.Fred\x0\x0'
+ end_guard: 'subK'
+
+or (again with data)::
+
+ start_guard: 'Kbus'
+ id: (0,0)
+ in_reply_to: (0,0)
+ to: 0
+ from: 0
+ name_len: 6
+ data_len: 7
+ name: NULL
+ data: NULL
+ end_guard: 'subK'
+ name_data: '$.Fred\x0\x0'
+ data_data: 'abc1234\x0'
+ end_guard: 'subK'
+
+Note that in these examples:
+
+1. The message name is padded out to 6 bytes of name, plus one of terminating
+ zero byte, plus another zero byte to make 8, but the message's ``name_len``
+ is still 6.
+2. When there is no data, there is no "data data" after the name data.
+3. When there is data, the data is presented after the name, and is padded out
+ to a multiple of 4 bytes (but without the necessity for a terminating zero
+ byte, so it is possible to have no pad bytes if the data length is already
+ a multiple of 4). Again, the ``data_len`` always reflects the "real" data
+ length.
+4. Although the data shown is presented as ASCII strings for these examples,
+ it really is just bytes, with no assumption of its content/meaning.
+
+When writing/sending messages, either form may be used (again, the "pointy"
+form may be simpler for C programmers).
+
+When reading messages, however, the "entire" form is always returned - this
+removes questions about needing to free multiple returned datastructures (for
+instance, what to do if the user were to ask for the NEXTMSG, read a few
+bytes, and then DISCARD the rest).
+
+Limits
+~~~~~~
+Message names may not be shorter than 3 characters (since they must be at
+least "$." plus another character). An arbitrary limit is also placed on the
+maximum message length - this is currently 1000 characters, but may be
+reviewed in the future.
+
+Message data may, of course, be of zero length.
+
+When reading a message, an "entire" message is always returned.
+
+ .. note:: When using C to work with KBUS messages, it is generally
+ ill-advised to reference the message name and data "directly"::
+
+ char *name = msg->name;
+ uint8_t *data = msg->data;
+
+ since this will work for "pointy" messages, but not for "entire"
+ messages (where the ``name`` field will be NULL). Instead, it
+ is always better to do::
+
+ char *name = kbus_msg_name_ptr(msg);
+ uint8_t *data = kbus_msg_data_ptr(msg);
+
+ regardless of the message type.
+
+Message flags
+-------------
+KBUS reserves the bottom 16 bits of the flags word for predefined purposes
+(although not all of those bits are yet used), and guarantees not to touch the
+top 16 bits, which are available for use by the programmer as a particular
+application may wish.
+
+The WANT_A_REPLY bit is set by the sender to indicate that a
+reply is wanted. This makes the message into a request.
+
+ Note that setting the WANT_A_REPLY bit (i.e., a request) and
+ setting 'in_reply_to' (i.e., a reply) is bound to lead to
+ confusion, and the results are undefined (i.e., don't do it).
+
+The WANT_YOU_TO_REPLY bit is set by KBUS on a particular message
+to indicate that the particular recipient is responsible for replying
+to (this instance of the) message. Otherwise, KBUS clears it.
+
+The SYNTHETIC bit is set by KBUS when it generates a Status message, for
+instance when a replier has gone away and will therefore not be sending a
+reply to a request that has already been queued.
+
+ Note that KBUS does not check that a sender has not set this
+ flag on a message, but doing so may lead to confusion.
+
+The URGENT bit is set by the sender if this message is to be
+treated as urgent - i.e., it should be added to the *front* of the
+recipient's message queue, not the back.
+
+Send flags
+~~~~~~~~~~
+There are two "send" flags, ALL_OR_WAIT and ALL_OR_FAIL.
+Either one may be set, or both may be unset.
+
+ If both are set, the message will be rejected as invalid.
+
+ Both flags are ignored in reply messages (i.e., messages with the
+ 'in_reply_to' field set).
+
+If a message has ALL_OR_FAIL set, then a SEND will only succeed if the message
+could be added to all the (intended) recipient's message queues. Otherwise,
+SEND returns -EBUSY.
+
+If a message has ALL_OR_WAIT set, then a SEND will only succeed if the message
+could be added to all the (intended) recipient's message queues. Otherwise
+SEND returns -EAGAIN. In this case, the message is still being sent, and the
+caller should either call DISCARD (to drop it), or else use poll/select to
+wait for the send to finish. It will not be possible to call "write" until the
+send has completed or been discarded.
+
+These are primarily intended for use in debugging systems. In particular, note
+that the mechanisms dealing with ALL_OR_WAIT internally are unlikely to be
+very efficient.
+
+.. note:: The send flags will be less effective when messages are being
+ mediated via Limpets, as remote systems are involved.
+
+Things KBUS changes in a message
+--------------------------------
+In general, KBUS leaves the content of a message alone - mostly so that an
+individual KBUS module can "pass through" messages from another domain.
+However, it does change:
+
+- the message id's serial number (but only if its network id is unset)
+- the 'from' id (to indicate the Ksock this message was sent from)
+- the WANT_YOU_TO_REPLY bit in the flags (set or cleared as appropriate)
+- the SYNTHETIC bit, which will always be unset in a message sent by a
+ Sender
+
+KBUS will always set the 'extra' field to zero.
+
+Limpets will change:
+
+- the network id in any field that has one.
+- the 'orig_from' and 'final_to' fields (which in general should only be
+ manipulated by Limpets).
+
+Types of message
+================
+There are four basic message types:
+
+* Announcement -- a message aimed at any listeners, expecting no reply
+* Request -- a message aimed at a replier, who is expected to reply
+* Reply -- a reply to a request
+* Status -- a message generated by KBUS
+
+The Python interface provides a Message base class, and subclasses thereof for
+each of the "user" message types (but not currently for Status).
+
+Announcements
+-------------
+An announcement is the "plain" message type. It is a message that is being
+sent for all bound listeners to "hear".
+
+When creating a new announcement message, it has:
+
+ :message id: see `Message ids`_
+ :in reply to: unset (it's not a reply)
+ :to: unset (all announcements are broadcast to any listeners)
+ :from: unset (KBUS will set it)
+ :flags: typically unset, see `Message flags`_
+ :message name: as appropriate
+ :message data: as appropriate
+
+The Python interface provides an ``Announcement`` class to help in creating an
+announcement message.
+
+Request message
+---------------
+A request message is a message that wants a reply.
+
+Since only one Ksock may bind as a replier for a given message name, a
+request message wants a reply from a single Ksock. By default, this is
+whichever Ksock has bound to the message name at the moment of sending, but
+see `Stateful transactions`_.
+
+When creating a new request message, it has:
+
+ :message id: see `Message ids`_
+ :in reply to: unset (it's not a reply)
+ :to: either unset, or a specific Ksock id if the request
+ should fail if that Ksock is (no longer) the replier
+ for this message name
+ :from: unset (KBUS will set it)
+ :flags: the "needs a reply" flag should be set.
+ KBUS will set the "you need to reply" flag in the
+ copy of the message delivered to its replier.
+ :message name: as appropriate
+ :message data: as appropriate
+
+When receiving a request message, the WANT_YOU_TO_REPLY flag will be set if it
+is this recipient's responsibility to reply.
+
+The Python interface provides a ``Request`` class to help in creating a
+request message.
+
+When a request message is sent, it is an error if there is no replier bound to
+that message name.
+
+The message will, as normal, be delivered to all listeners, and will have the
+"needs a reply" flag set wherever it is received. However, only the copy of
+the message received by the replier will be marked with the WANT_YOU_TO_REPLY
+flag.
+
+ So, if a particular file descriptor is bound as listener and replier
+ for '$.Fred', it will receive two copies of the original message (one
+ marked as needing reply from that file descriptor). However, when the
+ reply is sent, only the "plain" listener will receive a copy of the reply
+ message.
+
+Reply message
+-------------
+A reply message is the expected response after reading a request message.
+
+A reply message is distinguished by having a non-zero 'in reply to' value.
+
+Each reply message is in response to a specific request, as indicated by the
+'in reply to' field in the message.
+
+The replier is helped to remember that it needs to reply to a request, because
+the request has the WANT_YOU_TO_REPLY flag set.
+
+When a reply is sent, all listeners for that message name will receive it.
+However, the original replier will not.
+
+When creating a new reply message, it has:
+
+ :message id: see `Message ids`_
+ :in reply to: the request message's 'message id'
+ :to: the request message's 'from' id
+ :from: unset (KBUS will set it)
+ :flags: typically unset, see `Message flags`_
+ :message name: the request message's 'message name'
+ :message data: as appropriate
+
+The Python interface provides a ``Reply`` class to help in creating a reply
+message, but more usefully there is also a ``reply_to`` function that creates
+a Reply Message from the original Request.
+
+Status message
+--------------
+KBUS generates Status messages (also sometimes referred to as "synthetic"
+messages) when a request message has been successfully sent, but the replier
+is unable to reply (for instance, because it has closed its Ksock). KBUS thus
+uses a Status message to provide the "reply" that it guarantees the sender
+will get.
+
+As you might expect, a KBUS status message is thus (technically) a reply
+message.
+
+A status message looks like:
+
+ :message id: as normal
+ :in reply to: the 'message id' of the message whose sending or
+ processing caused this message.
+ :to: the Ksock id of the recipient of the message
+ :from: the Ksock id of the sender of the message - this will
+ be 0 if the sender is KBUS itself (which is assumed for
+ most exceptions)
+ :flags: typically unset, see `Message flags`_
+ :message name: for KBUS exceptions, a message name in '$.KBUS.*'
+ :message data: for KBUS exceptions, normally absent
+
+KBUS status messages always have '$.KBUS.<something>' names (this may be a
+multi-level <something>), and are always in response to a previous message, so
+always have an 'in reply to'.
+
+Requests and Replies
+--------------------
+KBUS guarantees that each Request will (eventually) be matched by a consequent
+Reply (or Status [1]_) message, and only one such.
+
+The "normal" case is when the replier reads the request, and sends its own
+reply back.
+
+If a Request message has been successfully SENT, there are the following other
+cases to consider:
+
+1. The replier unbinds from that message name before reading the request
+ message from its queue. In this case, KBUS removes the message from the
+ repliers queue, and issues a "$.KBUS.Replier.Unbound" message.
+
+2. The replier closes itself (close the Ksock), but has not yet read the
+ message. In this case, KBUS issues a "$.KBUS.Replier.GoneAway" message.
+
+3. The replier closes itself (closes the Ksock), has read the message, but has
+ not yet (and now cannot) replied to it. In this case, KBUS issues a
+ "$.KBUS.Replier.Ignored" message.
+
+4. SEND did not complete, and the replier closes itself before the message can
+ be added to its message queue (by the POLL mechanism). In this case, KBUS
+ issues a "$.KBUS.Replier.Disappeared" message.
+
+5. SEND did not complete, and an error occurs when the POLL mechanims tries to
+ send the message. In this case, KBUS issues a "$.KBUS.ErrorSending"
+ message.
+
+In all these cases, the 'in_reply_to' field is set to the original request's
+message id. In the first three cases, the 'from' field will be set to the
+Ksock id of the (originally intended) replier. In the last two cases, that
+information is not available, and a 'from' of 0 (indicating KBUS itself) is
+used.
+
+.. [1] Remember that a Status message is essentially a specialisation of a
+ Reply message.
+
+.. note:: Limpets introduce some extra messages, which will be documented when
+ the proper Limpet documentation is written.
+
+KBUS end points - Ksocks
+========================
+The KBUS devices
+----------------
+Message interactions happen via the KBUS devices. Installing the KBUS kernel
+module always creates ``/dev/kbus0``, it may also create ``/dev/kbus1``, and
+so on.
+
+ The number of devices to create is indicated by an argument at module
+ installation, for instance::
+
+ # insmod kbus.ko num_kbus_devices=10
+
+Messages are sent by writing to a KBUS device, and received by reading from
+the same device. A variety of useful ioctls are also provided. Each KBUS
+device is independent - messages cannot be sent from ``/dev/kbus0`` to
+``/dev/kbus1``, since there is no shared information.
+
+Ksocks
+------
+Specifically, messages are written to and read from KBUS device file
+descriptors. Each such is termed a *Ksock* - this is a simpler term than "file
+descriptor", and has some resonance with "socket".
+
+Each Ksock may be any (one or more) of:
+
+* a Sender (opening the device for read/write)
+* a Listener (only needing to open the device for read)
+* a Replier (opening the device for read/write)
+
+Every Ksock has an id. This is a 32-bit unsigned number assigned by KBUS when
+the device is opened. The value 0 is reserved for KBUS itself.
+
+ The terms "listener id", "sender id", "replier id", etc., thus all refer
+ to a Ksock id, depending on what it is being used for.
+
+Senders
+-------
+Message senders are called "senders". A sender should open a Ksock for read
+and write, as it may need to read replies and error/status messages.
+
+A message is sent by:
+
+1. Writing the message to the Ksock (using the standard ``write`` function)
+2. Calling the SEND ioctl on the Ksock, to actually send the message. This
+ returns (via its arguments) the message id of the message sent. It also
+ returns status information about the send
+
+ The status information is to be documented.
+
+The DISCARD ioctl can be used to "throw away" a partially written message,
+before SEND has been called on it.
+
+If there are no listeners (of any type) bound to that message name, then the
+message will be ignored.
+
+If the message is flagged as needing a reply, and there are no repliers bound
+to that message name, then an error message will be sent to the sender, by
+KBUS.
+
+It is not possible to send a message with a wildcard message name.
+
+ As a restriction this makes the life of the implementor and documentor
+ easier. I believe it would also be confusing if provided.
+
+The sender does not need to bind to any message names in order to receive
+error and status messages from KBUS.
+
+When a sender sends a Request, an internal note is made that it expects a
+corresponding Reply (or possible a Status message from KBUS if the Replier
+goes away or unbinds from that message name, before replying). A place for
+that Reply is reserved in the sender's message queue. If the message queue
+fills up (either with messages waiting to be read, or with reserved slots for
+Replies), then the sender will not be able to send another Request until there
+is room on the message queue again.
+
+ Hopefully, this can be resolved by the sender reading a message off its
+ queue. However, if there are no messages to be read, and the queue is all
+ reserved for replies, the only solution is for the sender to wait for a
+ replier to send it something that it can then read.
+
+.. note:: What order do we describe things in? Don't forget:
+
+ If the message being sent is a request, then the replier bound to that
+ message name will (presumably) write a reply to the request. Thus the normal
+ sequence for a request is likely to be:
+
+ 1. write the request message
+ 2. read the reply
+
+ The sender does *not* need to bind to anything in order to receive a reply to
+ a request it has sent.
+
+ Of course, if a sender binds to listen to the name it uses for its
+ request, then it will get a copy of the request as sent, and it will
+ also get (an extra) copy of the reply. But see `Receiving messages once
+ only`_.
+
+Listeners
+---------
+Message recipients are called "listeners".
+
+Listeners indicate that they want to receive particular messages, by using the
+BIND ioctl on a Ksock to specify the name of the message that is to be
+listened for. If the binding is to a wildcarded message name, then the
+listener will receive all messages with names that match the wildcard.
+
+An ordinary listener will receive all messages with that name (sent to the
+relevant Ksock). A listener may make more than one binding on the same Ksock
+(indeed, it is allowed to bind to the same name more than once).
+
+Messages are received by:
+
+1. Using the NEXTMSG ioctl to request the next message (this also returns the
+ messages length in bytes)
+2. Calling the standard ``read`` function to read the message data.
+
+If NEXTMSG is called again, the next message will be readied for reading,
+whether the previous message has been read (or partially read) or not.
+
+If a listener no longer wants to receive a particular message name, then they
+can unbind from it, using the UNBIND ioctl. The message name and flags used in
+an UNBIND must match those in the corresponding BIND. Any messages in the
+listener's message queue which match that unbinding will be removed from the
+queue (i.e., the listener will not actually receive them). This does *not*
+affect the message currently being read.
+
+ Note that this has implication for binding and unbinding wildcards,
+ which must also match.
+
+Closing the Ksock also unbinds all the message bindings made on it.
+It does not affect message bindings made on other Ksocks.
+
+Repliers
+--------
+Repliers are a special sort of listener.
+
+For each message name, there may be a single "replier". A replier binds to a
+message name in the same way as any other listener, but sets the "replier"
+flag. If someone else has already bound to the same Ksock as a replier for
+that message name, the request will fail.
+
+Repliers only receive Requests (messages that are marked as wanting a reply).
+
+A replier may (should? must?) reply to the request - this is done by sending
+a Reply message through the Ksock from which the Request was read.
+
+It is perfectly legitimate to bind to a message as both replier and listener,
+in which case two copies of the message will be read, once as replier, and
+once as (just) listener (but see `Receiving messages once only`_).
+
+
+When a request message is read by the appropriate replier, KBUS will mark
+*that particular message* with the "you must reply" flag. This will not be set
+on copies of that message read by any (non-replier) listeners.
+
+ So, in the case where a Ksock is bound as replier and listener for the
+ same message name, only one of the two copies of the message received will
+ be marked as "you must reply".
+
+If a replier binds to a wildcarded message name, then they are the *default*
+replier for any message names satisfying that wildcard. If another replier
+binds to a more specific message name (matching that wildcard),
+then the specific message name binding "wins" - the wildcard replier will no
+longer receive that message name.
+
+ In particular '$.Fred.Jim' is more specific than '$.Fred.%' which in turn
+ is more specific than '$.Fred.*'
+
+This means that if a wildcard replier wants to guarantee to see all the
+messages matching their wildcard, they also need to bind as a listener for the
+same wildcarded name.
+
+For example:
+
+ Assume message names are of the form '$.Sensors.<Room>' or
+ '$.Sensors.<Room>.<Measurement>'.
+
+ Replier 1 binds to '$.Sensors.*'. They will be the default replier for
+ all sensor requests.
+
+ Replier 2 binds to '$.Sensors.%'. They will take over as the default
+ replier for any room specific requests.
+
+ Replier 3 binds to '$.Sensors.Kitchen.Temperature'. They will take over as
+ the replier for the kitchen temperature.
+
+ So:
+
+ - A message named '$.Sensors.Kitchen.Temperature' will go to replier 3.
+ - A message named '$.Sensors.Kitchen' or '$.Sensors.LivingRoom' will go to
+ replier 2.
+ - A message named '$.Sensors.LivingRoom.Temperature' will go to replier 1.
+
+When a Replier is closed (technically, when its ``release`` function is
+called by the kernel) KBUS traverses its outstanding message queue, and for
+each Request that has not been answered, generates a Status message saying
+that the Replier has "GoneAway".
+
+Similarly, if a Replier unbinds from replying to a mesage, KBUS traverses its
+outstanding message queue, and for each Request that has not been answered, it
+generates a Status message saying that it has "Unbound" from being a replier
+for that message name. It also forgets the message, which it is now not going
+to reply to.
+
+Lastly, when a Replier is closed, if it has read any Requests (technically,
+called NEXTMSG to pop them from the message queue), but not actually replied
+to them, then KBUS will send an "Ignored" Status message for each such
+Request.
+
+More information
+================
+Stateful transactions
+---------------------
+It is possible to make stateful message transactions, by:
+
+1. sending a Request
+2. receiving the Reply, and noting the Ksock id of the replier
+3. sending another Request to that specific replier
+4. and so on
+
+Sending a request to a particular Ksock will fail if that Ksock is no longer
+bound as replier to the relevant message name. This allows a sender to
+guarantee that it is communicating with a particular instance of the replier
+for a message name.
+
+Queues filling up
+-----------------
+Messages are sent by a mechanism which:
+
+1. Checks the message is plausible (it has a plausible message name,
+ and the right sort of "shape")
+2. If the message is a Request, checks that the sender has room on its message
+ queue for the (eventual) Reply.
+3. Finds the Ksock ids of all the listeners and repliers bound to that
+ messages name
+4. Adds the message to the queue for each such listener/replier
+
+This can cause problems if one of the queues is already full (allowing
+infinite expansion of queues would also cause problems, of couse).
+
+If a *sender* attempts to send a Request, but does not have room on its
+message queue for the (corresponding) Reply, then the message will not be
+sent, and the send will fail. Note that the message id will not be set, and
+the blocking behaviours defined below do not occur.
+
+If a *replier* cannot receive a particular message, because its queue is full,
+then the message will not be sent, and the send will fail with an error. This
+does, however, set the message id (and thus the "last message id" on the
+sender).
+
+Moreover, a sender can indicate if it wants a message to be:
+
+1. Added to all the listener queues, regardless, in which case it will block
+ until that can be done (ALL_OR_WAIT, sender blocks)
+2. Added to all the listener queues, and fail if that can't be done
+ (ALL_OR_FAIL)
+3. Added to all the listener queues that have room (the default)
+
+See `Message flags`_ for more details.
+
+Urgent messages
+---------------
+Messages may be flagged urgent. In this case they will be added to the front
+of the destination message queue, rather than the end - in other words, they
+will be the next message to be "popped" by NEXTMSG.
+
+Note that this means that if two urgent messages are sent to the same target,
+and *then* a NEXTMSG/read occurs, the second urgent message will be popped and
+read first.
+
+Select, write/send and "next message", blocking
+-----------------------------------------------
+.. warning:: At the moment, ``read`` and ``write`` are always non-blocking.
+
+``read`` returns more of the currently selected message, or EOF if there is no
+more of that message to read (and thus also if there is no currently selected
+message). The NEXTMSG ioctl is used to select ("pop") the next message.
+
+``write`` writes to the end of the currently-being-written message. The
+DISCARD ioctl can be used to discard the data written so far, and the SEND
+ioctl to send the (presumably completed message). Whilst the message is being
+sent, it is not possible to use ``write``.
+
+Note that if SEND is used to send a Request, then KBUS ensures that there will
+always be either a Reply or a Status message in response to that request.
+
+Specifically, if:
+
+1. The Replier "goes away" (and its "release" function is called) before
+ reading the Request (specifically, before calling NEXTMSG to pop it from
+ the message queue)
+2. The Replier "goes away" (and its "release" function is called) before
+ replying to a Request that it has already read (i.e., used NEXTMSG to pop
+ from the message queue)
+3. The Replier unbinds from that Request message name before reading the
+ Request (with the same caveat on what that means)
+4. Select/poll attempts to send the Request, and discovers that the
+ Replier has disappeared since the initial SEND
+5. Select/poll attempts to send the Request, and some other error occurs
+
+then KBUS will "reply" with an appropriate Status message.
+
+--------------------------------------------------
+
+KBUS support its own particular variation on blocking of message sending.
+
+First of all, it supports use of "select" to determine if there are any
+messages waiting to be read. So, for instance (in Python)::
+
+ with Ksock(0,'rw') as sender:
+ with Ksock(0,'r') as listener:
+ (r,w,x) = select.select([listener],[],[],0)
+ assert r == []
+
+ listener.bind('$.Fred')
+ msg = Announcement('$.Fred','data')
+ sender.send_msg(msg)
+
+ (r,w,x) = select.select([listener],[],[],0)
+ assert r == [listener]
+
+This simply checks if there is a message in the Ksock's message list, waiting
+to be "popped" with NEXTMSG.
+
+Secondly, ``write``, SEND and DISCARD interact in what is hoped to be a
+sensible manner. Specifically:
+
+* When SEND (i.e., the SEND ioctl) is called, KBUS can either:
+
+ 1. Succeed in sending the message. The Ksock is now ready for ``write`` to
+ be called on it again.
+ 2. Failed in sending the message (possibly, if the message was a Request,
+ with EADDRNOTAVAIL, indicating that there is no Replier for that
+ Request). The Ksock is now ready for ``write`` to be called on it again.
+ 3. If the message was marked ALL_OR_WAIT, then it may fail with EAGAIN.
+ In this case, the Ksock is still in sending state, and an attempt to
+ call ``write`` will fail (with EALREADY). The caller can either use
+ DISCARD to discard the message, or use select/poll to wait for the
+ message to finish sending.
+
+Thus "select" for the write case checks whether it is allowed to call
+"write" - for instance::
+
+ with Ksock(0,'rw') as sender:
+ write_list = [sender]
+ with Ksock(0,'r') as listener1:
+ write_list= [sender,listener1]
+ read_list = [listener1]
+
+ (r,w,x) = select.select(read_list,write_list,[],0)
+ assert r == []
+ assert w == [sender]
+ assert x == []
+
+ with Ksock(0,'rw') as listener2:
+ write_list.append(listener2)
+ read_list.append(listener2)
+
+ (r,w,x) = select.select(read_list,write_list,[],0)
+ assert r == []
+ assert len(w) == 2
+ assert sender in w
+ assert listener2 in w
+ assert x == []
+
+Receiving messages once only
+----------------------------
+In normal usage (and by default), if a Ksock binds to a message name multiple
+times, it will receive multiple copies of a message. This can happen:
+
+* explicitly (the Ksock deliberately and explicitly binds to the same name
+ more than once, seeking this effect).
+* as a result of binding to a message name and a wildcard that includes the
+ same name, or two overlapping wildcards.
+* as a result of binding as Replier to a name, and also as Listener to the
+ same name (possibly via a wildcard). In this case, multiple copies will
+ only be received when a Request with that name is made.
+
+Several programmers have complained that the last case, in particular, is very
+inconvenient, and thus the "receive a message once only" facility has been
+added.
+
+Using the MSGONCEONLY IOCTL, it is possible to tell a Ksock that only one copy
+of a particular message should be received, even if multiple are "due". In the
+case of the Replier/Listener copies, it will always be the message to which
+the Replier should reply (the one with WANT_YOU_TO_REPLY set) that will be
+received.
+
+Please use this facility with care, and only if you really need it.
+
+IOCTLS
+------
+The KBUS ioctls are defined (with explanatory comments) in the kernel module
+header file (``kbus_defns.h``). They are:
+
+:RESET: Currently has no effect
+:BIND: Bind to a particular message name (possibly as replier).
+:UNBIND: Unbind from a binding - must match exactly.
+:KSOCKID: Determine the Ksock id of the Ksock used
+:REPLIER: Determine who is bound as replier to a particular message
+ name. This returns 0 or the Ksock id of the replier.
+:NEXTMSG: Pop the next message from the Ksock's message queue, ready
+ for reading (with ``read``), and return its length (in bytes).
+ If there is no next message, return a length of 0.
+ The length is always the length of an "entire" message (see
+ `Message implementation`_).
+:LENLEFT: Determine how many bytes of the message currently being read
+ are still to read.
+:SEND: Send the current outstanding message for this Ksock (i.e., the
+ bytes written to the Ksock since the last SEND or DISCARD).
+ Return the message id of the message, and maybe other status
+ information.
+:DISCARD: Discard (throw away) the current outstanding message for this
+ Ksock (i.e., any bytes written to the Ksock since the last
+ SEND or DISCARD).
+:LASTSENT: Determine the message id of the last message SENT on this
+ Ksock.
+:MAXMSGS: Set the maximum length of the (read) message queue for this
+ KSOCK, and return the actual length that is set. An attempt
+ to set the queue length to 0 will just return the current
+ queue length.
+:NUMMSGS: Determine how many messages are outstanding in this Ksock's
+ read queue.
+:UNREPLIEDTO: Determines how many Requests (marked "WANT_YOU_TO_REPLY")
+ this Ksock still needs to reply to. This is primarily a
+ development tool.
+:MSGONLYONCE: Determines whether only one copy of a message will be
+ received, even if the message name is bound to multiple times.
+ May also be used to query the current state.
+:VERBOSE: Determines whether verbose kernel messages should be output or
+ not. Affects the *device* (the entire Ksock).
+ May also be used to query the current state.
+:NEWDEVICE: Requests another KBUS device (``/dev/kbus/<n>``). The next
+ KBUS device number (up to a maximum of 255) will be allocated.
+ Returns the new device number.
+:REPORTREPLIERBINDS: Request synthetic messages announcing Replier BIND/UNBIND
+ events. These are messages named "$.KBUS.ReplierBindEvent",
+ and are the only predefined messages with data.
+ Both Python and C bindings provide a useful function to
+ extract the ``is_bind``, ``binder`` and ``name`` values from
+ the data.
+
+/proc/kbus/bindings
+-------------------
+``/proc/kbus/bindings`` is a debugging aid for reporting the listener id,
+exclusive flag and message name for each binding, for each kbus device.
+
+An example might be::
+
+ $ cat /proc/kbus/bindings
+ # <device> is bound to <Ksock-ID> in <process-PID> as <Replier|Listener> for <message-name>
+ 1: 1 22158 R $.Sensors.*
+ 1: 2 22158 R $.Sensors.Kitchen.Temperature
+ 1: 3 22158 L $.Sensors.*
+ 13: 4 22159 L $.Jim.*
+ 13: 1 22159 R $.Fred
+ 13: 1 22159 L $.Jim
+ 13: 14 23021 L $.Jim.*
+
+This describes two KBUS devices (``/dev/kbus1`` and ``/dev/kbus13``).
+
+The first has bindings on Ksock ids 1, 2 and 3, for the given message names. The
+"R" indicates a replier binding, the "L" indicates a listener (non-replier)
+binding.
+
+The second has bindings on Ksock ids 4, 1 and 14. The order of the bindings
+reported is *not* particularly significant.
+
+Note that there is no communication between the two devices, so Ksock id 1 on
+device 1 is not related to (and has no commonality with) Ksock id 1 on device
+13.
+
+/proc/kbus/stats
+----------------
+``/proc/kbus/stats`` is a debugging aid for reporting various statistics about
+the KBUS devices and the Ksocks open on them.
+
+An example might be::
+
+ $ cat /proc/kbus/stats
+ dev 0: next file 5 next msg 8 unsent unbindings 0
+ ksock 4 last msg 0:7 queue 1 of 100
+ read byte 0 of 0, wrote byte 52 (max 60), sending
+ outstanding requests 0 (size 16, max 0), unsent replies 0 (max 0)
+ ksock 3 last msg 0:5 queue 0 of 1
+ read byte 0 of 0, wrote byte 0 (max 0), not sending
+ outstanding requests 1 (size 16, max 0), unsent replies 0 (max 0)
+
+or::
+
+ $ cat /proc/kbus/stats
+ dev 0: next file 4 next msg 101 unsent unbindings 0
+ ksock 3 last msg 0:0 queue 100 of 100
+ read byte 0 of 0, wrote byte 0 (max 0), not sending
+ outstanding requests 0 (size 16, max 0), unsent replies 0 (max 0)
+ ksock 2 last msg 0:100 queue 0 of 100
+ read byte 0 of 0, wrote byte 0 (max 0), not sending
+ outstanding requests 100 (size 102, max 92), unsent replies 0 (max 0)
+
+
+Error numbers
+-------------
+The following error numbers get special use. In Python, they are all returned
+as values inside the IOError exception.
+
+ Since we're trying to fit into the normal Un*x convention that negative
+ values are error numbers, and since Un*x defines many of these for us,
+ it is natural to make use of the relevant definitions. However, this also
+ means that we are often using them in an unnatural sense. I've tried to
+ make the error numbers used bear at least a vague relationship to their
+ (mis)use in KBUS.
+
+:EADDRINUSE: On attempting to bind a message name as replier: There is
+ already a replier bound for this message
+:EADDRNOTAVAIL: On attempting to send a Request message: There is no replier
+ bound for this message's name.
+
+ On attempting to send a Reply message: The sender of the
+ original request (i.e., the Ksock mentioned as the ``to``
+ in the Reply) is no longer connected.
+:EALREADY: On attempting to write to a Ksock, when a previous send has
+ returned EAGAIN. Either DISCARD the message, or use
+ select/poll to wait for the send to complete, and write to be
+ allowed.
+:EBADMSG: On attempting to bind, unbind or send a message: The message
+ name is not valid. On sending, this can also be because the
+ message name is a wildcard.
+:EBUSY: On attempting to send, then:
+
+ 1. For a request, the replier's message queue is full.
+ 2. For any message, with ALL_OR_FAIL set, one of the
+ targetted listener/replier queues was full.
+
+:ECONNREFUSED: On attempting to send a Reply, the intended recipient (the
+ notional original sender of the Request) is not expecting
+ a Reply with that message id in its 'in_reply_to'. Or, in
+ other words, this appears to be an attempt to reply to the
+ wrong message id or the wrong Ksock.
+:EINVAL: Something went wrong (generic error).
+:EMSGSIZE: On attempting to write a message: Data was written after
+ the end of the message (i.e., after the final end guard
+ of the message).
+:ENAMETOOLONG: On attempting to bind, unbind or send a message: The message
+ name is too long.
+:ENOENT: On attempting to open a Ksock: There is no such device
+ (normally because one has tried to open, for instance,
+ '/dev/kbus9' when there are only 3 KBUS devices).
+:ENOLCK: On attempting to send a Request, when there is not enough room
+ in the sender's message queue to guarantee that it can
+ receive a reply for every Request already sent, *plus* this
+ one. If there are oustanding messages in the sender's message
+ queue, then the solution is to read some of them. Otherwise,
+ the sender will have to wait until one of the Repliers
+ replies to a previous Request (or goes away and KBUS replies
+ for it).
+
+ When this error is received, the send has failed (just as if
+ the message was invalid). The sender is not left in "sending"
+ state, nor has the message been assigned a message id.
+
+ Note that this is *not* EAGAIN, since we do not want to block
+ the sender (in the SEND) if it is up to the sender to perform
+ a read to sort things out.
+
+:ENOMSG: On attempting to send, when there is no message waiting to be
+ sent (either because there has been no write since the last
+ send, or because the message being written has been
+ discarded).
+:EPIPE: On attempting to send 'to' a specific replier, the replier
+ with that id is no longer bound to the given message's name.
+
+:EFAULT: Memory allocation, copy from user space, or other such failed. This
+ is normally very bad, it should not happen, UNLESS it is the result
+ of calling an ioctl, when it indicates that the ioctl argument
+ cannot be accessed.
+
+:ENOMEM: Memory allocation failed (return NULL). This is normally very bad,
+ it should not happen.
+
+:EAGAIN: On attempting to send, the message being sent had ALL_OR_WAIT set,
+ and one of the targetted listener/replier queues was full.
+
+ On attempting to unbind when Replier Bind Events have been
+ requested, one or more of the KSocks bound to receive
+ "$.KBUS.ReplierBindEvent" messages has a full message queue,
+ and thus cannot receive the unbind event. The unbind has not been
+ done.
+
+In the ``utils`` directory of the KBUS sources, there is a script called
+``errno.py`` which takes an ``errno`` integer or name and prints out both the
+"normal" meaning of that error number, and also (if there is one) the KBUS use
+of it. For instance::
+
+ $ errno.py 1
+ Error 1 (0x1) is EPERM: Operation not permitted
+ $
+ $ errno.py EPIPE
+ EPIPE is error 32 (0x20): Broken pipe
+
+ KBUS:
+ On attempting to send 'to' a specific replier, the replier with that id
+ is no longer bound to the given message's name.
+
--
1.7.4.1
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
