The patch titled
Subject: netconsole: implement extended console support
has been added to the -mm tree. Its filename is
netconsole-implement-extended-console-support.patch
This patch should soon appear at
http://ozlabs.org/~akpm/mmots/broken-out/netconsole-implement-extended-console-support.patch
and later at
http://ozlabs.org/~akpm/mmotm/broken-out/netconsole-implement-extended-console-support.patch
Before you just go and hit "reply", please:
a) Consider who else should be cc'ed
b) Prefer to cc a suitable mailing list as well
c) Ideally: find the original patch on the mailing list and do a
reply-to-all to that, adding suitable additional cc's
*** Remember to use Documentation/SubmitChecklist when testing your code ***
The -mm tree is included into linux-next and is updated
there every 3-4 working days
------------------------------------------------------
From: Tejun Heo <tj@xxxxxxxxxx>
Subject: netconsole: implement extended console support
printk logbuf keeps various metadata and optional key=value dictionary for
structured messages, both of which are stripped when messages are handed
to regular console drivers.
It can be useful to have this metadata and dictionary available to
netconsole consumers. This obviously makes logging via netconsole more
complete and the sequence number in particular is useful in environments
where messages may be lost or reordered in transit - e.g. when netconsole
is used to collect messages in a large cluster where packets may have to
travel congested hops to reach the aggregator. The lost and reordered
messages can easily be identified and handled accordingly using the
sequence numbers.
printk recently added extended console support which can be selected by
setting CON_EXTENDED flag. From console driver side, not much changes.
The only difference is that the text passed to the write callback is
formatted the same way as /dev/kmsg.
This patch implements extended console support for netconsole which can be
enabled by either prepending "+" to a netconsole boot param entry or
echoing 1 to "extended" file in configfs. When enabled, netconsole
transmits extended log messages with headers identical to /dev/kmsg
output.
There's one complication due to message fragments. netconsole limits the
maximum message size to 1k and messages longer than that are split into
multiple fragments. As all extended console messages should carry
matching headers and be uniquely identifiable, each extended message
fragment carries full copy of the metadata and an extra header field to
identify the specific fragment. The optional header is of the form
"ncfrag=OFF/LEN" where OFF is the byte offset into the message body and
LEN is the total length.
To avoid unnecessarily making printk format extended messages, Extended
netconsole is registered with printk when the first extended netconsole is
configured.
Signed-off-by: Tejun Heo <tj@xxxxxxxxxx>
Cc: David Miller <davem@xxxxxxxxxxxxx>
Cc: Kay Sievers <kay@xxxxxxxx>
Cc: Petr Mladek <pmladek@xxxxxxx>
Cc: Tetsuo Handa <penguin-kernel@xxxxxxxxxxxxxxxxxxx>
Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
---
Documentation/networking/netconsole.txt | 35 +++++
drivers/net/netconsole.c | 141 +++++++++++++++++++++-
2 files changed, 173 insertions(+), 3 deletions(-)
diff -puN Documentation/networking/netconsole.txt~netconsole-implement-extended-console-support Documentation/networking/netconsole.txt
--- a/Documentation/networking/netconsole.txt~netconsole-implement-extended-console-support
+++ a/Documentation/networking/netconsole.txt
@@ -2,6 +2,7 @@
started by Ingo Molnar <mingo@xxxxxxxxxx>, 2001.09.17
2.6 port and netpoll api by Matt Mackall <mpm@xxxxxxxxxxx>, Sep 9 2003
IPv6 support by Cong Wang <xiyou.wangcong@xxxxxxxxx>, Jan 1 2013
+Extended console support by Tejun Heo <tj@xxxxxxxxxx>, May 1 2015
Please send bug reports to Matt Mackall <mpm@xxxxxxxxxxx>
Satyam Sharma <satyam.sharma@xxxxxxxxx>, and Cong Wang <xiyou.wangcong@xxxxxxxxx>
@@ -24,9 +25,10 @@ Sender and receiver configuration:
It takes a string configuration parameter "netconsole" in the
following format:
- netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
+ netconsole=[+][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
+ + if present, enable extended console support
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
@@ -107,6 +109,7 @@ To remove a target:
The interface exposes these parameters of a netconsole target to userspace:
enabled Is this target currently enabled? (read-write)
+ extended Extended mode enabled (read-write)
dev_name Local network interface name (read-write)
local_port Source UDP port to use (read-write)
remote_port Remote agent's UDP port (read-write)
@@ -132,6 +135,36 @@ You can also update the local interface
useful if you want to use interfaces that have newly come up (and may not
have existed when netconsole was loaded / initialized).
+Extended console:
+=================
+
+If '+' is prefixed to the configuration line or "extended" config file
+is set to 1, extended console support is enabled. An example boot
+param follows.
+
+ linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
+
+Log messages are transmitted with extended metadata header in the
+following format which is the same as /dev/kmsg.
+
+ <level>,<sequnum>,<timestamp>,<contflag>;<message text>
+
+Non printable characters in <message text> are escaped using "\xff"
+notation. If the message contains optional dictionary, verbatim
+newline is used as the delimeter.
+
+If a message doesn't fit in certain number of bytes (currently 1000),
+the message is split into multiple fragments by netconsole. These
+fragments are transmitted with "ncfrag" header field added.
+
+ ncfrag=<byte-offset>/<total-bytes>
+
+For example, assuming a lot smaller chunk size, a message "the first
+chunk, the 2nd chunk." may be split as follows.
+
+ 6,416,1758426,-,ncfrag=0/31;the first chunk,
+ 6,416,1758426,-,ncfrag=16/31; the 2nd chunk.
+
Miscellaneous notes:
====================
diff -puN drivers/net/netconsole.c~netconsole-implement-extended-console-support drivers/net/netconsole.c
--- a/drivers/net/netconsole.c~netconsole-implement-extended-console-support
+++ a/drivers/net/netconsole.c
@@ -79,6 +79,12 @@ static LIST_HEAD(target_list);
/* This needs to be a spinlock because write_msg() cannot sleep */
static DEFINE_SPINLOCK(target_list_lock);
+/*
+ * Console driver for extended netconsoles. Registered on the first use to
+ * avoid unnecessarily enabling ext message formatting.
+ */
+static struct console netconsole_ext;
+
/**
* struct netconsole_target - Represents a configured netconsole target.
* @list: Links this target into the target_list.
@@ -105,6 +111,7 @@ struct netconsole_target {
struct config_item item;
#endif
bool enabled;
+ bool extended;
struct netpoll np;
};
@@ -187,6 +194,11 @@ static struct netconsole_target *alloc_p
nt->np.remote_port = 6666;
eth_broadcast_addr(nt->np.remote_mac);
+ if (*target_config == '+') {
+ nt->extended = true;
+ target_config++;
+ }
+
/* Parse parameters and setup netpoll */
err = netpoll_parse_options(&nt->np, target_config);
if (err)
@@ -257,6 +269,11 @@ static ssize_t show_enabled(struct netco
return snprintf(buf, PAGE_SIZE, "%d\n", nt->enabled);
}
+static ssize_t show_extended(struct netconsole_target *nt, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%d\n", nt->extended);
+}
+
static ssize_t show_dev_name(struct netconsole_target *nt, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%s\n", nt->np.dev_name);
@@ -328,6 +345,11 @@ static ssize_t store_enabled(struct netc
}
if (enabled) { /* true */
+ if (nt->extended && !(netconsole_ext.flags & CON_ENABLED)) {
+ netconsole_ext.flags |= CON_ENABLED;
+ register_console(&netconsole_ext);
+ }
+
/*
* Skip netpoll_parse_options() -- all the attributes are
* already configured via configfs. Just print them out.
@@ -355,6 +377,30 @@ static ssize_t store_enabled(struct netc
return strnlen(buf, count);
}
+static ssize_t store_extended(struct netconsole_target *nt,
+ const char *buf,
+ size_t count)
+{
+ int extended;
+ int err;
+
+ if (nt->enabled) {
+ pr_err("target (%s) is enabled, disable to update parameters\n",
+ config_item_name(&nt->item));
+ return -EINVAL;
+ }
+
+ err = kstrtoint(buf, 10, &extended);
+ if (err < 0)
+ return err;
+ if (extended < 0 || extended > 1)
+ return -EINVAL;
+
+ nt->extended = extended;
+
+ return strnlen(buf, count);
+}
+
static ssize_t store_dev_name(struct netconsole_target *nt,
const char *buf,
size_t count)
@@ -507,6 +553,7 @@ static struct netconsole_target_attr net
__CONFIGFS_ATTR(_name, S_IRUGO | S_IWUSR, show_##_name, store_##_name)
NETCONSOLE_TARGET_ATTR_RW(enabled);
+NETCONSOLE_TARGET_ATTR_RW(extended);
NETCONSOLE_TARGET_ATTR_RW(dev_name);
NETCONSOLE_TARGET_ATTR_RW(local_port);
NETCONSOLE_TARGET_ATTR_RW(remote_port);
@@ -517,6 +564,7 @@ NETCONSOLE_TARGET_ATTR_RW(remote_mac);
static struct configfs_attribute *netconsole_target_attrs[] = {
&netconsole_target_enabled.attr,
+ &netconsole_target_extended.attr,
&netconsole_target_dev_name.attr,
&netconsole_target_local_port.attr,
&netconsole_target_remote_port.attr,
@@ -727,6 +775,82 @@ static struct notifier_block netconsole_
.notifier_call = netconsole_netdev_event,
};
+/**
+ * send_ext_msg_udp - send extended log message to target
+ * @nt: target to send message to
+ * @msg: extended log message to send
+ * @msg_len: length of message
+ *
+ * Transfer extended log @msg to @nt. If @msg is longer than
+ * MAX_PRINT_CHUNK, it'll be split and transmitted in multiple chunks with
+ * ncfrag header field added to identify them.
+ */
+static void send_ext_msg_udp(struct netconsole_target *nt, const char *msg,
+ int msg_len)
+{
+ static char buf[MAX_PRINT_CHUNK]; /* protected by target_list_lock */
+ const char *header, *body;
+ int offset = 0;
+ int header_len, body_len;
+
+ if (msg_len <= MAX_PRINT_CHUNK) {
+ netpoll_send_udp(&nt->np, msg, msg_len);
+ return;
+ }
+
+ /* need to insert extra header fields, detect header and body */
+ header = msg;
+ body = memchr(msg, ';', msg_len);
+ if (WARN_ON_ONCE(!body))
+ return;
+
+ header_len = body - header;
+ body_len = msg_len - header_len - 1;
+ body++;
+
+ /*
+ * Transfer multiple chunks with the following extra header.
+ * "ncfrag=<byte-offset>/<total-bytes>"
+ */
+ memcpy(buf, header, header_len);
+
+ while (offset < body_len) {
+ int this_header = header_len;
+ int this_chunk;
+
+ this_header += scnprintf(buf + this_header,
+ sizeof(buf) - this_header,
+ ",ncfrag=%d/%d;", offset, body_len);
+
+ this_chunk = min(body_len - offset,
+ MAX_PRINT_CHUNK - this_header);
+ if (WARN_ON_ONCE(this_chunk <= 0))
+ return;
+
+ memcpy(buf + this_header, body + offset, this_chunk);
+
+ netpoll_send_udp(&nt->np, buf, this_header + this_chunk);
+
+ offset += this_chunk;
+ }
+}
+
+static void write_ext_msg(struct console *con, const char *msg,
+ unsigned int len)
+{
+ struct netconsole_target *nt;
+ unsigned long flags;
+
+ if ((oops_only && !oops_in_progress) || list_empty(&target_list))
+ return;
+
+ spin_lock_irqsave(&target_list_lock, flags);
+ list_for_each_entry(nt, &target_list, list)
+ if (nt->extended && nt->enabled && netif_running(nt->np.dev))
+ send_ext_msg_udp(nt, msg, len);
+ spin_unlock_irqrestore(&target_list_lock, flags);
+}
+
static void write_msg(struct console *con, const char *msg, unsigned int len)
{
int frag, left;
@@ -742,7 +866,7 @@ static void write_msg(struct console *co
spin_lock_irqsave(&target_list_lock, flags);
list_for_each_entry(nt, &target_list, list) {
- if (nt->enabled && netif_running(nt->np.dev)) {
+ if (!nt->extended && nt->enabled && netif_running(nt->np.dev)) {
/*
* We nest this inside the for-each-target loop above
* so that we're able to get as much logging out to
@@ -761,6 +885,12 @@ static void write_msg(struct console *co
spin_unlock_irqrestore(&target_list_lock, flags);
}
+static struct console netconsole_ext = {
+ .name = "netcon_ext",
+ .flags = CON_EXTENDED, /* starts disabled, registered on first use */
+ .write = write_ext_msg,
+};
+
static struct console netconsole = {
.name = "netcon",
.flags = CON_ENABLED,
@@ -783,7 +913,11 @@ static int __init init_netconsole(void)
goto fail;
}
/* Dump existing printks when we register */
- netconsole.flags |= CON_PRINTBUFFER;
+ if (nt->extended)
+ netconsole_ext.flags |= CON_PRINTBUFFER |
+ CON_ENABLED;
+ else
+ netconsole.flags |= CON_PRINTBUFFER;
spin_lock_irqsave(&target_list_lock, flags);
list_add(&nt->list, &target_list);
@@ -799,6 +933,8 @@ static int __init init_netconsole(void)
if (err)
goto undonotifier;
+ if (netconsole_ext.flags & CON_ENABLED)
+ register_console(&netconsole_ext);
register_console(&netconsole);
pr_info("network logging started\n");
@@ -827,6 +963,7 @@ static void __exit cleanup_netconsole(vo
{
struct netconsole_target *nt, *tmp;
+ unregister_console(&netconsole_ext);
unregister_console(&netconsole);
dynamic_netconsole_exit();
unregister_netdevice_notifier(&netconsole_netdev_notifier);
_
Patches currently in -mm which might be from tj@xxxxxxxxxx are
origin.patch
gfp-add-__gfp_noaccount.patch
gfp-add-__gfp_noaccount-v2.patch
gfp-add-__gfp_noaccount-v2-fix.patch
kernfs-do-not-account-ino_ida-allocations-to-memcg.patch
block-restore-proc-partitions-to-not-display-non-partitionable-removable-devices.patch
fs-mpagec-forgotten-write_sync-in-case-of-data-integrity-write.patch
printk-guard-the-amount-written-per-line-by-devkmsg_read.patch
printk-factor-out-message-formatting-from-devkmsg_read.patch
printk-implement-support-for-extended-console-drivers.patch
netconsole-remove-unnecessary-netconsole_target_get-out-from-write_msg.patch
netconsole-make-netconsole_target-enabled-a-bool.patch
netconsole-make-all-dynamic-netconsoles-share-a-mutex.patch
netconsole-implement-extended-console-support.patch
bitmap-remove-explicit-newline-handling-using-scnprintf-format-string.patch
bitmap-remove-explicit-newline-handling-using-scnprintf-format-string-fix.patch
ipc-msgc-msgsnd-use-freezable-blocking-call.patch
msgrcv-use-freezable-blocking-call.patch
linux-next.patch
printk-improve-the-description-of-dev-kmsg-line-format.patch
--
To unsubscribe from this list: send the line "unsubscribe mm-commits" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
