Re: Splitting large message written to stdout, explanation?

[Lennart Poettering <lennart@xxxxxxxxxxxxxx>]

On So, 21.05.23 15:32, Virendra Negi (virendra.negi@xxxxxxxxxxxxxxxxxxxx) wrote:

> It's been over a week I have been chasing this
> https://github.com/rsyslog/rsyslog/issues/5137
>
> I was unsure how to ensure that the systemd (since I was getting nowhere
> with rsyslog)  split the message instead of the application program doing
> this.

How did you send the original message? via stdout/stderr? via syslog?
via journald's native protocol?

The syslog protocol is not really suitable for passing around massive
amounts of data. First of all, it is mostly commonly used
datagram-based transports, which means 64K (in case of AF_UNIX) or
~1.5K (in case of AF_INET/AF_INET6 on ethernet without fragmentation)
size limits.

systemd-journald does not split up messages it receives via the syslog
protocol, since clients send their messages via SOCK_DGRAM/AF_UNIX a
64K limit applies however. YOu have to subtract a bunch of bytes of
those, since the timestamp/identifier/pid will take away some of the
datagram however.

If you use the native journal protocol systemd-journald is actually
happy to take arbitrary sized data, since besides datagrams it also
accepts payload in memfds, which can be more or less unlimited in
size.

If the message comes via stdout/stderr systemd will break up lines for
four different reasons:

1. A newline byte was seen
2. A NUL byte was seen
3. The sending PID changed (i.e. two programs log interleaved to the
   same stdout/stderr stream)
4. EOF was seen on the stream
5. The maximum line limit was hit (as configured via journald.conf's
   LineMax=, which defaults to 48K)

Which of the 5 it is you can see from _LINE_BREAK= (which is
suppressed in case of regular newline, however).

Generally, I'd suggest that apps remain conservative with the log
message sizes they generate, for compat with classic syslogs. Sending
megabytes of data in a single log message is a pretty poor idea I am sure.

> StandardOutput=syslog
> StandardError=syslog
> SyslogIdentifier=sbagent
>
> And set the MaxMessageSize to 64K and what I saw was the 1.5MB long message
> that was truncating earlier went through this time without truncation and a
> split happened the way I wanted it to be.

So apparently your are logging via stdout/stderr. In that case
LineMax= as mentioned above will help you. Still though: bad idea to
send a 1.5 MB line that way.

Lennart

--
Lennart Poettering, Berlin