Le mardi 25 juillet 2023 à 11:09 +0200, Paul Kocialkowski a écrit : > Hi Nicolas, > > On Mon 24 Jul 23, 10:03, Nicolas Dufresne wrote: > > Le vendredi 21 juillet 2023 à 20:19 +0200, Michael Grzeschik a écrit : > > > > As a result, we cannot expect that any given encoder is able to produce frames > > > > for any set of headers. Reporting related constraints and limitations (beyond > > > > profile/level) seems quite difficult and error-prone. > > > > > > > > So it seems that keeping header generation in-kernel only (close to where the > > > > hardware is actually configured) is the safest approach. > > > > > > For the case with the rkvenc, the headers are also not created by the > > > kernel driver. Instead we use the gst_h264_bit_writer_sps/pps functions > > > that are part of the codecparsers module. > > > > One level of granularity we can add is split headers (like SPS/PPS) and > > slice/frame headers. > > Do you mean asking the driver to return a buffer with only SPS/PPS and then > return another buffer with the slice/frame header? > > Looks like there's already a control for it: V4L2_CID_MPEG_VIDEO_HEADER_MODE > which takes either > - V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE: looks like what you're suggesting > - V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME: usual case > > So that could certainly be supported to easily allow userspace to stuff extra > NALUs in-between. Good point, indeed. > > > It remains that in some cases, like HEVC, when the slice > > header is byte aligned, it can be nice to be able to handle it at application > > side in order to avoid limiting SVC support (and other creative features) by our > > API/abstraction limitations. > > Do you see something in the headers that we expect the kernel to generate that > would need specific changes to support features like SVC? Getting the kernel to set the layer IDs, unless we have a full SVC configuration would just be extra indirections. That being said, if we mention HEVC, these IDs can be modified in-place as they use a fixed number of bytes. If you can split the headers appart, generating per layer headers in application makes a lot of sense. Traditionally, slice headers are made by stateless accelerators, but not the SPS/PPS and friend. > > From what I can see there's a svc_extension_flag that's only set for specific > NALUs (prefix_nal_unit/lice_layer_extension) so these could be inserted by > userspace. > > Also I'm not very knowledgeable about SVC so it's not very clear to me if it's > possible to take an encoder that doesn't support SVC and turn the resulting > stream into something SVC-ready by adding extra NAL units or if the encoder > should be a lot more involved. You can use any encoders to create a temporal SVC. Its only about the referencing pattern, made so you can reduce the framerate (dividing by 2 usually). For spatial layer, the encoders need scaling capabilities. I'm not totally sure how multi-view work, but this is most likely just using left eye as reference (not having an I frame ever for the second eye). > > Also do you know if we have stateful codecs supporting SVC? We don't at the moment, they all produce headers with layer id hardcoded to 0 as far as I'm aware. The general plan (if it had continued) might have been to offer a memu based control, and drivers could offer from a list of preset SVC pattern. Mimicking what browsers needs: https://www.w3.org/TR/webrtc-svc/ > > > I think a certain level of "per CODEC" reasoning is > > also needed. Just like, I would not want to have to ask the kernel to generate > > user data SEI and other in-band data. > > Yeah it looks like there is definitely a need for adding extra NALUs from > userspace without passing that data to the kernel. > > Cheers, > > Paul >
