Hi Hans, On Sunday 23 September 2012 11:18:45 Hans Verkuil wrote: > On Sat September 22 2012 14:38:07 Sakari Ailus wrote: > > Hans Verkuil wrote: > > > On Thu September 20 2012 22:21:22 Sakari Ailus wrote: > > >> Hi all, > > >> > > >> > > >> This RFC intends to summarise and further the recent discussion on > > >> linux-media regarding the proposed changes of timestamping V4L2 > > >> buffers. > > >> > > >> > > >> The problem > > >> =========== > > >> > > >> The V4L2 has long used realtime timestamps (such as > > >> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before > > >> handing them over to the user. This has been found problematic in > > >> associating the video buffers with data from other sources: realtime > > >> clock may jump around due to daylight saving time, for example, and > > >> ALSA (audio-video synchronisation is a common use case) user space API > > >> does not provide the user with realtime timestamps, but instead uses > > >> monotonic time (i.e. clock_gettime(CLOCK_MONOTONIC, ...)). > > >> > > >> This is especially an issue in embedded systems where video recording > > >> is a common use case. Drivers typically used in such systems have > > >> silently switched to use monotonic timestamps. While against the spec, > > >> this is necessary for those systems to operate properly. > > >> > > >> In general, realtime timestamps are seen of little use in other than > > >> debugging purposes, but monotonic timestamps are fine for that as well. > > >> It's still possible that an application I'm not aware of uses them in > > >> a peculiar way that would be adversely affected by changing to > > >> monotonic timestamps. Nevertheless, we're not supposed to break the > > >> API (or ABI). It'd be also very important for the application to know > > >> what kind of timestamps are provided by the device. > > >> > > >> > > >> Requirements, wishes and constraints > > >> ==================================== > > >> > > >> Now that it seems to be about the time to fix these issues, it's worth > > >> looking a little bit to the future to anticipate the coming changes to > > >> be able to accommodate them better later on. > > >> > > >> - The new default should be monotonic. As the monotonic timestamps are > > >> seen to be the most useful, they should be made the default. > > >> > > >> - timeval vs. timespec. The two structs can be used to store timestamp > > >> information. They are not compatible with each other. It's a little bit > > >> uncertain what's the case with all the architectures but it looks like > > >> the timespec fits into the space of timeval in all cases. If timespec > > >> is considered to be used somewhere the compatibility must be ensured. > > >> Timespec is better than timeval since timespec has more precision and > > >> it's the same struct that's used everywhere else in the V4L2 API: > > >> timespec does not need conversion to timespec in the user space. > > >> > > >> struct timespec { > > >> __kernel_time_t tv_sec; /* seconds */ > > >> long tv_nsec; /* nanoseconds */ > > >> }; > > >> > > >> struct timeval { > > >> __kernel_time_t tv_sec; /* seconds */ > > >> __kernel_suseconds_t tv_usec; /* microseconds */ > > >> }; > > >> > > >> To be able to use timespec, the user would have to most likely > > >> explicitly choose to do that. > > >> > > >> - Users should know what kind of timestamps the device produces. This > > >> includes existing and future kernels. What should be considered are > > >> uninformed porting drivers back and forth across kernel versions and > > >> out-of-date kernel header files. > > >> > > >> - Device-dependent timestamps. Some devices such as the uvcvideo ones > > >> produce device-dependent timestamps for synchronising video and audio, > > >> both produced by the same physical hardware device. For uvcvideo these > > >> timestamps are unsigned 32-bit integers. > > >> > > >> - There's also another clock, Linux-specific raw monotonic clock (as in > > >> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some > > >> use cases than the regular monotonic clock. The difference is that the > > >> raw monotonic clock is free from the NTP adjustments. It would be nice > > >> for the user to be able to choose the clock used for timestamps. This > > >> is especially important for device-dependent timestamps: not all > > >> applications can be expected to be able to use them. > > >> > > >> - The field adjacent to timestamp, timecode, is 128 bits wide, and not > > >> used by a single driver. This field could be re-used. > > >> > > >> > > >> Possible solutions > > >> ================== > > >> > > >> Not all of the solutions below that have been proposed are mutually > > >> exclusive. That's also what's making the choice difficult: the ultimate > > >> solution to the issue of timestamping may involve several of these --- > > >> or possibly something better that's not on the list. > > >> > > >> > > >> Use of timespec > > >> --------------- > > >> > > >> If we can conclude timespec will always fit into the size of timeval > > >> (or timecode) we could use timespec instead. The solution should still > > >> make the use of timespec explicit to the user space. This seems to > > >> conflict with the idea of making monotonic timestamps the default: the > > >> default can't be anything incompatible with timeval, and at the same > > >> time it's the most important that the monotonic timestamps are > > >> timespec. > > > > > > We have to keep timeval. Changing this will break the ABI. I see > > > absolutely no reason to use timespec for video. At 60 Hz a frame takes > > > 16.67 ms, and that's far, far removed from ns precisions. Should we ever > > > have to support high-speed cameras running at 60000 Hz, then we'll talk > > > again. > > > > > > For me this is a non-issue. > > > > > >> Kernel version as indicator of timestamp > > >> ---------------------------------------- > > >> > > >> Conversion of drivers to use monotonic timestamp is trivial, so the > > >> conversion could be done once and for all drivers. The kernel version > > >> could be used to indicate the type of the timestamp. > > >> > > >> If this approach is taken care must be taken when new drivers are > > >> integrated: developers sometimes use old kernels for development and > > >> might also use an old driver for guidance on timestamps, thus using > > >> real-time timestamps when monotonic timestamps should be used. > > > > > > More importantly, this also fails when users use out-of-tree drivers. > > > > Could you mention some examples what we could be breaking in particular? > > The version reported in QUERYCAP by out-of-tree drivers is unreliable. It > has no relationship with a kernel version in general. Actually, even inside > the kernel I've come across drivers that still fill in the version number > themselves. > > > >> This approach has an advantage over the capability flag below: which is > > >> that we don't populate the interface with essentially dead definitions. > > > > > > Using a kernel version to decide whether some feature is available or > > > not is IMHO something of a last resort. It's very application > > > unfriendly. > > > > Could be, but that's a passing pain. We're going to live with the flags > > for the foreseeable future, whether we need them or not. > > Never underestimate the glacial speed (or lack of speed) with which > applications adapt to new features. > > > >> Capability flag for monotonic timestamps > > >> ---------------------------------------- > > >> > > >> A capability flag can be used to tell whether the timestamp is > > >> monotonic. However, it's not extensible cleanly to provide selectable > > >> timestamps. These are not features that are needed right now, though. > > >> > > >> The upside of this option is ease of implementation and use, but it's > > >> not extensible. Also we're left with a flag that's set for all drivers: > > >> in the end it provides no information to the user and is only noise in > > >> the spec. > > >> > > >> > > >> Control for timestamp type > > >> -------------------------- > > >> > > >> Using a control to tell the type of the timestamp is extensible but not > > >> as easy to implement than the capability flag: each and every device > > >> would get an additional control. The value should likely be also file > > >> handle specific, and we do not have file handle specific controls yet. > > > > > > Yes, we do. You can make per-file handle controls. M2M devices need > > > that. > > > > Thanks for correcting me. > > > > > I'm not sure why this would be filehandle specific, BTW. > > > > Good point. I thought that as other properties of the buffers are > > specific to file handles, including format when using CREATE_BUFS, it'd > > make sense to make the timestamp source file-handle specific as well. > > > > What do you think? > > I don't think it makes sense to have different streams from the same device > use different clocks. Not on different streams, but on the same stream for two consecutive applications. If application A selects a device-specific timestamps, application B that is not aware of timestamp selection should get standard timestamps when it streams from the device. > > >> In the meantime the control could be read-only, and later made > > >> read-write when the timestamp type can be made selectable. Much of he > > >> work of timestamping can be done by the framework: drivers can use a > > >> single helper function and need to create one extra standard control. > > >> > > >> Should the control also have an effect on the types of the timestamps > > >> in V4L2 events? Likely yes. > > > > > > You are missing one other option: > > > > > > Using v4l2_buffer flags to report the clock > > > ------------------------------------------- > > > > > > By defining flags like this: > > > > > > V4L2_BUF_FLAG_CLOCK_MASK 0x7000 > > > /* Possible Clocks */ > > > V4L2_BUF_FLAG_CLOCK_UNKNOWN 0x0000 /* system or monotonic, we don't > > > know */ V4L2_BUF_FLAG_CLOCK_MONOTONIC 0x1000 > > > > > > you could tell the application which clock is used. > > > > > > This does allow for more clocks to be added in the future and clock > > > selection would then be done by a control or possibly an ioctl. For now > > > there are no plans to do such things, so this flag should be > > > sufficient. And it can be implemented very efficiently. It works with > > > existing drivers as well, since they will report CLOCK_UNKNOWN. > > > > > > I am very much in favor of this approach. > > > > Thanks for adding this. I knew I was forgetting something but didn't > > remember what --- I swear it was unintentional! :-) > > > > If we'd add more clocks without providing an ability to choose the clock > > from the user space, how would the clock be selected? It certainly isn't > > the driver's job, nor I think it should be system-specific either > > (platform data on embedded systems). > > IF a driver supports more than one clock (which I really don't see happening > anytime soon), uvcvideo will support both PTS and monotonic system time (and possibly raw monotonic system time). > then we either need a control to select the clock or an ioctl. And something > as well to enumerate the available clocks. I'm leaning towards ioctls, but I > think this should be decided if we ever get an actual use-case for this. Wouldn't a control be easier ? The control API already provides everything we need. > > It's up to the application and its needs. That would suggest we should > > always provide monotonic timestamps to applications (besides a potential > > driver-specific timestamp), and for that purpose the capability flag --- > > I admit I disliked the idea at first --- is enough. > > > > What comes to buffer flags, the application would also have to receive > > the first buffer from the device to even know what kind of timestamps > > the device uses, or at least call QUERYBUF. And in principle the flag > > should be checked on every buffer, unless we also specify the flag is > > the same for all buffers. And at certain point this will stop to make > > any sense... > > It should definitely be the same for all buffers. And since apps will > typically call querybuf anyway I don't see this as a problem. These > clocks are also specific to the streaming I/O API, so reporting this as > part of that API makes sense to me as well. > > > A capability flag is cleaner solution from this perspective, and it can > > be amended by a control (or an ioctl) later on: the flag can be > > disregarded by applications whenever the control is present. > > Yuck. > > > If the application doesn't know about the control it can still rely on the > > flag. (I think this would be less clean than to go for the control right > > from the beginning, but better IMO.) > > > > >> Device-dependent timestamp > > >> -------------------------- > > >> > > >> Should we agree on selectable timestamps, the existing timestamp field > > >> (or a union with another field of different type) could be used for > > >> the device-dependent timestamps. > > > > > > No. Device timestamps should get their own field. You want to be able to > > > relate device timestamps with the monotonic timestamps, so you need > > > both. > > > > > >> Alternatively we can choose to re-use the existing timecode field. > > >> > > >> At the moment there's no known use case for passing device-dependent > > >> timestamps at the same time with monotonic timestamps. > > > > > > Well, the use case is there, but there is no driver support. The device > > > timestamps should be 64 bits to accomodate things like PTS and DTS from > > > MPEG streams. Since timecode is 128 bits we might want to use two u64 > > > fields or perhaps 4 u32 fields. > > > > That should be an union for different kinds (or rather types) of > > device-dependent timestamps. On uvcvideo I think this is u32, not u64. > > We should be also able to tell what kind device dependent timestamp > > there is --- should buffer flags be used for that as well? > > That's definitely part of the buffer flags. The presence of timecode is > already signalled using that. And not every buffer may have device > timestamps (that depends on the hardware), so you have to signal it through > the buffer flags. > > An anonymous union might be best with the buffer flags signalling the type > of the union. What I don't know is how to specify the type. Shall we just > specify the type of the union (e.g. 4 u32 fields or 2 u64 fields) and leave > the interpretation of those fields up to the application based on the > driver name? Or shall the type act more like a fourcc in that it also > uniquely identifies the interpretation of the timestamps? > > Or should all device timestamps be converted to a timespec by the driver? Conversion to a timespec requires floating point operation to be accurate in the general case. Implementing proper conversion algorithms requires userspace code. > Answers on a postcard. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html