On Friday 21 September 2012 11:33:24 Hans Verkuil wrote: > On Thu September 20 2012 22:21:22 Sakari Ailus wrote: > > Hi all, > > > > > > This RFC intends to summarise and further the recent discussion on > > linux-media regarding the proposed changes of timestamping V4L2 buffers. > > > > > > The problem > > =========== > > > > The V4L2 has long used realtime timestamps (such as > > clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before > > handing them over to the user. This has been found problematic in > > associating the video buffers with data from other sources: realtime clock > > may jump around due to daylight saving time, for example, and ALSA > > (audio-video synchronisation is a common use case) user space API does not > > provide the user with realtime timestamps, but instead uses monotonic time > > (i.e. clock_gettime(CLOCK_MONOTONIC, ...)). > > > > This is especially an issue in embedded systems where video recording is a > > common use case. Drivers typically used in such systems have silently > > switched to use monotonic timestamps. While against the spec, this is > > necessary for those systems to operate properly. > > > > In general, realtime timestamps are seen of little use in other than > > debugging purposes, but monotonic timestamps are fine for that as well. > > It's still possible that an application I'm not aware of uses them in a > > peculiar way that would be adversely affected by changing to monotonic > > timestamps. Nevertheless, we're not supposed to break the API (or ABI). > > It'd be also very important for the application to know what kind of > > timestamps are provided by the device. > > > > > > Requirements, wishes and constraints > > ==================================== > > > > Now that it seems to be about the time to fix these issues, it's worth > > looking a little bit to the future to anticipate the coming changes to be > > able to accommodate them better later on. > > > > - The new default should be monotonic. As the monotonic timestamps are > > seen > > to be the most useful, they should be made the default. > > > > - timeval vs. timespec. The two structs can be used to store timestamp > > information. They are not compatible with each other. It's a little bit > > uncertain what's the case with all the architectures but it looks like the > > timespec fits into the space of timeval in all cases. If timespec is > > considered to be used somewhere the compatibility must be ensured. > > Timespec > > is better than timeval since timespec has more precision and it's the same > > struct that's used everywhere else in the V4L2 API: timespec does not need > > conversion to timespec in the user space. > > > > struct timespec { > > > > __kernel_time_t tv_sec; /* seconds */ > > long tv_nsec; /* nanoseconds */ > > > > }; > > > > struct timeval { > > > > __kernel_time_t tv_sec; /* seconds */ > > __kernel_suseconds_t tv_usec; /* microseconds */ > > > > }; > > > > To be able to use timespec, the user would have to most likely explicitly > > choose to do that. > > > > - Users should know what kind of timestamps the device produces. This > > includes existing and future kernels. What should be considered are > > uninformed porting drivers back and forth across kernel versions and > > out-of-date kernel header files. > > > > - Device-dependent timestamps. Some devices such as the uvcvideo ones > > produce device-dependent timestamps for synchronising video and audio, > > both > > produced by the same physical hardware device. For uvcvideo these > > timestamps are unsigned 32-bit integers. > > > > - There's also another clock, Linux-specific raw monotonic clock (as in > > clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use > > cases than the regular monotonic clock. The difference is that the raw > > monotonic clock is free from the NTP adjustments. It would be nice for the > > user to be able to choose the clock used for timestamps. This is > > especially > > important for device-dependent timestamps: not all applications can be > > expected to be able to use them. > > > > - The field adjacent to timestamp, timecode, is 128 bits wide, and not > > used > > by a single driver. This field could be re-used. > > > > > > Possible solutions > > ================== > > > > Not all of the solutions below that have been proposed are mutually > > exclusive. That's also what's making the choice difficult: the ultimate > > solution to the issue of timestamping may involve several of these --- or > > possibly something better that's not on the list. > > > > > > Use of timespec > > --------------- > > > > If we can conclude timespec will always fit into the size of timeval (or > > timecode) we could use timespec instead. The solution should still make > > the use of timespec explicit to the user space. This seems to conflict > > with > > the idea of making monotonic timestamps the default: the default can't be > > anything incompatible with timeval, and at the same time it's the most > > important that the monotonic timestamps are timespec. > > We have to keep timeval. Changing this will break the ABI. I see absolutely > no reason to use timespec for video. At 60 Hz a frame takes 16.67 ms, and > that's far, far removed from ns precisions. Should we ever have to support > high-speed cameras running at 60000 Hz, then we'll talk again. > > For me this is a non-issue. > > > Kernel version as indicator of timestamp > > ---------------------------------------- > > > > Conversion of drivers to use monotonic timestamp is trivial, so the > > conversion could be done once and for all drivers. The kernel version > > could > > be used to indicate the type of the timestamp. > > > > If this approach is taken care must be taken when new drivers are > > integrated: developers sometimes use old kernels for development and might > > also use an old driver for guidance on timestamps, thus using real-time > > timestamps when monotonic timestamps should be used. > > More importantly, this also fails when users use out-of-tree drivers. > > > This approach has an > > advantage over the capability flag below: which is that we don't populate > > the interface with essentially dead definitions. > > Using a kernel version to decide whether some feature is available or not is > IMHO something of a last resort. It's very application unfriendly. > > Capability flag for monotonic timestamps > > ---------------------------------------- > > > > A capability flag can be used to tell whether the timestamp is monotonic. > > However, it's not extensible cleanly to provide selectable timestamps. > > These are not features that are needed right now, though. > > > > The upside of this option is ease of implementation and use, but it's not > > extensible. Also we're left with a flag that's set for all drivers: in the > > end it provides no information to the user and is only noise in the spec. > > > > > > Control for timestamp type > > -------------------------- > > > > Using a control to tell the type of the timestamp is extensible but not as > > easy to implement than the capability flag: each and every device would > > get > > an additional control. The value should likely be also file handle > > specific, and we do not have file handle specific controls yet. > > Yes, we do. You can make per-file handle controls. M2M devices need that. > > I'm not sure why this would be filehandle specific, BTW. > > > In the meantime the control could be read-only, and later made read-write > > when the timestamp type can be made selectable. Much of he work of > > timestamping can be done by the framework: drivers can use a single helper > > function and need to create one extra standard control. > > > > Should the control also have an effect on the types of the timestamps in > > V4L2 events? Likely yes. > > You are missing one other option: > > Using v4l2_buffer flags to report the clock > ------------------------------------------- > > By defining flags like this: > > V4L2_BUF_FLAG_CLOCK_MASK 0x7000 > /* Possible Clocks */ > V4L2_BUF_FLAG_CLOCK_UNKNOWN 0x0000 /* system or monotonic, we don't know */ > V4L2_BUF_FLAG_CLOCK_MONOTONIC 0x1000 > > you could tell the application which clock is used. > > This does allow for more clocks to be added in the future and clock > selection would then be done by a control or possibly an ioctl. For now > there are no plans to do such things, so this flag should be sufficient. > And it can be implemented very efficiently. It works with existing drivers > as well, since they will report CLOCK_UNKNOWN. > > I am very much in favor of this approach. > > > Device-dependent timestamp > > -------------------------- > > > > Should we agree on selectable timestamps, the existing timestamp field (or > > a union with another field of different type) could be used for the > > device-dependent timestamps. > > No. Device timestamps should get their own field. You want to be able to > relate device timestamps with the monotonic timestamps, so you need both. > > Alternatively we can choose to re-use the > > existing timecode field. > > > > At the moment there's no known use case for passing device-dependent > > timestamps at the same time with monotonic timestamps. > > Well, the use case is there, but there is no driver support. The device > timestamps should be 64 bits to accomodate things like PTS and DTS from > MPEG streams. Since timecode is 128 bits we might want to use two u64 fields > or perhaps 4 u32 fields. > > > Now what? > > ========= > > > > Almost as many options have been presented as there were opinions, but we > > need to agree to have a single one. My personal leaning is on using a > > control for the purpose as it is the most flexible alternative. I'd still > > need to see an implementation of that but it doesn't seem that difficult, > > especially when it's read-only. And even for read-write control the vast > > majority of the work can be done by the V4L2 framework. > > > > Questions, comments and opinions are very, very welcome. > > Regards, > > Hans -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html