Hi, > > I guess that makes changing drm_mode_legacy_fb_format + drivers a > > reasonable option ... > > Yeah, I came to the same conclusion after chatting with some > folks on irc. > > So my current idea is that we change any driver that wants to follow the > CPU endianness to declare support for big endian formats if the CPU is > big endian. Presumably these are mostly the virtual GPU drivers. Agree. Easy. > Additonally we'll make the mapping performed by drm_mode_legacy_fb_format() > driver controlled. I don't think this is useful. IMO drm_mode_legacy_fb_format should return host endian formats unconditionally. > That way drivers that got changed to follow CPU > endianness can return a framebuffer that matches CPU endianness. And > drivers that expect the GPU endianness to not depend on the CPU > endianness will keep working as they do now. The downside is that users > of the legacy addfb ioctl will need to magically know which endianness > they will get, but that is apparently already the case. Existing userspace expects host endian, and IMO we should maintain that behavior. > And users of > addfb2 will keep on specifying the endianness explicitly with > DRM_FORMAT_BIG_ENDIAN vs. 0. I'd drop DRM_FORMAT_BIG_ENDIAN. At least for the virt drivers it doesn't buy us anything. They support 32bpp / 8 bpc formats only[1], and for those I can specify the byteswapped format version without a bigendian flag because we have fourccs for everything we need. There is a WIP patch series at https://www.kraxel.org/cgit/linux/log/?h=drm-byteorder Needs more testing and better commit messages. /me plans to polish & post next week, but feel free to look and comment. cheers, Gerd [1] Everything else is a PITA to deal with on the host side because I can't offload that to pixman. There is no support for PIXMAN_r5g6b5 or PIXMAN_x2b10g10r10 in non-host byte order.
