On Tue, Aug 20, 2024 at 11:07:40AM GMT, Jocelyn Falempe wrote:
> Mesa doesn't support BGRX8888, that means most wayland compositors
> don't work on big endian guests.
So you are doing a hard switch from native endian to little endian.
While this should be fine for modern userspace API (aka ADDFB2 ioctl) it
is not for older APIs (ADDFB ioctl, also fbdev emulation) where only
depth=32 is specified and userspace typically expects a framebuffer in
native byte order.
Ideally virtio-gpu would support both big endian and little endian
framebuffer formats (simliar to bochs drm driver). That probably is a
somewhat more invasive change because the DRM_IOCTL_MODE_CREATE_DUMB
doesn't tell use the format which will be used. Possible options I see:
(1) Be lazy on creating host resources, i.e. call
virtio_gpu_cmd_create_resource() not at DRM_IOCTL_MODE_CREATE_DUMB
time but later when the resource will be actually be used (and
specifically after DRM_IOCTL_MODE_ADDFB(2) ioctl so we know the
format). Needs additional state tracking (whenever the resource
has been created or not) in possibly lots of places.
(2) Support changing the resource format, i.e. in case
DRM_IOCTL_MODE_ADDFB(2) is called with a format different from the
current one go through a destroy-and-recreate cycle for the host
resource. Might have tricky corner cases (resource being in use
when DRM_IOCTL_MODE_ADDFB(2) is called).
HTH & take care,
Gerd
