Hi Marek, I would like to recompile and install the kernel 6.6 on my ARM Chromebook. I would like to know if your patch has been accepted and included there. Thanks. On Wed, Nov 1, 2023 at 8:48 AM Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx> wrote: > > On 10/31/2023 8:08 AM, Marek Szyprowski wrote: > > Hi, > > > > On 31.10.2023 00:03, Mario Marietto wrote: > >> We are a team of linux enthusiasts who are trying to boot Xen on a > >> Samsung XE303C12 Chromebook aka "snow" following the suggestions in > >> the slide show presentation here: > >> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm > >> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it > >> is a Samsung armv7 chip with virtualization extensions. In particular, > >> we have it working fairly well both on the bare metal with a recent > >> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with > >> KVM, the older LTS kernel version is used to test KVM because support > >> for KVM on arm v7 was removed from Linux around kernel version 5.7. So > >> we know we have the hypervisor mode enabled because we were able to > >> use it with KVM. For Xen, we are using the latest Debian build of Xen > >> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre > >> (Debian 4.17.1+2-gb773c48e36-1) > >> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc > >> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The > >> Linux kernel is a custom build that adds the Xen config kernel options > >> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same > >> Chromebook model on the bare metal. I can provide the config options > >> of the kernel that was used if that is helpful. Our method of booting > >> is to have u-boot boot the Xen hypervisor and load the device tree > >> after adding the dom0 to the otherwise unaltered device tree from the > >> Linux kernel using u-boot fdt commands to add a /chosen node, as > >> described on the Xen wiki and in the pages linked from there. We have > >> also tried adding and loading an initrd.img using the device tree > >> /chosen node but that made no difference in our tests. We actually > >> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen > >> using the same version of u-boot that we used for KVM, but with a big > >> problem. The problem we see is that when booting the 6.1.59 kernel > >> version as dom0 with Xen, the screen is totally dark and the only way > >> to access the system is remotely through ssh. Logs indicate most > >> everything else is working, such as the wifi card so we can access it > >> remotely via ssh and a USB optical mouse lights up when connected so > >> USB is also working. Obviously, the disk is also working. The > >> Chromebook is configured to boot from the device's SD card slot by > >> turning on Chrome OS developer mode options to enable booting from the > >> SD card slot. The mystery is that when booting the exact same 6.1.59 > >> kernel on the bare metal instead of booting it as dom0 on Xen, it > >> boots up with full access to the screen and we can interact with the > >> system using the X.org windows system. But booting as dom0 with Xen, > >> the screen is totally dark and the only access we have to the system > >> is through the network via ssh. Also, when booting the 5.4.257 kernel > >> with KVM in hypervisor mode, the screen works and we can interact with > >> the system through the X.org windows system. Exploring the log file,we > >> have seen the errors below : > >> > >> Without Xen (or in bare metal): > >> > >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for > >> DMA mapping operations > >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops > >> 0xc0d96354) > >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops > >> 0xc0d97554) > >> devuan-bunsen kernel: exynos-drm exynos-drm: bound > >> 145b0000.dp-controller (ops 0xc0d97278) > >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops > >> 0xc0d97bd0) > >> ... > >> devuan-bunsen kernel: Console: switching to colour frame buffer device > >> 170x48 > >> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb > >> frame buffer device > >> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for > >> exynos-drm on minor 0 > >> > >> In this case,the kernel is able to use the exynos-drm kernel to start > >> the fb0 device. But with Xen we get this error with exynos-drm: > >> > >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for > >> DMA mapping operations > >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops > >> 0xc0d96354) > >> devuan-bunsen kernel: exynos-mixer 14450000.mixer: > >> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks > >> support for IOMMU > >> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind > >> 14450000.mixer (ops 0xc0d97554): -22 > >> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22 > >> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller > >> failed with error -22 > >> > >> I'm trying to find for a solution and I've googled a little bit and I > >> found this web site : > >> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@xxxxxxxxxxxxx/ > >> with your email address and I tried to ask for some help for fixing > >> the bug. Any ideas why booting the same Linux kernel that results in a > >> working X.org display on the bare metal instead as dom0 on Xen would > >> cause the display to remain dark, but most other basic functions would > >> work, such as network, disk, and USB ? thanks. > > > > > > Thanks for the detailed description! Good to hear that those boards are > > still being used for various projects. I also have Snow Chromebook and > > use it for daily tests of linux-next branch. > > Adding Julien Grall and Stefano Stabellini > > Hi Marek, > > Thanks for responding to Mario's question. I also have been doing these > experiments with a Chromebook Snow, and I am the one who reported this > problem on the xen-users ML here: > > https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html > > You might find that thread interesting, especially here with some additional > log messages from the exynos_drm driver (exynos_drm_dma.c, I believe): > > https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html > > This issue is also discussed some on the xen-devel ML here: > > https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html > > > > > Frankly speaking I have no idea what might happen wrong. There have been > > some changes recently in the Exynos IOMMU driver related to > > initialization, maybe your changes related to Xen enabling changed > > somehow the order of device initialization during boot. I assume that > > the device-tree you use for the bare metal run and Xen enabled run > > doesn't differ in the areas describing the hardware blocks. > > > > Please check if cherry-picking the commit > > https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f > > to your v6.1.59 based kernel helps anyhow. > > I tried adding that fix of the exynos IOMMU initialization from > Linux > 6.2 on top of the 6.1.59 kernel I used for the original report, > but that made no difference on Xen - it still failed with the mixer lacks > support for IOMMU message and the screen is totally dark. > > > > > If not, then as a temporary workaround please disable > > CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config > > and check what will happen (You will lose the HDMI output, but maybe > > this won't a big issue). > > This change causes the GPU to work fairly well AFAICS. Removing the mixer > and HDMI allowed the GPU to initialize, and the display manager started > normally and enabled logging into an ordinary X11 session. Based on the log > messages I was seeing, this was an obvious thing to try. Thanks for > suggesting it. > > But I have a question: > > How are the mixer and HDMI devices related to the main drm device? The problem > in the exynos_drm_dma driver was that on Xen, the main drm device wanted to > use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if > it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops, > but on bare metal all three devices want to use the IOMMU version of dma_ops. > > The problem obviously has something to do with the fact that Xen does not > expose the same IOMMU capability to Linux as is available when running on > the bare metal. > > Cheers, > > > > > > > Best regards > -- Mario.
