Hi Alvise , I have tried to use vfio-platform-v4 and vfio-platform-v5 kernel, but it doesn't work. Kernel Panic as mentioned before : ------------[ cut here ]------------ kernel BUG at mm/slub.c:3392! Internal error: Oops - BUG: 0 [#1] SMP ARM Modules linked in: CPU: 1 PID: 1711 Comm: qemu-system-arm Tainted: G W 3.14.0-rc4+ #2 task: edd76780 ti: edfcc000 task.ti: edfcc000 PC is at kfree+0x128/0x134 LR is at vfio_set_trigger.isra.5+0x40/0x104 pc : [<800da170>] lr : [<803197cc>] psr: 400b0013 sp : edfcde64 ip : 00080385 fp : 7e800100 r10: 00000000 r9 : edfcc000 r8 : ee5c4e80 r7 : 0000000e r6 : 80620584 r5 : 00000000 r4 : eea0b000 r3 : 00000400 r2 : 80385db0 r1 : eea120a0 r0 : 00000400 Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user Control: 30c53c7d Table: edf93fc0 DAC: 00000000 Process qemu-system-arm (pid: 1711, stack limit = 0xedfcc240) Stack: (0xedfcde64 to 0xedfce000) de60: eddb3c00 00000000 eddb3c00 0000000e ee5c4e80 00000000 803197cc de80: eddb3c00 00003b6e 00000000 ee5c4e80 eddb3c00 00003b6e 0000000d 80319940 dea0: 0000000e ee5c4e80 804929d8 ee5c4e80 eddee780 80319dc8 00000024 009331e4 dec0: 009331e4 803190ac 00000001 009331e4 00000018 00000024 00000000 00000000 dee0: 00000001 edd76780 edd00480 80614840 ee06a2a0 009331d0 eddee780 80315bfc df00: 80315bdc 800ef7e8 80477a18 00080802 edfcc000 800e1674 00000020 edfcdf50 df20: edfcdf60 00000003 8061e808 00080802 edfcdf60 800e180c 00000000 80652438 df40: 80477a18 8061e808 edf93e00 8011b3dc 00000000 00000009 8053bb0c 00000000 df60: ee501190 00000001 eddee780 009331d0 00003b6e 0000000d edfcc000 00000000 df80: 7e800100 800efd64 0000000e 00000001 009331d0 00429ed8 0000000d 00000036 dfa0: 8001c4a4 8001c320 009331d0 00429ed8 0000000d 00003b6e 009331d0 00000001 dfc0: 009331d0 00429ed8 0000000d 00000036 00000000 0094fa60 00000000 7e800100 dfe0: 003fa3fc 7e80008c 000bc5d0 76d552ec 200b0010 0000000d 00000000 8062c870 [<800da170>] (kfree) from [<803197cc>] (vfio_set_trigger.isra.5+0x40/0x104) [<803197cc>] (vfio_set_trigger.isra.5) from [<80319940>] (vfio_platform_set_irq_trigger+0xb0/0x144) [<80319940>] (vfio_platform_set_irq_trigger) from [<80319dc8>] (vfio_platform_set_irqs_ioctl+0x44/0x58) [<80319dc8>] (vfio_platform_set_irqs_ioctl) from [<803190ac>] (vfio_platform_ioctl+0x2e8/0x31c) [<803190ac>] (vfio_platform_ioctl) from [<80315bfc>] (vfio_device_fops_unl_ioctl+0x20/0x2c) [<80315bfc>] (vfio_device_fops_unl_ioctl) from [<800ef7e8>] (do_vfs_ioctl+0x78/0x5b8) [<800ef7e8>] (do_vfs_ioctl) from [<800efd64>] (SyS_ioctl+0x3c/0x5c) [<800efd64>] (SyS_ioctl) from [<8001c320>] (ret_fast_syscall+0x0/0x30) As you say, if using vfio-platform-v6 kernel and pl330-vfio-dev-new qemu, the vosys_dmatest should be successful? Is the qemu cmdline to start guest different? I got the following error: qemu-system-arm: -device pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,vfio_container=/dev/vfio/vfio: Parameter 'driver' expects pluggable device type Thanks, Shannon On 2014/6/24 22:50, Alvise Rigo wrote: > Hello, > > This is a known error, caused by using the last version of VFIO with an > old version of QEMU. > Try to use the vfio-platform-v4 kernel, if it still doesn't work, use > the vfio-platform-v5. > > Regards, > alvise > > Il 24/06/2014 15:19, Shannon Zhao ha scritto: >> Hi Alvise, >> >> Thanks for your help. >> I'm using pl330-vfio-dev branch but it's not up-to-date. >> Then I update the qemu code. >> Now I can start up the guest. >> >> I run the following cmd in guest : >> >> echo 1 > /sys/kernel/debug/vosys_dmatest/start >> dmaengine: dma_sync_wait: timeout! >> vosys dmatest: tx error >> vosys dmatest: src and destination do not match >> dmesg | grep vosys >> vosys dmatest: vosys dmatest module init >> vosys dmatest: tx error >> vosys dmatest: tx completed >> And at the terminal running isim_system there is an error : >> >> ERROR: DMAC330 channel thread abort. >> ERROR: DMAC330: Instruction fetch ERROR, thread set to FAULTING state >> >> Is there something wrong? >> >> Thanks, >> Shannon >> >> >> On 2014/6/24 17:14, Alvise Rigo wrote: >>> This is a QEMU error. Are you using the pl330-vfio-dev branch? Is it >>> up-to-date? >>> >>> alvise >>> >>> Il 24/06/2014 11:00, Shannon Zhao ha scritto: >>>> Hi Bharat, >>>> >>>> >>>> ls -l /sys/bus/platform/devices/2c0a0000.dma/iommu_group/devices >>>> total 0 >>>> lrwxrwxrwx 1 root root 0 Jan 1 02:58 2c0a0000.dma -> ../../../../devices/2c0a0000.dma >>>> >>>> Then I run "echo 2c0a0000.dma > /sys/bus/platform/drivers/vfio-platform/bind". There is /dev/vfio/0. >>>> >>>> But there is also an error reported. >>>> >>>> -nographic can not be used with -daemonize >>>> char device redirected to /dev/pts/3 (label pty0-base) >>>> PL330_VFIO: === VFIO device file descriptor 13 === >>>> Device has 1 region(s): >>>> PL330_VFIO: - Region 0: size=0x1000 offset=0x0 flags=0x7 >>>> PL330_VFIO: - IRQ 0: range of 1, flags=0x7 >>>> PL330_VFIO: - IRQ 1: range of 1, flags=0x7 >>>> PL330_VFIO: - IRQ 2: range of 1, flags=0x7 >>>> PL330_VFIO: - IRQ 3: range of 1, flags=0x7 >>>> PL330_VFIO: - IRQ 4: range of 1, flags=0x7 >>>> qemu-system-arm: -device pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,vfio_container=/dev/vfio/vfio: Property '.vfio_group' not found >>>> >>>> Thanks, >>>> Shannon >>>> On 2014/6/24 16:08, Bharat.Bhushan@xxxxxxxxxxxxx wrote: >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: Shannon Zhao [mailto:zhaoshenglong@xxxxxxxxxx] >>>>>> Sent: Tuesday, June 24, 2014 1:16 PM >>>>>> To: Bhushan Bharat-R65777; Alvise Rigo; a.motakis@xxxxxxxxxxxxxxxxxxxxxx; >>>>>> iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx; >>>>>> zhangchengxing@xxxxxxxxxx >>>>>> Subject: Re: Vfio-on-arm: the device tree does not contain the DMA device >>>>>> >>>>>> Hi Bharat, >>>>>> >>>>>> Thanks for your help. >>>>>> >>>>>> I executed the following cmd: >>>>>> echo vfio-platform > >>>>>> /sys/bus/platform/devices/2c0a0000.dma/driver_override >>>>> >>>>> You have to bind the device with vfio-platform driver after the above command. >>>>> >>>>>> But there is no /dev/vfio/0 and error reported starting guest >>>>>> >>>>>> -nographic can not be used with -daemonize >>>>>> char device redirected to /dev/pts/3 (label pty0-base) >>>>>> PL330_VFIO: Group is not viable (not all devices bound for vfio) >>>>> >>>>> A bit surprised; it should be failing when opening /dev/vfio/0 but it is failing in VFIO_GROUP_GET_STATUS ioctl. >>>>> >>>>> Also looks like there are more device attached to the said iommu-group. >>>>> Can check by running below command: >>>>> #ls -l /sys/bus/pci/devices/2c0a0000.dma/iommu_group/devices >>>>> >>>>> Thanks >>>>> -Bharat >>>>> >>>>>> qemu-system-arm: Error while probing real device. >>>>>> >>>>>> qemu-system-arm: Initialization of device pl330 failed Is there something >>>>>> wrong? >>>>>> >>>>>> Thank you very much. >>>>>> Shannon >>>>>> >>>>>> On 2014/6/24 15:05, Bharat.Bhushan@xxxxxxxxxxxxx wrote: >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Shannon Zhao [mailto:zhaoshenglong@xxxxxxxxxx] >>>>>>>> Sent: Tuesday, June 24, 2014 12:30 PM >>>>>>>> To: Alvise Rigo; Bhushan Bharat-R65777; >>>>>>>> a.motakis@xxxxxxxxxxxxxxxxxxxxxx; iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx; >>>>>>>> kvmarm@xxxxxxxxxxxxxxxxxxxxx; zhangchengxing@xxxxxxxxxx >>>>>>>> Subject: Re: Vfio-on-arm: the device tree does not contain the DMA >>>>>>>> device >>>>>>>> >>>>>>>> Hi Alvise, >>>>>>>> >>>>>>>> Thanks for your help. >>>>>>>> Now I can see /dev/vfio/vfio and /dev/vfio/0 after executing "echo >>>>>>>> 2c0a0000.dma >>>>>>>>> /sys/bus/platform/drivers/vfio-platform/vfio_bind". >>>>>>>> But when using qemu to start guest, there is an error : >>>>>>>> >>>>>>>> -nographic can not be used with -daemonize char device redirected to >>>>>>>> /dev/pts/3 (label pty0-base) >>>>>>>> PL330_VFIO: === VFIO device file descriptor 13 === Device has 1 >>>>>>>> region(s): >>>>>>>> PL330_VFIO: - Region 0: size=0x1000 offset=0x2c0a0000 flags=0x7 >>>>>>>> PL330_VFIO: - IRQ 0: range of 1, flags=0x7 ------------[ cut here >>>>>>>> ]------------ kernel BUG at mm/slub.c:3386! >>>>>>>> Internal error: Oops - BUG: 0 [#1] SMP ARM Modules linked in: >>>>>>>> CPU: 2 PID: 1746 Comm: qemu-system-arm Tainted: G W 3.14.0-rc1+ #19 >>>>>>>> task: ee665580 ti: ed0b8000 task.ti: ed0b8000 PC is at >>>>>>>> kfree+0x128/0x134 LR is at vfio_set_trigger.isra.5+0x40/0x104 >>>>>>>> pc : [<800d9e3c>] lr : [<80319094>] psr: 400f0013 >>>>>>>> sp : ed0b9e64 ip : 000800f3 fp : 000014ac >>>>>>>> r10: 00000000 r9 : ed0b8000 r8 : edeb3b80 >>>>>>>> r7 : 0000000e r6 : 80620584 r5 : 00000000 r4 : eea0b000 >>>>>>>> r3 : 00000400 r2 : 800f38b0 r1 : eea0ce60 r0 : 00000400 >>>>>>>> Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user >>>>>>>> Control: 30c53c7d Table: eddac080 DAC: 00000000 Process >>>>>>>> qemu-system-arm (pid: 1746, stack limit = 0xed0b8240) >>>>>>>> Stack: (0xed0b9e64 to 0xed0ba000) >>>>>>>> 9e60: ee603700 00000000 ee603700 0000000e edeb3b80 00000000 80319094 >>>>>>>> 9e80: ee603700 00003b6e 00000000 edeb3b80 ee603700 00003b6e 0000000d >>>>>>>> 80319208 >>>>>>>> 9ea0: 0000000e edeb3b80 80492a18 edeb3b80 ee57f780 80319690 00000024 >>>>>>>> 008b2074 >>>>>>>> 9ec0: 008b2074 8031886c 00000001 008b2074 00000018 00000024 00000000 >>>>>>>> 00000000 >>>>>>>> 9ee0: 00000001 00000004 00000001 00000001 ee088e70 008b2060 ee57f780 >>>>>>>> 803155d0 >>>>>>>> 9f00: 803155b0 800ef430 80477a58 00080802 ed0b8000 800e1358 00000020 >>>>>>>> ed0b9f50 >>>>>>>> 9f20: ed0b9f60 00000003 8061e808 00080802 ed0b9f60 800e14f0 00000000 >>>>>>>> 806522f8 >>>>>>>> 9f40: 80477a58 8061e808 ede4c1c0 8011af18 00000000 00000009 8053bbc0 >>>>>>>> 00000000 >>>>>>>> 9f60: ee683610 00000001 ee57f780 008b2060 00003b6e 0000000d ed0b8000 >>>>>>>> 00000000 >>>>>>>> 9f80: 000014ac 800ef9ac 0000000e 00000001 008b2060 00429eb0 0000000d >>>>>>>> 00000036 >>>>>>>> 9fa0: 8001c464 8001c2e0 008b2060 00429eb0 0000000d 00003b6e 008b2060 >>>>>>>> 00000001 >>>>>>>> 9fc0: 008b2060 00429eb0 0000000d 00000036 00000000 7ea99040 0089a220 >>>>>>>> 000014ac >>>>>>>> 9fe0: 003fa474 7ea98fc4 000bc5c8 76ceb2ec 200f0010 0000000d 00000000 >>>>>>>> 8062c770 [<800d9e3c>] (kfree) from [<80319094>] >>>>>>>> (vfio_set_trigger.isra.5+0x40/0x104) >>>>>>>> [<80319094>] (vfio_set_trigger.isra.5) from [<80319208>] >>>>>>>> (vfio_platform_set_irq_trigger+0xb0/0x144) >>>>>>>> [<80319208>] (vfio_platform_set_irq_trigger) from [<80319690>] >>>>>>>> (vfio_platform_set_irqs_ioctl+0x44/0x58) >>>>>>>> [<80319690>] (vfio_platform_set_irqs_ioctl) from [<8031886c>] >>>>>>>> (vfio_platform_ioctl+0x2e0/0x314) >>>>>>>> [<8031886c>] (vfio_platform_ioctl) from [<803155d0>] >>>>>>>> (vfio_device_fops_unl_ioctl+0x20/0x2c) >>>>>>>> [<803155d0>] (vfio_device_fops_unl_ioctl) from [<800ef430>] >>>>>>>> (do_vfs_ioctl+0x78/0x5b8) >>>>>>>> [<800ef430>] (do_vfs_ioctl) from [<800ef9ac>] (SyS_ioctl+0x3c/0x5c) >>>>>>>> [<800ef9ac>] (SyS_ioctl) from [<8001c2e0>] >>>>>>>> (ret_fast_syscall+0x0/0x30) >>>>>>>> Code: 15913038 e1a01003 e8bd45f0 eaff28e5 (e7f001f2) >>>>>>>> >>>>>>>> I'm using branch vfio-platform-v4 for kernel and pl330-vfio-dev for qemu. >>>>>>>> Following is qemu cmdline: >>>>>>>> ./qemu-system-arm -enable-kvm -M vexpress-a15 -cpu cortex-a15 \ >>>>>>>> -chardev pty,id=pty0,mux=on -monitor chardev:pty0 \ >>>>>>>> -serial chardev:pty0 -daemonize \ >>>>>>>> -device >>>>>>>> pl330,vfio_group="/dev/vfio/0",vfio_device="2c0a0000.dma",vfio_contai >>>>>>>> ner="/dev/v >>>>>>>> fio/vfio" \ >>>>>>>> -append "earlyprintk ip=dhcp console=ttyAMA0 rootwait" \ >>>>>>>> -kernel zImage -dtb guest-a15.dtb -initrd filesystem.cpio.gz >>>>>>>> - nographic >>>>>>>> >>>>>>>> If I change to branch vfio-platform-v6 for kernel, there is "/dev/vfio/vfio". >>>>>>>> But no "/sys/bus/platform/drivers/vfio-platform/vfio_bind". >>>>>>> >>>>>>> Can you Check if you have driver_override at >>>>>> "/sys/bus/platform/devices/DEVICE-XXX/ ? >>>>>>> If yes then you should use driver_override mechanism to bind that device to >>>>>> vfio-platform driver. >>>>>>> >>>>>>> Thanks >>>>>>> -Bharat >>>>>>> >>>>>>>> And if I use above qemu cmdline to start guest, there is also an error : >>>>>>>> PL330_VFIO: Group is not viable (not all devices bound for vfio) >>>>>>>> qemu-system-arm: Error while probing real device. >>>>>>>> >>>>>>>> qemu-system-arm: Initialization of device pl330 failed >>>>>>>> >>>>>>>> The usage about v4 and v6 has changed ? >>>>>>>> And would you tell me when you will update the guideline about vfio-on-arm? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Shannon >>>>>>>> On 2014/6/20 16:37, Alvise Rigo wrote: >>>>>>>>> Linaro has a great set of pre-built images here http://releases.linaro.org/. >>>>>>>>> >>>>>>>>> For example these should work fine: >>>>>>>>> http://releases.linaro.org/13.12/ubuntu/saucy-images >>>>>>>>> >>>>>>>>> alvise >>>>>>>>> >>>>>>>>> Il 20/06/2014 10:23, Shannon Zhao ha scritto: >>>>>>>>>> Hi Alvise, >>>>>>>>>> >>>>>>>>>> I'm so sorry that I can't use qemu-debootstrap because my >>>>>>>>>> environment can't >>>>>>>> access some links using proxy. >>>>>>>>>> Are there other ways to get a proper fs? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Shannon >>>>>>>>>> >>>>>>>>>> On 2014/6/20 15:15, Alvise Rigo wrote: >>>>>>>>>>> Hi Shannon, >>>>>>>>>>> >>>>>>>>>>> If you follow the steps in the section "Creating the File System" >>>>>>>>>>> of the guide you should be able to obtain a working fs. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> alvise >>>>>>>>>>> >>>>>>>>>>> Il 20/06/2014 04:57, Shannon Zhao ha scritto: >>>>>>>>>>>> Hi Bharat, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for your advice . >>>>>>>>>>>> >>>>>>>>>>>> I'm using the rootfs getting from the url >>>>>>>> "http://www.virtualopensystems.com/downloads/guides/kvm_on_arm/fs-ali >>>>>>>> p- >>>>>>>> armel.cramfs". >>>>>>>>>>>> >>>>>>>>>>>> Can you give me another link to get a newer rootfs? >>>>>>>>>>>> >>>>>>>>>>>> Thank you >>>>>>>>>>>> Shannon >>>>>>>>>>>> >>>>>>>>>>>> On 2014/6/19 20:20, Bharat.Bhushan@xxxxxxxxxxxxx wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Shannon Zhao [mailto:zhaoshenglong@xxxxxxxxxx] >>>>>>>>>>>>>> Sent: Thursday, June 19, 2014 5:47 PM >>>>>>>>>>>>>> To: Bhushan Bharat-R65777; Alvise Rigo; >>>>>>>>>>>>>> a.motakis@xxxxxxxxxxxxxxxxxxxxxx; >>>>>>>>>>>>>> iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx >>>>>>>>>>>>>> Subject: Re: Vfio-on-arm: the device tree does not contain the >>>>>>>>>>>>>> DMA device >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2014/6/19 20:12, Bharat.Bhushan@xxxxxxxxxxxxx wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>> From: Shannon Zhao [mailto:zhaoshenglong@xxxxxxxxxx] >>>>>>>>>>>>>>>> Sent: Thursday, June 19, 2014 5:38 PM >>>>>>>>>>>>>>>> To: Alvise Rigo; Bhushan Bharat-R65777; >>>>>>>>>>>>>>>> a.motakis@xxxxxxxxxxxxxxxxxxxxxx; >>>>>>>>>>>>>>>> iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx; >>>>>>>>>>>>>>>> kvmarm@xxxxxxxxxxxxxxxxxxxxx >>>>>>>>>>>>>>>> Subject: Re: Vfio-on-arm: the device tree does not contain >>>>>>>>>>>>>>>> the DMA device >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Alvise, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I run following cmds : >>>>>>>>>>>>>>>> echo 2c0a0000.dma > >>>>>>>>>>>>>>>> /sys/bus/platform/drivers/vfio-platform/vfio_bind >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> run "ls /dev/vfio/" before and after the above command >>>>>>>>>>>>>> Hi Bharat, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Nothing appeared either before or after the cmd "echo >>>>>>>>>>>>>> 2c0a0000.dma > /sys/bus/platform/drivers/vfio-platform/vfio_bind". >>>>>>>>>>>>>> There is only /dev/vfio. >>>>>>>>>>>>> >>>>>>>>>>>>> I am not sure, but may be you are using pretty old rootfs with >>>>>>>>>>>>> broken udev >>>>>>>> rules. Try to fetch a newer rootfs form somewhere. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> -Bharat >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> -Bharat >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ./qemu-system-arm -enable-kvm -M vexpress-a15 -cpu cortex-a15 \ >>>>>>>>>>>>>>>> -device >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> pl330,vfio_group="/dev/vfio/0",vfio_device="2c0a0000.dma",vfio_contai >>>>>>>> ner="/dev/v >>>>>>>>>>>>>>>> fio/vfio" \ >>>>>>>>>>>>>>>> -append "earlyprintk ip=dhcp console=ttyAMA0 rootwait" \ >>>>>>>>>>>>>>>> -kernel zImage -dtb guest-a15.dtb -initrd >>>>>>>>>>>>>>>> filesystem.cpio.gz >>>>>>>> - >>>>>>>>>>>>>>>> nographic But the error still exists. >>>>>>>>>>>>>>>> qemu-system-arm: -device >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,vfio_container= >>>>>>>> /dev/vfio/v >>>>>>>>>>>>>>>> fio: Parameter 'driver' expects pluggable device type >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Shannon >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2014/6/19 19:26, Alvise Rigo wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Il 19/06/2014 13:11, Shannon Zhao ha scritto: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2014/6/19 18:47, Alvise Rigo wrote: >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Il 19/06/2014 11:38, Shannon Zhao ha scritto: >>>>>>>>>>>>>>>>>>>> Hi Bharat, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2014/6/19 17:19, Bharat.Bhushan@xxxxxxxxxxxxx wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>>>>>>>> From: kvmarm-bounces@xxxxxxxxxxxxxxxxxxxxx >>>>>>>>>>>>>>>>>>>>>> [mailto:kvmarm- bounces@xxxxxxxxxxxxxxxxxxxxx] On >>>>>>>>>>>>>>>>>>>>>> Behalf Of Shannon Zhao >>>>>>>>>>>>>>>>>>>>>> Sent: Thursday, June 19, 2014 2:45 PM >>>>>>>>>>>>>>>>>>>>>> To: Alvise Rigo; a.motakis@xxxxxxxxxxxxxxxxxxxxxx; >>>>>>>>>>>>>>>>>>>>>> iommu@lists.linux- foundation.org; >>>>>>>>>>>>>>>>>>>>>> kvmarm@xxxxxxxxxxxxxxxxxxxxx >>>>>>>>>>>>>>>>>>>>>> Subject: Re: Vfio-on-arm: the device tree does not >>>>>>>>>>>>>>>>>>>>>> contain the DMA device >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Alvise, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply. >>>>>>>>>>>>>>>>>>>>>> I have apply the patch and could see the DMA devices. >>>>>>>>>>>>>>>>>>>>>> But there is no vosys_dmatest int the directory >>>>>>>>>>>>>>>>>>>>>> "/sys/kernel/debug/" after mounting debugfs. >>>>>>>>>>>>>>>>>>>>>> I found that the branch vfio-platform-v4 doesn't >>>>>>>>>>>>>>>>>>>>>> contain the module vosys_dmatest. >>>>>>>>>>>>>>>>>>>>>> Then I port it from the branch pl330_dmatest and run >>>>>>>> vosys_dmatest >>>>>>>>>>>>>>>> successfully. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> echo 1 > /sys/kernel/debug/vosys_dmatest/start >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [ 6145.343419] dmaengine: __dma_request_channel: >>>>>>>>>>>>>>>>>>>>>> success >>>>>>>>>>>>>>>> (dma0chan0) >>>>>>>>>>>>>>>>>>>>>> [ 6145.344000] vosys dmatest: callback: transfer completed >>>>>>>>>>>>>>>>>>>>>> [ 6145.344147] vosys dmatest: tx completed >>>>>>>>>>>>>>>>>>>>>> [ 6145.344589] vosys dmatest: source and destination match. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Download the qemu and use following command to start a guest: >>>>>>>>>>>>>>>>>>>>>> ./qemu-system-arm -enable-kvm -M vexpress-a15 -cpu >>>>>>>>>>>>>>>>>>>>>> cortex- >>>>>>>> a15 \ >>>>>>>>>>>>>>>>>>>>>> -device >>>>>>>>>>>>>>>> pl330,vfio_group="/dev/vfio/0",vfio_device="2c0a0000.dma",\ >>>>>>>>>>>>>>>>>>>>>> vfio_container="/dev/vfio/vfio" \ >>>>>>>>>>>>>>>>>>>>>> -append "earlyprintk ip=dhcp console=ttyAMA0 rootwait" >>>>>>>> \ >>>>>>>>>>>>>>>>>>>>>> -kernel zImage -dtb guest.dtb -initrd >>>>>>>> filesystem.cpio.gz >>>>>>>>>>>>>>>>>>>>>> - nographic An error reported: >>>>>>>>>>>>>>>>>>>>>> qemu-system-arm: -device >>>>>>>>>>>>>>>>>>>>>> pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Why do we need to give "vfio_group=/dev/vfio/0" ? >>>>>>>>>>>>>>>>>>>> I refer to the url >>>>>>>>>>>>>>>> "http://www.virtualopensystems.com/en/solutions/guides/vfio-on-arm/";. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> could not open disk image vfio_container=/dev/vfio: >>>>>>>>>>>>>>>>>>>>>> Could not open 'vfio_container=/dev/vfio': No such >>>>>>>>>>>>>>>>>>>>>> file or directory >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Are you unbinding "2c0a0000.dma" from its driver and >>>>>>>>>>>>>>>>>>>>> binding with >>>>>>>> "vfio- >>>>>>>>>>>>>>>> playform" driver ? >>>>>>>>>>>>>>>>>>>> How to unbind "2c0a0000.dma" from its driver? >>>>>>>>>>>>>>>>>>>> I have tried following cmd to bind "2c0a0000.dma" with >>>>>>>>>>>>>>>>>>>> "vfio- >>>>>>>> playform" >>>>>>>>>>>>>>>> driver. >>>>>>>>>>>>>>>>>>>> echo 2c0a0000.dma > >>>>>>>>>>>>>>>>>>>> /sys/bus/platform/drivers/vfio-platform/vfio_bind >>>>>>>>>>>>>>>>>>>> But there is no /dev/vfio/0 or /dev/vfio/vfio . >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Can you please verify that while configuring the host you enabled: >>>>>>>>>>>>>>>>>>> - Device Drivers -> IOMMU Hardware Support -> ARM Ltd. >>>>>>>>>>>>>>>>>>> System MMU >>>>>>>>>>>>>>>>>>> (SMMU) Support and >>>>>>>>>>>>>>>>>>> - Device Drivers -> VFIO Non-Privileged userspace driver >>>>>>>>>>>>>>>>>>> framework >>>>>>>>>>>>>>>>>>> -> VFIO support for platform devices >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Alvise, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have checked the two configuration are enabled. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Then try the following cmd, but another error reported: >>>>>>>>>>>>>>>>>> qemu-system-arm: -device >>>>>>>>>>>>>>>>>> pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,vfio_ >>>>>>>>>>>>>>>>>> container= >>>>>>>>>>>>>>>>>> /dev/vfio/vfio: Parameter 'driver' expects pluggable device >>>>>>>>>>>>>>>>>> type >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Do I need to unbind the "2c0a0000.dma" from its driver and >>>>>>>>>>>>>>>>>> binding >>>>>>>> with >>>>>>>>>>>>>>>> "vfio-platform" driver as Bharat mentioned ? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes, the 2c0a0000.dma should not be bound to any driver at >>>>>>>>>>>>>>>>> boot, so you just need to: >>>>>>>>>>>>>>>>> echo 2c0a0000.dma > >>>>>>>>>>>>>>>>> /sys/bus/platform/drivers/vfio-platform/vfio_bind >>>>>>>>>>>>>>>>> to bind the device to the VFIO driver. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> alvise >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Then try to update the pl330-vfio-dev branch and run QEMU >>>>>>>>>>>>>>>>>>> with the >>>>>>>>>>>>>> following >>>>>>>>>>>>>>>> command: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ./qemu-system-arm -enable-kvm -M vexpress-a15 -cpu cortex-a15 \ >>>>>>>>>>>>>>>>>>> -device >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> pl330,vfio_group="/dev/vfio/0",vfio_device="2c0a0000.dma",vfio_contai >>>>>>>> ner="/dev/v >>>>>>>>>>>>>>>> fio/vfio" \ >>>>>>>>>>>>>>>>>>> -append "earlyprintk ip=dhcp console=ttyAMA0 rootwait" \ >>>>>>>>>>>>>>>>>>> -kernel zImage -dtb guest.dtb -initrd >>>>>>>>>>>>>>>>>>> filesystem.cpio.gz >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> that means without breaking the "-device pl330" options. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> alvise >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>> -Bharat >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Is there something I'm wrong or do I miss some steps? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>>>>>> Shannon >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2014/6/18 21:24, Alvise Rigo wrote: >>>>>>>>>>>>>>>>>>>>>>> Hi Shannon, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thank you for reporting this issue, the device tree >>>>>>>>>>>>>>>>>>>>>>> file was >>>>>>>> indeed >>>>>>>>>>>>>>>> missing. >>>>>>>>>>>>>>>>>>>>>>> To solve the problem, you can apply the following >>>>>>>>>>>>>>>>>>>>>>> patch to the repository you mentioned >>>>>>>>>>>>>>>>>>>>>>> (git://github.com/virtualopensystems/arm-dts.git). >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> For your information, soon we will update the guide >>>>>>>>>>>>>>>>>>>>>>> with a newer version of VFIO with this issue fixed. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>>>>>>> alvise >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>>> fast_models/rtsm_ve-cortex_a15x4_a7x4.dts | 100 >>>>>>>>>>>>>>>>>>>>>>> +++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>>>>>>> 1 file changed, 97 insertions(+), 3 deletions(-) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> diff --git a/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts >>>>>>>>>>>>>>>>>>>>>>> b/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts >>>>>>>>>>>>>>>>>>>>>>> index ef1275b..79c6af2 100644 >>>>>>>>>>>>>>>>>>>>>>> --- a/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts >>>>>>>>>>>>>>>>>>>>>>> +++ b/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts >>>>>>>>>>>>>>>>>>>>>>> @@ -142,7 +142,7 @@ >>>>>>>>>>>>>>>>>>>>>>> cpu7: cpu@7 { >>>>>>>>>>>>>>>>>>>>>>> device_type = "cpu"; >>>>>>>>>>>>>>>>>>>>>>> compatible = "arm,cortex-a7"; >>>>>>>>>>>>>>>>>>>>>>> - reg = <0x104>; >>>>>>>>>>>>>>>>>>>>>>> + reg = <0x103>; >>>>>>>>>>>>>>>>>>>>>>> cluster = <&cluster1>; >>>>>>>>>>>>>>>>>>>>>>> core = <&core7>; >>>>>>>>>>>>>>>>>>>>>>> }; >>>>>>>>>>>>>>>>>>>>>>> @@ -159,7 +159,7 @@ >>>>>>>>>>>>>>>>>>>>>>> #address-cells = <0>; >>>>>>>>>>>>>>>>>>>>>>> interrupt-controller; >>>>>>>>>>>>>>>>>>>>>>> reg = <0 0x2c001000 0 0x1000>, >>>>>>>>>>>>>>>>>>>>>>> - <0 0x2c002000 0 0x2000>, >>>>>>>>>>>>>>>>>>>>>>> + <0 0x2c002000 0 0x1000>, >>>>>>>>>>>>>>>>>>>>>>> <0 0x2c004000 0 0x2000>, >>>>>>>>>>>>>>>>>>>>>>> <0 0x2c006000 0 0x2000>; >>>>>>>>>>>>>>>>>>>>>>> interrupts = <1 9 0xf04>; @@ -183,6 +183,101 @@ >>>>>>>>>>>>>>>>>>>>>>> <1 10 0xf08>; >>>>>>>>>>>>>>>>>>>>>>> }; >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> + smmu@2c0b0000 { >>>>>>>>>>>>>>>>>>>>>>> + compatible = "arm,smmu-v2"; >>>>>>>>>>>>>>>>>>>>>>> + reg = <0x0 0x2c0b0000 0x0 0x10000>; >>>>>>>>>>>>>>>>>>>>>>> + #global-interrupts = <2>; >>>>>>>>>>>>>>>>>>>>>>> + interrupts = <0 58 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 59 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 60 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 60 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 60 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 60 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 60 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 60 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 60 4>; >>>>>>>>>>>>>>>>>>>>>>> + mmu-masters = <&dma0 0x0 0x8>, >>>>>>>>>>>>>>>>>>>>>>> + <&dma1 0x1 0x9>, >>>>>>>>>>>>>>>>>>>>>>> + <&dma2 0x2 0xa>, >>>>>>>>>>>>>>>>>>>>>>> + <&dma3 0x3 0xb>; >>>>>>>>>>>>>>>>>>>>>>> + }; >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> + dma0: dma@2c0a0000 { >>>>>>>>>>>>>>>>>>>>>>> + compatible = "arm,pl330"; >>>>>>>>>>>>>>>>>>>>>>> + reg = <0 0x2c0a0000 0 0x1000>; >>>>>>>>>>>>>>>>>>>>>>> + interrupts = <0 64 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 65 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 66 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 67 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 68 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 69 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 70 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 71 4>; >>>>>>>>>>>>>>>>>>>>>>> + #dma-cells = <1>; >>>>>>>>>>>>>>>>>>>>>>> + #stream-id-cells = <2>; >>>>>>>>>>>>>>>>>>>>>>> + clocks = <&oscclk6a>; >>>>>>>>>>>>>>>>>>>>>>> + clock-names = "apb_pclk"; >>>>>>>>>>>>>>>>>>>>>>> + }; >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> + dma1: dma@2c0a1000 { >>>>>>>>>>>>>>>>>>>>>>> + compatible = "arm,pl330", "arm,primecell"; >>>>>>>>>>>>>>>>>>>>>>> + reg = <0 0x2c0a1000 0 0x1000>; >>>>>>>>>>>>>>>>>>>>>>> + interrupts = <0 96 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 97 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 98 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 99 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 100 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 101 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 102 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 103 4>; >>>>>>>>>>>>>>>>>>>>>>> + #dma-cells = <1>; >>>>>>>>>>>>>>>>>>>>>>> + #stream-id-cells = <2>; >>>>>>>>>>>>>>>>>>>>>>> + clocks = <&oscclk6a>; >>>>>>>>>>>>>>>>>>>>>>> + clock-names = "apb_pclk"; >>>>>>>>>>>>>>>>>>>>>>> + }; >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> + dma2: dma@2c0a2000 { >>>>>>>>>>>>>>>>>>>>>>> + compatible = "arm,pl330", "arm,primecell"; >>>>>>>>>>>>>>>>>>>>>>> + reg = <0 0x2c0a2000 0 0x1000>; >>>>>>>>>>>>>>>>>>>>>>> + interrupts = <0 128 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 129 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 130 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 131 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 132 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 133 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 134 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 135 4>; >>>>>>>>>>>>>>>>>>>>>>> + #dma-cells = <1>; >>>>>>>>>>>>>>>>>>>>>>> + #stream-id-cells = <2>; >>>>>>>>>>>>>>>>>>>>>>> + clocks = <&oscclk6a>; >>>>>>>>>>>>>>>>>>>>>>> + clock-names = "apb_pclk"; >>>>>>>>>>>>>>>>>>>>>>> + }; >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> + dma3: dma@2c0a3000 { >>>>>>>>>>>>>>>>>>>>>>> + compatible = "arm,pl330", "arm,primecell"; >>>>>>>>>>>>>>>>>>>>>>> + reg = <0 0x2c0a3000 0 0x1000>; >>>>>>>>>>>>>>>>>>>>>>> + interrupts = <0 160 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 161 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 162 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 163 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 164 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 165 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 166 4>, >>>>>>>>>>>>>>>>>>>>>>> + <0 167 4>; >>>>>>>>>>>>>>>>>>>>>>> + #dma-cells = <1>; >>>>>>>>>>>>>>>>>>>>>>> + #stream-id-cells = <2>; >>>>>>>>>>>>>>>>>>>>>>> + clocks = <&oscclk6a>; >>>>>>>>>>>>>>>>>>>>>>> + clock-names = "apb_pclk"; >>>>>>>>>>>>>>>>>>>>>>> + }; >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> + oscclk6a: oscclk6a { >>>>>>>>>>>>>>>>>>>>>>> + /* Reference 24MHz clock */ >>>>>>>>>>>>>>>>>>>>>>> + compatible = "fixed-clock"; >>>>>>>>>>>>>>>>>>>>>>> + #clock-cells = <0>; >>>>>>>>>>>>>>>>>>>>>>> + clock-frequency = <24000000>; >>>>>>>>>>>>>>>>>>>>>>> + clock-output-names = "oscclk6a"; >>>>>>>>>>>>>>>>>>>>>>> + }; >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> smb { >>>>>>>>>>>>>>>>>>>>>>> compatible = "simple-bus"; >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> @@ -244,4 +339,3 @@ >>>>>>>>>>>>>>>>>>>>>>> /include/ "rtsm_ve-motherboard.dtsi" >>>>>>>>>>>>>>>>>>>>>>> }; >>>>>>>>>>>>>>>>>>>>>>> }; >>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> Shannon >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>> kvmarm mailing list >>>>>>>>>>>>>>>>>>>>>> kvmarm@xxxxxxxxxxxxxxxxxxxxx >>>>>>>>>>>>>>>>>>>>>> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Shannon >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Shannon >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> . >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> . >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Shannon >>>>>>> >>>>>> >>>>>> -- >>>>>> Shannon >>>>> >>>> >>> >>> . >>> >> > > . > -- Shannon _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
