On Mon, Jan 13, 2025 at 08:53:40PM +0000, Jon Kohler wrote:
>
>
> > On Jan 13, 2025, at 10:36 AM, Andrew Jones <andrew.jones@xxxxxxxxx> wrote:
> >
> > !-------------------------------------------------------------------|
> > CAUTION: External Email
> >
> > |-------------------------------------------------------------------!
> >
> > On Mon, Jan 13, 2025 at 02:49:11PM +0000, Jon Kohler wrote:
> >>
> >>
> >>> On Jan 13, 2025, at 8:07 AM, Alexandru Elisei <alexandru.elisei@xxxxxxx> wrote:
> >>>
> >>> !-------------------------------------------------------------------|
> >>> CAUTION: External Email
> >>>
> >>> |-------------------------------------------------------------------!
> >>>
> >>> Hi,
> >>>
> >>> On Sun, Jan 05, 2025 at 10:57:23AM -0700, Jon Kohler wrote:
> >>>> Add a 'portable' mode that packages all relevant flat files and helper
> >>>> scripts into a tarball named 'kut-portable.tar.gz'.
> >>>>
> >>>> This mode is useful for compiling tests on one machine and running them
> >>>> on another without needing to clone the entire repository. It allows
> >>>> the runner scripts and unit test configurations to remain local to the
> >>>> machine under test.
> >>>
> >>> Have you tried make standalone? You can then copy the tests directory, or even a
> >>> particular test.
> >>
> >> Yes, standalone does not work when copying tests from one host to another. The
> >> use case for portable mode is to be able to compile within one environment and
> >> test in completely separate environment. I was not able to accomplish that with
> >> standalone mode by itself.
> >>
> >
> > standalone scripts should be portable. If they're missing something, then
> > we should fix that. Also 'make install' should include everything
> > necessary, otherwise it should be fixed. Then, we could consider adding
> > another target like 'make package' which would do 'make install' to a
> > temporary directory and tar/gzip or whatever the installation into a
> > package.
>
> Thanks, Drew. The standalone scripts are not portable in my experience, as
> in I can not just pick them up, copy them as is, and put them on another host
> with different directory layouts, etc (and importantly, no kvm-unit-tests repo
> whatsoever).
They should work without a kvm-unit-tests repo (they used to anyway). If
they're not working when moved to a host that doesn't have anything except
bash and qemu, then we should fix them. Which architecture are you mainly
testing?
>
> The make package idea is effectively what I’m doing here, but it happens to
> use the word portable instead of package (and not using make install to do
> the data movement).
I know, but I'm suggesting that we do use 'make install' rather than
introduce an independent target which should be 'make install' plus
packaging. We don't want to maintain multiple independent targets
which do almost the same thing.
>
> That bit is important:
> The biggest tidbit is that things like the errata path is hard coded, so
> in my “make portable” it fixes those things to be not hard coded, and all is well
> after files are tar’d up and then untar'd later.
So let's fix 'make install' to make sure nothing installed has hard coded
paths (which standalone should do, since it should only depend on /tmp)
>
> Also, in standalone mode, you’d be missing the runner scripts, etc, so it
> makes it harder to tweak and iterate locally on the system under test.
standalone scripts don't need the run_tests.sh since they include
everything from unittests.cfg already. It probably would make sense
to provide a script that allows running all or a subset of tests at
once, which would just be a for-loop, e.g.
run_subset() { for t in ${tests}/${1} ${tests}/${1}_*; do [ -e $t ] && $t; done; }
Running tests with different parameters than what are specified in the
unittests.cfg file isn't currently possible, unless they're also
configurable with environment variables like MACHINE, ACCEL, TIMEOUT,
and MAX_SMP. If more configuration changes are necessary, then a make
target for packaging $TEST_DIR/run and the .flat files is probably the
right way to go, but it still isn't clear to me if that's what you're
trying to do.
>
> Happy to take guidance if I’m missing something painfully obvious with the
> existing standalone setup (I felt this way when I started looking at this!) as it
> doesn’t do what I’d think it would do.
There's a good chance that standalone hasn't been maintained along with
the rest of the scripts, since we don't currently have a CI test for it.
But, unless you need to modify test parameters, then we should try to
fix it (and add CI for it).
Thanks,
drew
