Re: Parallelization of shell scripts for 'configure' etc.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Four months ago I had created a couple of m4 macros to allow
parallelization of the configure process:

https://github.com/madmurphy/not-autotools/blob/master/m4/not-parallel-configure.m4

If you look at the source code you will find also a sample usage:

    # Start a new unnamed thread....
    NC_THREAD_NEW([
        AC_MSG_NOTICE([Starting a parallel test...])
        # Do something expensive here
        sleep 3
        # ...
        AS_VAR_SET([TEST_RESULT_1], [foo])
        AC_MSG_NOTICE([Parallel test 1 has terminated])
    ])

    # Start another thread, named this time....
    NC_THREAD_NEW([my_named_thread], [
        # Do something expensive here
        # ...
        sleep 1
        # ...
        AS_VAR_SET([TEST_RESULT_2], [bar])
        AC_MSG_NOTICE([Parallel test 2 has terminated])
    ])

    # This message will appear immediately, without waiting for the tests
    # to be completed
    AC_MSG_NOTICE([Hello world])

    # Do other unrelated things here
    # ...

    AC_MSG_NOTICE([Wait until all the threads have returned...])
    NC_JOIN_THREADS

    AC_MSG_NOTICE([The result of the first test was ${TEST_RESULT_1}])
    AC_MSG_NOTICE([The result of the second test was ${TEST_RESULT_2}])

    AC_OUTPUT

Or you can see the example above in action under
not-autotools/examples/not-parallel-configure
<https://github.com/madmurphy/not-autotools/tree/master/examples/not-parallel-configure>
.

--madmurphy

On Tue, Jun 14, 2022 at 3:00 AM Alex Ameen <alex.ameen.tx@xxxxxxxxx> wrote:

> You can try to use the `requires` toposort routine to identify "Strongly
> Connected Sub-Components", which is where I imagine you'll get the
> best results. What you'll need to watch out for is undeclared ordering
> requirements that parallelism would break.
>
> The `m4sh` and `m4sugar` source code is documented in a lot of detail. The
> manuals exclude that type of documentation because it's internal; but you
> could keep yourself occupied for at least a month or two before you ran out
> of topics to explore.
>
> On Mon, Jun 13, 2022, 8:45 PM Dale R. Worley <worley@xxxxxxxxxxxx> wrote:
>
> > Paul Eggert <eggert@xxxxxxxxxxx> writes:
> > > In many Gnu projects, the 'configure' script is the biggest barrier to
> > > building because it takes soooo long to run. Is there some way that we
> > > could improve its performance without completely reengineering it, by
> > > improving Bash so that it can parallelize 'configure' scripts?
> >
> > It seems to me that bash provides the needed tools -- "( ... ) &" lets
> > you run things in parallel.  Similarly, if you've got a lot of small
> > tasks with a complex web of dependencies, you can encode that in a
> > "makefile".
> >
> > It seems to me that the heavy work is rebuilding how "configure" scripts
> > are constructed based on which items can be run in parallel.  I've never
> > seen any "metadocumentation" that laid out how all that worked.
> >
> > Dale
> >
> >
>



[Index of Archives]     [GCC Help]     [Kernel Discussion]     [RPM Discussion]     [Red Hat Development]     [Yosemite News]     [Linux USB]     [Samba]

  Powered by Linux