From: Prarit Bhargava <prarit@xxxxxxxxxx>
process_configs.sh: Add parallel processing of configs in dist-configs
The dist-configs target takes an annoying long time.
Add parallel processing of configs in dist-configs.
Testing Info
------------
make dist-configs time 1m22.909s
make -j20 dist-configs time 11.801s
Removed redhat/configs/common/generic/x86/x86_64/CONFIG_ACPI_HOTPLUG_MEMORY
to test unset error.
>Found unset config items in x86_64-debug, please set them to an appropriate value
>CONFIG_ACPI_HOTPLUG_MEMORY=n
>Found unset config items in x86_64-kgcov, please set them to an appropriate value
>CONFIG_ACPI_HOTPLUG_MEMORY=n
>Found unset config items in x86_64, please set them to an appropriate value
>CONFIG_ACPI_HOTPLUG_MEMORY=n
Changed redhat/configs/common/generic/s390x/CONFIG_HID from 'is not set'
to '=n' to test mismatch error.
>Error: Mismatches found in configuration files for s390x
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
>Error: Mismatches found in configuration files for s390x-kgcov
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
>Error: Mismatches found in configuration files for s390x-debug
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
>Error: Mismatches found in configuration files for s390x-zfcpdump
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
>Error: Mismatches found in configuration files for s390x-debug
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
>Error: Mismatches found in configuration files for s390x-kgcov
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
>Error: Mismatches found in configuration files for s390x-zfcpdump
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
>Error: Mismatches found in configuration files for s390x
>Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source tree
Changed redhat/configs/common/generic/CONFIG_PREEMPT to '=y' to test
misconfigured error.
Found misconfigured config items in aarch64-debug, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-aarch64-debug.config.orig:4245:warning: override: PREEMPT changes choice state
Found misconfigured config items in aarch64-kgcov, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-aarch64-kgcov.config.orig:4224:warning: override: PREEMPT changes choice state
Found misconfigured config items in ppc64le-kgcov, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-ppc64le-kgcov.config.orig:4028:warning: override: PREEMPT changes choice state
Found misconfigured config items in ppc64le, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-ppc64le.config.orig:4026:warning: override: PREEMPT changes choice state
Found misconfigured config items in x86_64-debug, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-x86_64-debug.config.orig:4212:warning: override: PREEMPT changes choice state
Found misconfigured config items in x86_64-kgcov, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-x86_64-kgcov.config.orig:4192:warning: override: PREEMPT changes choice state
Found misconfigured config items in x86_64, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-x86_64.config.orig:4190:warning: override: PREEMPT changes choice state
Found misconfigured config items in aarch64, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-aarch64.config.orig:4222:warning: override: PREEMPT changes choice state
Found misconfigured config items in ppc64le-debug, please set them to an appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-ppc64le-debug.config.orig:4045:warning: override: PREEMPT changes choice state
Signed-off-by: Prarit Bhargava <prarit@xxxxxxxxxx>
diff --git a/redhat/Makefile b/redhat/Makefile
index blahblah..blahblah 100644
--- a/redhat/Makefile
+++ b/redhat/Makefile
@@ -121,12 +121,12 @@ dist-kabi-dw-check: dist-kabi
@rm -rf $(KABIDW)/base/$(CURARCH).tmp
dist-configs-commit: dist-configs-prep
- @cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh "$(FLAVOR)" 1; \
- ./process_configs.sh -z $(PACKAGE_NAME) $(KVERSION) "" $(FLAVOR)
+ +@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh "$(FLAVOR)" 1; \
+ ./process_configs.sh -z "$(PACKAGE_NAME)" "$(KVERSION)" "" $(FLAVOR)
dist-configs: dist-configs-prep
- @cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh "$(FLAVOR)" 1; \
- ./process_configs.sh $(PROCESS_CONFIGS_OPTS) $(PACKAGE_NAME) $(KVERSION)
+ +@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh "$(FLAVOR)" 1; \
+ ./process_configs.sh $(PROCESS_CONFIGS_OPTS) "$(PACKAGE_NAME)" "$(KVERSION)"
dist-fedora-configs: FLAVOR = fedora
dist-fedora-configs: dist-configs
@@ -137,11 +137,11 @@ dist-rhel-configs: dist-configs
rh-configs: dist-rhel-configs
dist-configs-check: dist-configs-prep
- cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS) $(PACKAGE_NAME)
+ +cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS) $(PACKAGE_NAME)
dist-configs-prep: dist-clean-configs dist-buildreq-check
- cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)" "snip"
- cd $(REDHAT)/configs; ./build_configs.sh "$(PACKAGE_NAME)" "$(ARCH_MACH)" "$(FLAVOR)"
+ +cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)" "snip"
+ +cd $(REDHAT)/configs; ./build_configs.sh "$(PACKAGE_NAME)" "$(ARCH_MACH)" "$(FLAVOR)"
dist-configs-arch: ARCH_MACH = $(MACH)
dist-configs-arch: dist-configs
diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -53,15 +53,25 @@ function combine_config_layer()
function merge_configs()
{
+ local archvar
+ local arch
+ local configs
+ local order
+ local flavor
+ local count
+ local name
+ local skip_if_missing
+
archvar=$1
arch=$(echo "$archvar" | cut -f1 -d"-")
configs=$2
order=$3
flavor=$4
+ count=$5
name=$OUTPUT_DIR/$PACKAGE_NAME-$archvar-$flavor.config
- echo -n "Building $name ... "
- touch config-merging config-merged
+ echo "Building $name ... "
+ touch config-merging.$count config-merged.$count
# apply based on order
skip_if_missing=""
@@ -73,10 +83,10 @@ function merge_configs()
test -n "$skip_if_missing" && test ! -e "$cfile" && continue
- if ! perl merge.pl "$cfile" config-merging > config-merged; then
+ if ! perl merge.pl "$cfile" config-merging.$count > config-merged.$count; then
die "Failed to merge $cfile"
fi
- mv config-merged config-merging
+ mv config-merged.$count config-merging.$count
done
# first configs in $order is baseline, all files should be
@@ -96,9 +106,9 @@ function merge_configs()
else
echo "# $arch" > "$name"
fi
- sort config-merging >> "$name"
- rm -f config-merged config-merging
- echo "done"
+ sort config-merging.$count >> "$name"
+ rm -f config-merged.$count config-merging.$count
+ echo "Building $name complete"
}
function build_flavor()
@@ -141,9 +151,15 @@ function build_flavor()
esac
fi
- merge_configs "$arch" "$configs" "$order" "$flavor"
+ merge_configs "$arch" "$configs" "$order" "$flavor" "$count" &
+ waitpids[$count]=$!
+ ((count++))
fi
done < "$control_file"
+
+ for pid in ${waitpids[*]}; do
+ wait $pid
+ done
}
while read -r line
diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -60,6 +60,9 @@ switch_to_toplevel()
checkoptions()
{
+ count=$3
+ variant=$4
+
/usr/bin/awk '
/is not set/ {
@@ -82,10 +85,10 @@ checkoptions()
print "Found "a[1]"="a[2]" after generation, had " a[1]"="configs[a[1]]" in Source tree";
}
}
- ' "$1" "$2" > .mismatches
+ ' "$1" "$2" > .mismatches${count}
checkoptions_error=false
- if test -s .mismatches
+ if test -s .mismatches${count}
then
while read -r LINE
do
@@ -97,14 +100,15 @@ checkoptions()
checkoptions_error=true
break
fi
- done < .mismatches
+ done < .mismatches${count}
! $checkoptions_error && return
- echo "Error: Mismatches found in configuration files"
- cat .mismatches
- RETURNCODE=1
+ sed -i "1s/^/Error: Mismatches found in configuration files for ${variant}\n/" .mismatches${count}
+ cat .mismatches${count}
[ "$CONTINUEONERROR" ] || exit 1
+ else
+ rm -f .mismatches${count}
fi
}
@@ -218,73 +222,105 @@ function commit_new_configs()
git commit -m "[redhat] AUTOMATIC: New configs"
}
-function process_configs()
+function process_config()
{
- # assume we are in $source_tree/configs, need to get to top level
- pushd "$(switch_to_toplevel)" &>/dev/null
+ local cfg
+ local arch
+ local cfgtmp
+ local cfgorig
+ local count
+ local variant
+
+ cfg=$1
+ count=$2
+ arch=$(head -1 "$cfg" | cut -b 3-)
+ variant=$(basename "$cfg" | cut -d"-" -f3- | cut -d"." -f1)
+
+ if [ "$arch" = "EMPTY" ]
+ then
+ # This arch is intentionally left blank
+ return
+ fi
- for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
- do
- arch=$(head -1 "$cfg" | cut -b 3-)
+ cfgtmp="${cfg}.tmp"
+ cfgorig="${cfg}.orig"
+ cat "$cfg" > "$cfgorig"
- if [ "$arch" = "EMPTY" ]
- then
- # This arch is intentionally left blank
- continue
- fi
+ echo "Processing $cfg ... "
- cfgtmp="${cfg}.tmp"
- cfgorig="${cfg}.orig"
- cat "$cfg" > "$cfgorig"
+ make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch) KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig${count}
+ grep -E 'CONFIG_' .listnewconfig${count} > .newoptions${count}
+ if test -n "$NEWOPTIONS" && test -s .newoptions${count}
+ then
+ echo "Found unset config items in ${variant}, please set them to an appropriate value" >> .errors${count}
+ cat .newoptions${count} >> .errors${count}
+ rm .newoptions${count}
+ [ "$CONTINUEONERROR" ] || exit 1
+ fi
+ rm .newoptions${count}
- echo -n "Processing $cfg ... "
+ grep -E 'config.*warning' .listnewconfig${count} > .warnings${count}
+ if test -n "$CHECKWARNINGS" && test -s .warnings${count}
+ then
+ echo "Found misconfigured config items in ${variant}, please set them to an appropriate value" >> .errors${count}
+ cat .warnings${count} >> .errors${count}
+ rm .warnings${count}
+ [ "$CONTINUEONERROR" ] || exit 1
+ fi
+ rm .warnings${count}
- make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch) KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig
- grep -E 'CONFIG_' .listnewconfig > .newoptions
- if test -n "$NEWOPTIONS" && test -s .newoptions
- then
- echo "Found unset config items, please set them to an appropriate value"
- cat .newoptions
- rm .newoptions
- RETURNCODE=1
- [ "$CONTINUEONERROR" ] || exit 1
- fi
- rm .newoptions
+ rm .listnewconfig${count}
- grep -E 'config.*warning' .listnewconfig > .warnings
- if test -n "$CHECKWARNINGS" && test -s .warnings
- then
- echo "Found misconfigured config items, please set them to an appropriate value"
- cat .warnings
- rm .warnings
- RETURNCODE=1
- [ "$CONTINUEONERROR" ] || exit 1
- fi
- rm .warnings
+ make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch) KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
+ echo "# $arch" > "$cfgtmp"
+ cat "$cfgorig" >> "$cfgtmp"
+ if test -n "$CHECKOPTIONS"
+ then
+ checkoptions "$cfg" "$cfgtmp" "$count" "$variant"
+ fi
+ # if test run, don't overwrite original
+ if test -n "$TESTRUN"
+ then
+ rm -f "$cfgtmp"
+ else
+ mv "$cfgtmp" "$cfg"
+ fi
+ rm -f "$cfgorig"
+ echo "Processing $cfg complete"
+}
- rm .listnewconfig
+function process_configs()
+{
+ # assume we are in $source_tree/configs, need to get to top level
+ pushd "$(switch_to_toplevel)" &>/dev/null
- make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch) KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
- echo "# $arch" > "$cfgtmp"
- cat "$cfgorig" >> "$cfgtmp"
- if test -n "$CHECKOPTIONS"
- then
- checkoptions "$cfg" "$cfgtmp"
- fi
- # if test run, don't overwrite original
- if test -n "$TESTRUN"
- then
- rm -f "$cfgtmp"
- else
- mv "$cfgtmp" "$cfg"
- fi
- rm -f "$cfgorig"
- echo "done"
+ count=0
+ for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
+ do
+ process_config "$cfg" "${count}" &
+ waitpids[${count}]=$!
+ ((count++))
+ done
+
+ for pid in ${waitpids[*]}; do
+ wait ${pid}
done
+
rm "$SCRIPT_DIR"/*.config*.old
+ if ls .errors* 1> /dev/null 2>&1; then
+ RETURNCODE=1
+ cat .errors*
+ rm .errors* -f
+ fi
+ if ls .mismatches* 1> /dev/null 2>&1; then
+ RETURNCODE=1
+ cat .mismatches*
+ rm .mismatches* -f
+ fi
+
popd > /dev/null
- echo "Processed config files are in $SCRIPT_DIR"
+ [ $RETURNCODE -eq 0 ] && echo "Processed config files are in $SCRIPT_DIR"
}
CHECKOPTIONS=""
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
_______________________________________________
kernel mailing list -- kernel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to kernel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/kernel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
