Skip to content

Commit

Permalink
build_all.sh: building csources 5X faster thanks to make -j (#13300)
Browse files Browse the repository at this point in the history 
* build_all.sh: building csources 5X faster thanks to make -j
* fix for freebsd
* use OS-dependent formula to get number of logical cores
* make is an optional dependency
  • Loading branch information
@timotheecour
timotheecour authored Jan 31, 2020
1 parent adc52b0 commit 45a5c64
Showing 1 changed file with 24 additions and 4 deletions.
28 changes: 24 additions & 4 deletions build_all.sh
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#! /bin/sh

# build development version of the compiler; can be rerun safely
# build development version of the compiler; can be rerun safely.
# arguments can be passed, eg `--os freebsd`

set -u # error on undefined variables
set -e # exit on first error
Expand All @@ -10,14 +11,33 @@ echo_run(){
"$@"
}

[ -d csources ] || echo_run git clone --depth 1 https://github.com/nim-lang/csources.git
[ -d csources ] || echo_run git clone -q --depth 1 https://github.com/nim-lang/csources.git

nim_csources=bin/nim_csources

build_nim_csources_via_script(){
echo_run cd csources
echo_run sh build.sh "$@"
}

build_nim_csources(){
## avoid changing dir in case of failure
(
echo_run cd csources
echo_run sh build.sh $@
if [[ $# -ne 0 ]]; then
# some args were passed (eg: `--cpu i386`), need to call build.sh
build_nim_csources_via_script "$@"
else
# no args, use multhreaded (5X faster on 16 cores: 10s instead of 50s)
makeX=make
unamestr=$(uname)
if [ "$unamestr" = 'FreeBSD' ]; then
makeX=gmake
fi
# see https://stackoverflow.com/a/53427092/1426932
logicalCpus=$(nproc 2>/dev/null || sysctl -n hw.logicalcpu 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null)
# +1: see https://unix.stackexchange.com/questions/519092/what-is-the-logic-of-using-nproc-1-in-make-command
which $makeX && echo_run $makeX -C csources -j $(($logicalCpus + 1)) || build_nim_csources_via_script
fi
)
# keep $nim_csources in case needed to investigate bootstrap issues
# without having to rebuild from csources
Expand Down

1 comment on commit 45a5c64

@c-blake
Copy link
Contributor

@c-blake c-blake commented on 45a5c64 Feb 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a big flat list of jobs with no dependency analysis as build.sh has generally been, the make -j job server is more than one strictly needs. gmake -j is perhaps more available on Linux than bash version > 4.3 released in 2014, but also maybe not on BSDs. The latter provides another easy way to get parallel speed-up via its wait -n. E.g., the below stripes however many jobs across N cpus (N provided as $1 here):

#!/bin/bash
TMP=`mktemp -t stripen.XXXXXX`   #Could just be a fixed path if mktemp unavail
while read cmd; do
    jobs > $TMP                             #jobs | wc -l does not work
    if [ $(wc -l < $TMP) -ge $1 ]; then
        wait -n                             #Wait for 1/more jobs to finish
    fi                      # Could also exit above if $? is ever non-zero
    ( eval "$cmd" ) < /dev/null &           #Run job in a subshell in bg
done
rm -f $TMP
wait                                        #Wait for all to finish

Example usage (if above is saved in stripen and chmod'd) is like

find -type f -not -name '*.gz' -printf 'gzip -9 %P\n'  | stripen 16

Of course, the above shell logic also shows how one could do a very small C program to orchestrate striping a flat list of jobs across CPUs, and that build is already assuming a C compiler. That would be the most portable approach of all.

Please sign in to comment.