Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ddlog update #376

Merged
merged 5 commits into from
Sep 24, 2015
Merged

Ddlog update #376

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e5958de
DDlog update - incremental function call
feiranwang Sep 1, 2015
0b387b3
Add support for autoset parallelism for extractors in ddlog
feiranwang Sep 4, 2015
70e8342
Add support for command: deepdive initdb TABLE
feiranwang Sep 4, 2015
05852e4
Refactor setting PARALLELISM
feiranwang Sep 20, 2015
2bec633
Refactor deepdive-initdb
feiranwang Sep 20, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add support for autoset parallelism for extractors in ddlog
  • Loading branch information
@feiranwang
feiranwang committed Sep 4, 2015
commit 0b387b388a2d182e0b4e26f4bff3ba7ee1f74864
2 changes: 1 addition & 1 deletion ddlog
Open in desktop
Submodule ddlog updated 10 files
+2 −0 src/main/scala/org/deepdive/ddlog/DeepDiveLogCompiler.scala
+4 −5 src/main/scala/org/deepdive/ddlog/DeepDiveLogParser.scala
+2 −1 src/main/scala/org/deepdive/ddlog/DeepDiveLogPrettyPrinter.scala
+4 −2 test/expected-output-test/chunking_example/compile.expected
+3 −0 test/expected-output-test/spouse_example/compile-incremental.expected
+3 −0 test/expected-output-test/spouse_example/compile-materialization.expected
+3 −0 test/expected-output-test/spouse_example/compile.expected
+3 −0 test/expected-output-test/spouse_example_new_feature/compile-incremental.expected
+3 −0 test/expected-output-test/spouse_example_new_inference/compile-incremental.expected
+3 −0 test/expected-output-test/spouse_example_new_inference/compile.expected
1 change: 1 addition & 0 deletions examples/spouse_example/postgres/incremental/env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ export PGUSER=${PGUSER:-`whoami`}
export PGPASSWORD=${PGPASSWORD:-}
export PGPORT=${PGPORT:-5432}
export PGHOST=${PGHOST:-localhost}
export PARALLELISM=1
9 changes: 9 additions & 0 deletions shell/deepdive-run
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,15 @@ fullConfig=$run_dir/deepdive.conf
ddlog compile "${ddlogFiles[@]}"
export PIPELINE= # XXX ddlog shouldn't emit this
: ${Pipeline:=endtoend}

# set PARALLELISM env var, use max parallelism if the variable is not set
if [[ $(uname) = 'Linux' ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking if nproc or sysctl is available makes more sense than relying on uname. You could chain the options with something like:

: ${PARALLELISM:=$({
    # Linux typically has coreutils which includes nproc
    nproc ||
    # OS X
    sysctl -n hw.ncpu ||
    # fall back to 1
    echo 1
} 2>/dev/null)}

parallelism=$(nproc)
else
parallelism=$(sysctl -n hw.ncpu)
fi
: ${PARALLELISM:=$parallelism}
export PARALLELISM
}

# user's deepdive.conf overrides anything compiled from DDlog
Expand Down