Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table locking & column preallocation #586

Merged
152 commits merged into from
Feb 6, 2023
Merged
Changes from 1 commit
Commits
Show all changes
152 commits
Select commit Hold shift + click to select a range
815ad6a
Add an AddScoresStage to the sid pipeline, and validate the scores ag…
dagardner-nv Oct 27, 2022
3a8109a
Parametarize running the test for both row and column major
dagardner-nv Oct 28, 2022
e24d4bc
Use get_meta method to ensure slixing is applied
dagardner-nv Oct 28, 2022
3761187
Parametarize the pipeline_batch_size
dagardner-nv Oct 28, 2022
feb8589
Perform cuda copy using cudf::type_dispatcher
dagardner-nv Oct 29, 2022
be4a60c
Parametarize values
dagardner-nv Oct 29, 2022
d35343a
Move compare_class_to_scores to utils.py
dagardner-nv Oct 29, 2022
6cc99ab
Add add-scores test
dagardner-nv Oct 29, 2022
f1c2636
Merge branch 'branch-22.11' into david-add-scores-bug
dagardner-nv Oct 29, 2022
ac3fda4
Concatinate the input rather that repeating it, this forces the
dagardner-nv Oct 31, 2022
4be8737
Merge branch 'david-add-scores-bug' of github.com:dagardner-nv/Morphe…
dagardner-nv Oct 31, 2022
eb1dc97
fix tests
dagardner-nv Oct 31, 2022
e1f65e6
Fix type hint on convmesg class
dagardner-nv Oct 31, 2022
81207a3
Fix offset in ConvMdg
dagardner-nv Oct 31, 2022
0edcf0a
Remove unused imports
dagardner-nv Oct 31, 2022
86317f6
Parametrize column ordering, pipeline size, and data size in filter d…
dagardner-nv Oct 31, 2022
a6dae37
Adding fixes for making sliced copies and for calling set_meta() with…
mdemoret-nv Oct 31, 2022
a35743c
FIrst pass at making a TableInfo and MutableTableInfo
mdemoret-nv Oct 31, 2022
c2be38c
Merge branch 'branch-22.11' into mdd_table-info-locking
mdemoret-nv Oct 31, 2022
4843335
Partially working
mdemoret-nv Nov 1, 2022
62f4dd3
All tests passing locally
mdemoret-nv Nov 1, 2022
61e9759
Undoing incorrectly committed changes to test_sid.py
mdemoret-nv Nov 2, 2022
2f2598b
General cleanup
mdemoret-nv Nov 2, 2022
3064543
Forgot to add imports
mdemoret-nv Nov 2, 2022
dcdd653
Fixing file source stage and ensuring the repeat option works correctly
mdemoret-nv Nov 2, 2022
3a2cf11
Fixing imports
mdemoret-nv Nov 2, 2022
a165a92
Merge branch 'branch-22.11' into mdd_table-info-locking
mdemoret-nv Nov 4, 2022
f4e9f98
Merge branch 'branch-22.11' into david-mdd_table-info-locking
dagardner-nv Nov 14, 2022
927f6b8
Remove warning about tests not having a return value, also removes co…
dagardner-nv Nov 14, 2022
a98f415
wip
dagardner-nv Nov 14, 2022
0e091a9
Merge branch 'branch-22.11' into david-mdd_table-info-locking
dagardner-nv Nov 14, 2022
215391f
Add debug logging
dagardner-nv Nov 14, 2022
91393fb
Fixed preallocation in python only exec
dagardner-nv Nov 14, 2022
81e9a30
wip
dagardner-nv Nov 15, 2022
cf9de2e
wip
dagardner-nv Nov 15, 2022
954ddd1
logging
dagardner-nv Nov 15, 2022
6432a09
Fix handling of bool string '?' as '?1' isn't supported by numpy
dagardner-nv Nov 15, 2022
7c7d28c
wip
dagardner-nv Nov 15, 2022
994f233
Clean up includes
dagardner-nv Nov 15, 2022
5e369c7
Handle multi messages
dagardner-nv Nov 15, 2022
afcabae
Use an ordered dict
dagardner-nv Nov 15, 2022
6052d41
Use an ordered dict and remove debug logs
dagardner-nv Nov 15, 2022
3a957f2
PreallocateStage is now a template class allowing support for both Me…
dagardner-nv Nov 15, 2022
95ecb35
Add some doc strings
dagardner-nv Nov 15, 2022
21cc146
Perform column pre-allocation on a per-segment basis
dagardner-nv Nov 15, 2022
f011732
Change type string to one supported by Morpheus' DType class
dagardner-nv Nov 15, 2022
39a369f
Update ConvMsg to set a specific type for probs
dagardner-nv Nov 15, 2022
601a17c
Merge branch 'branch-22.11' into david-mdd_table-info-locking
dagardner-nv Nov 15, 2022
4bb82db
Merge branch 'branch-22.11' into mdd_table-info-locking
mdemoret-nv Nov 15, 2022
411a91c
Move 'Building Segment' logs to the inner_build so that they corelate…
dagardner-nv Nov 15, 2022
ebdbf01
Use the C++ impl of Preallocate if we can even if the souce stage doe…
dagardner-nv Nov 15, 2022
028d2ff
Merge branch 'mdd_table-info-locking' into david-mdd_table-info-locking
dagardner-nv Nov 15, 2022
caba4a2
Add docstring for probs_type
dagardner-nv Nov 15, 2022
dc5c973
Remove wip prints
dagardner-nv Nov 15, 2022
bccfb43
Fixing merge issue
mdemoret-nv Nov 16, 2022
ddf5e6a
Return the boundary egress/ingress nodes that were created
dagardner-nv Nov 16, 2022
ecb890d
Tests for preallocation feature
dagardner-nv Nov 16, 2022
541b67a
Group the docstring to apply to both overloads
dagardner-nv Nov 16, 2022
6abe441
Move preallocation logic to a mixin class
dagardner-nv Nov 16, 2022
43992fd
Add a docstring for the class and re-introduce support for DataFrames
dagardner-nv Nov 16, 2022
e95b879
wip
dagardner-nv Nov 17, 2022
5432ab6
Move inheritance order
dagardner-nv Nov 17, 2022
ba5ec3b
Add PreallocatorMixin class to the other sources
dagardner-nv Nov 17, 2022
ce47522
Replace needed_columns property with get_needed_columns & set_needed_…
dagardner-nv Nov 17, 2022
22ff5a2
Add some docstrings
dagardner-nv Nov 17, 2022
a487f37
Pre-instantiate PreallocateStage & PreallocateStageInterfaceProxy for…
dagardner-nv Nov 17, 2022
4d4aa1d
wip
dagardner-nv Nov 17, 2022
4acb7ac
Use a single vector of tuples rather than two vectors
dagardner-nv Nov 17, 2022
0df54ad
Merge branch 'mdd_table-info-locking' into david-mdd_table-info-locking
dagardner-nv Nov 17, 2022
2425dc7
Tighten up the set_meta method
dagardner-nv Nov 17, 2022
3793491
Catch any potential exceptions encountered fetching the column, and t…
dagardner-nv Nov 17, 2022
280e0c3
Move error handling to get_meta which is where the exception is throw…
dagardner-nv Nov 17, 2022
b099550
wip
dagardner-nv Nov 17, 2022
11de2e5
wip
dagardner-nv Nov 17, 2022
5e32eae
Revert "wip"
dagardner-nv Nov 18, 2022
84a265e
Update the real world phishing example to reflect that the returned d…
dagardner-nv Nov 18, 2022
f7ec0db
Merge branch 'david-mdd_table-info-locking-mixin' into david-mdd_tabl…
dagardner-nv Nov 18, 2022
2a8b6c8
Add PreallocatorMixin to NLPVizFileSource
dagardner-nv Nov 18, 2022
48927ca
Merge DataType & DType classes, rename type_util.cu to type_util.cpp …
dagardner-nv Nov 21, 2022
f2d4381
Rename type_util to dtype
dagardner-nv Nov 21, 2022
b0ff09b
Replace make_str_to_type_id with an initializer list, make str_to_typ…
dagardner-nv Nov 21, 2022
6280c0f
Expose TypeId enum to python
dagardner-nv Nov 21, 2022
792ee37
Use TypeId enum on python side to specify needed column types
dagardner-nv Nov 21, 2022
a369c8a
Remove unused function
dagardner-nv Nov 22, 2022
7434bb5
Use cudf.Scalar to define the new columns instead of cupy.zeros as th…
dagardner-nv Nov 22, 2022
5e07b46
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Dec 13, 2022
9e1f184
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Dec 29, 2022
fe5d394
Fix missed srf ref
dagardner-nv Dec 29, 2022
9cf28d0
srf->mrc
dagardner-nv Dec 29, 2022
aa5e26a
Remove reundant definition of size_in_bits and fix doc group
dagardner-nv Dec 29, 2022
cb95bef
Fix merge error
dagardner-nv Dec 29, 2022
51765c1
srf->mrc
dagardner-nv Dec 29, 2022
c430ecb
First pass at a context manager around mutating the dataframe
dagardner-nv Dec 29, 2022
9f4ec9d
First pass at exposing MutableTableInfo to python as a context manage…
dagardner-nv Dec 29, 2022
db966bb
wip
dagardner-nv Dec 30, 2022
1b81dec
Define explicit constructors for MessageMeta and subclasses, dataclas…
dagardner-nv Dec 30, 2022
0adf557
Update MultiMessage to use the mutable_dataframe ctx manager and _df …
dagardner-nv Dec 30, 2022
69a4781
Update preallocate mixin to use mutable_dataframe ctx manager
dagardner-nv Dec 30, 2022
ada52cb
Unittest for new MessageMeta functionality
dagardner-nv Dec 30, 2022
6c9d8ac
Fixes a problem where `with` blocks don't have their own scope, and t…
dagardner-nv Dec 31, 2022
4fe3b68
Both MutableTableInfo and pybind11::object are now unique_ptrs, allow…
dagardner-nv Jan 3, 2023
4ac1dc8
Update to mutable dataframe changes
dagardner-nv Jan 3, 2023
50f2e00
Mark any test that executes a pipeline as slow
dagardner-nv Jan 3, 2023
6d83cb4
Ignore our own warning about the df property when runnin tests
dagardner-nv Jan 3, 2023
59bd99e
Manipulate the dataframe in-place using the new mutable_dataframe con…
dagardner-nv Jan 3, 2023
f6b0183
Add a comment explaining the usage of the _needed_columns attribute
dagardner-nv Jan 3, 2023
a7af011
Use the _sep_token attribute
dagardner-nv Jan 3, 2023
0f09ed5
Update docs to reflect the new preallocate and mutable_dataframe changes
dagardner-nv Jan 3, 2023
da76b60
Restore _preallocate_df method
dagardner-nv Jan 3, 2023
008309c
Document PreallocatorMixin and fix type-os
dagardner-nv Jan 3, 2023
24c9999
punctuation
dagardner-nv Jan 3, 2023
2bc98ea
Add blank line
dagardner-nv Jan 3, 2023
b5ca5bb
Update copyright year
dagardner-nv Jan 4, 2023
0db9003
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Jan 4, 2023
4c0c3eb
Add MORPHEUS_CONTAINER as an arg
dagardner-nv Jan 4, 2023
caa432d
Update CR year
dagardner-nv Jan 4, 2023
1af8a82
Fix DFP example due to recent changes
dagardner-nv Jan 4, 2023
3089088
Formatting fixes
dagardner-nv Jan 4, 2023
4677c06
Remove unused import
dagardner-nv Jan 4, 2023
3c803ef
Remove unused imports
dagardner-nv Jan 4, 2023
df9ae01
Formatting fixes
dagardner-nv Jan 4, 2023
026dbcd
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Jan 6, 2023
bb7cd0c
Move TypeId & tyepid_to_numpy_str into the common module
dagardner-nv Jan 19, 2023
6b0f0be
Rename MutableCtxMgr --> MutableTableCtxMgr per PR feedback
dagardner-nv Jan 19, 2023
b68925c
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Jan 19, 2023
804ad30
Don't call get_mutable_info until the __enter__ method
dagardner-nv Jan 19, 2023
32a887b
Update to return the dataframe from the enter method of the context m…
dagardner-nv Jan 20, 2023
a16fa52
Move MutableTableCtxMgr to its own compilation unit
dagardner-nv Jan 20, 2023
6f6a369
Add mutable_table_ctx_mgr.cpp to cmake
dagardner-nv Jan 20, 2023
cea873a
Update docs
dagardner-nv Jan 20, 2023
340363e
MutableTableCtxMgr moved to separate compilation unit
dagardner-nv Jan 21, 2023
31484f4
py::attribute_error is probably a better error than a runtime
dagardner-nv Jan 21, 2023
bc44913
MutableTableCtxMgr moved to separate compilation unit
dagardner-nv Jan 21, 2023
550ebd5
Update python to match cpp changes
dagardner-nv Jan 21, 2023
6a98a61
More tests
dagardner-nv Jan 21, 2023
1d665b6
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Jan 21, 2023
9a30f5b
Fix CR year
dagardner-nv Jan 21, 2023
6ff759c
Add CR header
dagardner-nv Jan 21, 2023
ced00b8
Merge branch 'david-mdd_table-info-locking' of github.com:dagardner-n…
dagardner-nv Jan 21, 2023
29ac4ae
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Jan 25, 2023
d1e44c3
Fix merge issues
dagardner-nv Jan 25, 2023
d0dabbd
Fix merge errors
dagardner-nv Jan 25, 2023
fc10fd1
fix_all.sh updates
dagardner-nv Jan 26, 2023
7594eda
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Jan 26, 2023
ef879ec
Merge branch 'branch-23.01' into david-mdd_table-info-locking
dagardner-nv Jan 27, 2023
e4622a1
Merge branch 'branch-23.03' into david-mdd_table-info-locking
dagardner-nv Jan 31, 2023
4faa7ca
Merge branch 'branch-23.03' into david-mdd_table-info-locking
dagardner-nv Feb 1, 2023
64b98de
Merge branch 'branch-23.03' into david-mdd_table-info-locking
dagardner-nv Feb 1, 2023
6d972c9
Merge branch 'branch-23.03' into david-mdd_table-info-locking
dagardner-nv Feb 6, 2023
bd553ca
Merge branch 'david-mdd_table-info-locking' of github.com:dagardner-n…
dagardner-nv Feb 6, 2023
2bf0059
Remove redundant import
dagardner-nv Feb 6, 2023
678bc71
Merge branch 'branch-23.03' into david-mdd_table-info-locking
dagardner-nv Feb 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add MORPHEUS_CONTAINER as an arg
  • Loading branch information
dagardner-nv committed Jan 4, 2023
commit 4c0c3eb11bf6b53611c88b50eaed0bd9745a8007
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ services:
dockerfile: ./Dockerfile
target: jupyter
args:
- MORPHEUS_CONTAINER=${MORPHEUS_CONTAINER:-nvcr.io/nvidia/morpheus/morpheus}
- MORPHEUS_CONTAINER_VERSION=${MORPHEUS_CONTAINER_VERSION:-v23.01.00-runtime}
deploy:
resources:
Expand Down Expand Up @@ -71,6 +72,7 @@ services:
dockerfile: ./Dockerfile
target: runtime
args:
- MORPHEUS_CONTAINER=${MORPHEUS_CONTAINER:-nvcr.io/nvidia/morpheus/morpheus}
- MORPHEUS_CONTAINER_VERSION=${MORPHEUS_CONTAINER_VERSION:-v23.01.00-runtime}
image: dfp_morpheus
container_name: morpheus_pipeline
Expand Down