Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add source & stage decorators #1364

Merged
merged 116 commits into from
Nov 29, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
7fd1c4e
Remove unused method
dagardner-nv Nov 3, 2023
fcac874
Update out of date comments and variable names
dagardner-nv Nov 3, 2023
19acdf0
Don't run milvus tests by default, running 'pytest --run_slow' now ta…
dagardner-nv Oct 17, 2023
8e82ba3
Consolidate milvus fixtures
dagardner-nv Oct 17, 2023
5c9379b
Set an explicit storage dir for milvus, ensuring that we always get a…
dagardner-nv Oct 17, 2023
5513dad
Defining MORPHEUS_MILVUS_URI env var avoids milvus startup, milvus_se…
dagardner-nv Oct 18, 2023
e849600
Add _get_random_port method
dagardner-nv Nov 3, 2023
2c48a07
Use new tuple type-hint
dagardner-nv Nov 4, 2023
f79475a
Update code snippets, todo: Updating text
dagardner-nv Nov 4, 2023
bbde0a8
Update to reflect recent changes [no ci]
dagardner-nv Nov 6, 2023
e906235
Use newer tuple type hint
dagardner-nv Nov 6, 2023
42e1388
Update docs for RecipientFeaturesStage
dagardner-nv Nov 6, 2023
4c49102
wip [no ci]
dagardner-nv Nov 6, 2023
65ae66e
wip [no ci]
dagardner-nv Nov 6, 2023
e289018
wip [no ci]
dagardner-nv Nov 6, 2023
9d71d75
Set log level to INFO to ensure monitor output [no ci]
dagardner-nv Nov 6, 2023
efedc35
wip [no ci]
dagardner-nv Nov 6, 2023
077af27
Fix long-standing bug in pass-thru example, the C++ impl only works w…
dagardner-nv Nov 6, 2023
04f2396
Update the example to include two instances of the PassThruStage, the…
dagardner-nv Nov 6, 2023
a154a04
WIP [no ci]
dagardner-nv Nov 6, 2023
f65d8b4
Remove uneeded return statement
dagardner-nv Nov 6, 2023
fa7738c
WIP [no ci]
dagardner-nv Nov 6, 2023
e776419
WIP [no ci]
dagardner-nv Nov 6, 2023
ff76387
No need to rename the stage
dagardner-nv Nov 6, 2023
d2ee5a7
WIP [no ci]
dagardner-nv Nov 6, 2023
e9b7d44
Remove pylint directive from docs [no ci]
dagardner-nv Nov 6, 2023
70d16a5
Remove unused imports and formatting [no ci]
dagardner-nv Nov 6, 2023
480c86c
formatting [no ci]
dagardner-nv Nov 6, 2023
3e91fb7
WIP [no ci]
dagardner-nv Nov 6, 2023
27135a9
Formatting [no ci]
dagardner-nv Nov 7, 2023
442da9b
WIP [no ci]
dagardner-nv Nov 7, 2023
73b3797
WIP
dagardner-nv Nov 7, 2023
4d74e25
WIP [no ci]
dagardner-nv Nov 7, 2023
e090091
Ensure the module instance always receives a unique name by prependin…
dagardner-nv Nov 7, 2023
fcc8844
Add missing module config to builder.load_module calls, update module…
dagardner-nv Nov 7, 2023
affa261
Update to use a simple df of floats
dagardner-nv Nov 7, 2023
e3546ff
wip: [no ci]
dagardner-nv Nov 7, 2023
5231c81
Formatting [no ci]
dagardner-nv Nov 7, 2023
15fad5a
formatting [no ci]
dagardner-nv Nov 7, 2023
b6bd48f
formatting [no ci]
dagardner-nv Nov 7, 2023
d5f9201
Update code snippets to match api changes [no ci]
dagardner-nv Nov 7, 2023
50293db
Add end-to-end test for the code snippets used in the python_modules …
dagardner-nv Nov 7, 2023
262800f
Formatting [no ci]
dagardner-nv Nov 7, 2023
9bf6508
Make task names lower-case to be consistent [no ci]
dagardner-nv Nov 7, 2023
8d3336d
Rename out_streams to out_nodes
dagardner-nv Nov 7, 2023
b9290af
Fix path to renamed script
dagardner-nv Nov 7, 2023
57ecb18
Fix import order
dagardner-nv Nov 7, 2023
e1d75a7
First pass at a stage decorator
dagardner-nv Nov 14, 2023
7dab9f0
Add source decorator
dagardner-nv Nov 15, 2023
aef5ece
Sort imports
dagardner-nv Nov 15, 2023
5d5bcfc
WIP
dagardner-nv Nov 15, 2023
5ef9326
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Nov 15, 2023
250c330
Docstring cleanups [no ci]
dagardner-nv Nov 15, 2023
1ecaa96
WIP [no ci]
dagardner-nv Nov 16, 2023
37d7926
WIP docstrings [no ci]
dagardner-nv Nov 16, 2023
d3699c2
WIP docstrings [no ci]
dagardner-nv Nov 16, 2023
5155dcf
First pass at source decorator [no ci]
dagardner-nv Nov 16, 2023
83f290a
lint cleanups [no ci]
dagardner-nv Nov 16, 2023
45725b5
Fix annotations [no ci]
dagardner-nv Nov 16, 2023
78c3ec3
Disable pylint warnings [no ci]
dagardner-nv Nov 16, 2023
d15f7dc
Fix handling of *args, and more tests [no ci]
dagardner-nv Nov 17, 2023
a430eff
wip [no ci]
dagardner-nv Nov 17, 2023
a7e35db
WIP: [no ci]
dagardner-nv Nov 17, 2023
ab16c49
wip [no ci]
dagardner-nv Nov 17, 2023
113e642
wip [no ci]
dagardner-nv Nov 17, 2023
618444a
wip [no ci]
dagardner-nv Nov 17, 2023
97f7dcd
wip [no ci]
dagardner-nv Nov 17, 2023
6e5cd52
Formatting fixes
dagardner-nv Nov 17, 2023
943e8b7
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Nov 17, 2023
b1f783f
Expose the needed_columns functionality to the stage decorator
dagardner-nv Nov 20, 2023
054a150
wip
dagardner-nv Nov 20, 2023
9a08075
wip
dagardner-nv Nov 20, 2023
44a2c70
Expose the needed_columns functionality to the stage decorator
dagardner-nv Nov 20, 2023
a317299
Add decorated version of recipient_features_stage
dagardner-nv Nov 21, 2023
90acd43
Optionally use the decorator version of the recipient features stage
dagardner-nv Nov 21, 2023
749fa09
Bind default values for keyword arguments to method
dagardner-nv Nov 21, 2023
ee3ae31
Remove binding the keyword argument
dagardner-nv Nov 21, 2023
46c69f5
Test default values for keyword args
dagardner-nv Nov 21, 2023
5763504
Merge branch 'david-stage-decorator' into david-stage-decorator-docs
dagardner-nv Nov 21, 2023
af67576
Make sep_token an explicit keyword only argument
dagardner-nv Nov 21, 2023
f92d44b
Add test for the decorated version of the pass-thru stage
dagardner-nv Nov 21, 2023
a0bbdac
Expand tests to test the decorated version of the recipient features …
dagardner-nv Nov 21, 2023
6e5d83a
Document stage decorator
dagardner-nv Nov 21, 2023
d1dd3ae
Expose the other options as command line flags
dagardner-nv Nov 21, 2023
d08b9d0
WIP
dagardner-nv Nov 21, 2023
824a59a
Rename use_decorator to use_stage_function
dagardner-nv Nov 21, 2023
ce4286b
Remove unused import
dagardner-nv Nov 22, 2023
d321771
Merge branch 'branch-23.11' into david-stage-decorator
dagardner-nv Nov 22, 2023
7163dba
Merge branch 'branch-23.11' into david-23.11-docs-1252
dagardner-nv Nov 22, 2023
c708b4c
Merge branch 'branch-23.11' into david-stage-decorator
dagardner-nv Nov 22, 2023
bbaaa20
Merge branch 'david-stage-decorator' into david-stage-decorator-docs
dagardner-nv Nov 22, 2023
979cc03
Document that C++ impls are available to classes only
dagardner-nv Nov 22, 2023
0c356d6
Merge branch 'david-23.11-docs-1252' into david-stage-decorator-docs
dagardner-nv Nov 22, 2023
6b577b1
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Nov 22, 2023
1b6ea45
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Nov 27, 2023
7b72d03
Merge branch 'david-stage-decorator' into david-stage-decorator-docs …
dagardner-nv Nov 27, 2023
3420efa
Make type annotations a requirement [no ci]
dagardner-nv Nov 27, 2023
9989a79
wip [no ci]
dagardner-nv Nov 27, 2023
b16a25e
wip [no ci]
dagardner-nv Nov 27, 2023
48a22fb
wip [no ci]
dagardner-nv Nov 27, 2023
fe0f7a4
Handle return type of typing.Any [no ci]
dagardner-nv Nov 27, 2023
4d11732
Cleanups [no ci]
dagardner-nv Nov 27, 2023
b4b4a20
Allow config to be specified positionally [no ci]
dagardner-nv Nov 27, 2023
762501f
Allow config to be a positional argument
dagardner-nv Nov 27, 2023
6907139
Update tests [no ci]
dagardner-nv Nov 27, 2023
73ba7a0
WIP: Updating tests [no ci]
dagardner-nv Nov 28, 2023
fac9826
Fill out tests
dagardner-nv Nov 28, 2023
77f67c5
linting fixes
dagardner-nv Nov 28, 2023
02983e8
linting fixes
dagardner-nv Nov 28, 2023
ee10ad0
Merge branch 'branch-23.11' into david-stage-decorator
dagardner-nv Nov 28, 2023
650cee0
Updates for developer guide
dagardner-nv Nov 28, 2023
d18ea97
misc cleanups
dagardner-nv Nov 28, 2023
e6c1407
Document usage of the source decorator [no ci]
dagardner-nv Nov 28, 2023
0787b9b
document the name argument
dagardner-nv Nov 28, 2023
739f887
Remove explicit name, so that when we run the pipeline with both pass…
dagardner-nv Nov 28, 2023
cc7cb00
lint fix
dagardner-nv Nov 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Consolidate milvus fixtures
  • Loading branch information
dagardner-nv committed Nov 3, 2023
commit 8e82ba30bfaef79dbcd78dc0d13c6dfb929605e2
7 changes: 7 additions & 0 deletions tests/_utils/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,3 +167,10 @@ def remove_module(mod_to_remove: str):
for mod_name in list(sys.modules.keys()):
if mod_name == mod_to_remove or mod_name.startswith(mod_prefix):
del sys.modules[mod_name]


def load_json_file(filename):
filepath = os.path.join(TEST_DIRS.tests_data_dir, filename)

with open(filepath, 'r', encoding="utf-8") as json_file:
return json.load(json_file)
25 changes: 25 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -976,3 +976,28 @@ def milvus_server_uri():
default_server.stop()
except Exception as exec_inf:
logger.error("Error in stopping Milvus server: %s", exec_inf)


@pytest.fixture(scope="function", name="milvus_service")
def milvus_service_fixture(milvus_server_uri: str):
from morpheus.service.milvus_vector_db_service import MilvusVectorDBService
service = MilvusVectorDBService(uri=milvus_server_uri)
yield service


@pytest.fixture(scope="session", name="milvus_data")
def milvus_data_fixture():
inital_data = [{"id": i, "embedding": [i / 10.0] * 10, "age": 25 + i} for i in range(10)]
yield inital_data


@pytest.fixture(scope="session", name="idx_part_collection_config")
def idx_part_collection_config_fixture():
from _utils import load_json_file
yield load_json_file(filename="service/milvus_idx_part_collection_conf.json")


@pytest.fixture(scope="session", name="simple_collection_config")
def simple_collection_config_fixture():
from _utils import load_json_file
yield load_json_file(filename="service/milvus_simple_collection_conf.json")
Loading