Manual testing of Morpheus with Kafka & Validation improvements #290
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dagardner-nv
added
non-breaking
…SV output, due to a known issue in cudf & pandas (rapidsai/cudf#11317 & pandas-dev/pandas#37600) this option has no effect on JSON output
… modules in morpheus.utils to import the std logger
…s() will return non-None but diff_rows=0
|
LGTM per our discussion @dagardner-nv |
…, and add cleanup instructions
dagardner-nv
changed the title
Merged
mdemoret-nv
requested changes
Aug 1, 2022
Done |
|
rerun tests |
pdmack
approved these changes
Aug 2, 2022
There was a problem hiding this comment.
LGTM
If I'm being honest there is a LOT of rote details here that are somewhat simpler in Kubernetes. But it's nice to have the comprehensive detail for anyone looking for it.
I do think there are sometimes confusing points between the spectrum of CONTRIBUTING.md and the new kafka_testing.md as to whether commands are issued inside a container or in the conda env. Potentially a Good First Issue for an external contributor to weave in a bit more clarity.
It may be worth adding a comment about a low fd ulimit (1024) impacting morpheus/librdkafka.
mdemoret-nv
approved these changes
Aug 8, 2022
|
@gpucibot merge |
ghost
merged commit
ghost
pushed a commit
that referenced
this pull request
Sep 7, 2022
Breaking changes (#296): * Default `group_id` for `KafkaSourceStage` changes from "custreamz" to "morpheus" * Remove default topic value for `KafkaSourceStage`. Other changes: * Adds Kafka integration tests for ABP, Hammah, Phishing & Sid validation pipelines. * Uses pytest_kafka to add zookeeper & kafka pytest fixtures * Configures pytest to only run kafka tests when `--run_kafka` command line flag is set. * Adds a `stop_after` argument to `KafkaSourceStage` when defined the source will stop once `stop_after` records have been read from the source. * Fixes bug in C++ impl preventing KafkaSourceStage from respecting max_batch_size (#309) Note: Many of the kafka tests are also slower integration tests in effect requiring both the `--run_kafka` & `--run_slow` flags being passed to `pytest` requires PR #290 fixes #258 fixes #296 fixes #309 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #308
ghost
pushed a commit
that referenced
this pull request
Nov 14, 2022
…#422) * Address feedback from #290 * Add note about automated Kafka tests, which didn't exist at time the document was first written * Fix spelling errors Fixes #322 Authors: - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) - Pete MacKinnon (https://github.com/pdmack) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Pete MacKinnon (https://github.com/pdmack) URL: #422
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
scripts/validationinclude_index_colflag to control exporting the Dataframe's index column. Note due to a limitation of cudf & pandas this has no impact on JSON:morpheus.utils.loggingrenamed tomorpheus.utils.loggerso that other modules inmorpheus.utilscan import the standard lib logging module.ValidationStagehas been moved to it's own modulemorpheus.utils.compare_dfso that the functionality can be used outside of the stage.fixes #265