Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding HTTP sources & sinks #977

Merged
merged 168 commits into from
Aug 24, 2023
Merged

Adding HTTP sources & sinks #977

merged 168 commits into from
Aug 24, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Jun 8, 2023 •
edited
Loading

Description

  • Adds four new stages ([FEA]: Create REST Source/Sink Stages #864):
    1. HttpServerSourceStage - Spins up a rest server to listen for incoming POSTS
    2. HttpClientSourceStage - Source that polls a remote HTTP server for incoming data
    3. HttpClientSinkStage - Sink that POSTS data to a remote HTTP server
    4. HttpServerSinkStage - Sink that statrs a rest server listening for GET requests
  • Fix TensorIndex usage for the stop_after feature of the KafkaSourceStage

fixes #864

Currently HttpServerSourceStage is the only one of the four with a C++ impl. The rest server itself is written in C++ with Python bindings and is used by both C++ & Python stages.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

dagardner-nv added 30 commits May 23, 2023 15:13
@dagardner-nv
POC: using the fmt lib for text formatting, doesn't belong in TestTyp…
12bc307 
…eUtils
@dagardner-nv
POC: Rough drafts
f58e549
@dagardner-nv
wip
313b211
@dagardner-nv
Add from-rest to cli
0e4fc40
@dagardner-nv
wip
d266015
@dagardner-nv
Add todo comments
03fe9a7
@dagardner-nv
wip
0abc244
@dagardner-nv
WIP: Proof of concept C++ impl
cc370c9
@dagardner-nv
WIP: Proof of concept C++ impl
0b2609c
@dagardner-nv
Support propper json arrays of objects as a config param
de57f49
@dagardner-nv
wip
ec3fe8e
@dagardner-nv
Replace queue with boost fiber buffered_channel
dfe1213
@dagardner-nv
Pass a parsing function into rest server, keeping queueing logic and …
47e4067 
…cudf logic outside of server, but allowing the rest server to return an accurate http response
@dagardner-nv
Switch to using a a fiber buffered channel for queueing, pass a paylo…
36262b0 
…ad parser to the service

Clean up includes
@dagardner-nv
Add Python bindings for RestServer
cdaf7aa
@dagardner-nv
Remove python webserver
6087ce9
@dagardner-nv
Put should raise full not empty when fulll, and will never receive an…
af031f7 
… empty status
@dagardner-nv
Use the C++ rest server impl when in Python mode
ca38eac
@dagardner-nv
wip
4964920
@dagardner-nv
Set a timeout for pushing to the queue
d62d956
@dagardner-nv
WIP: async refactor
e6f106b
@dagardner-nv
WIP: async refactor
d603048
@dagardner-nv
Pass num_threads, cleanup includes
7392822
@dagardner-nv
Expose max payload and request timeouts to the stage
9301315
@dagardner-nv
Fix docstring
722565f
@dagardner-nv
First pass at a polling rest client source, WIP: untested
e42646d
@dagardner-nv
misc cleanups
14c5bce
@dagardner-nv
wip
1eb222b
@dagardner-nv
Use urllib3's Retry class instead of home-grown logic, use a request …
4099f26 
…session
@dagardner-nv
wip
17df1dd
@dagardner-nv dagardner-nv mentioned this pull request Aug 14, 2023
Closed
2 tasks
@cwharris
Copy link
Contributor

cwharris commented Aug 23, 2023 •
edited
Loading

I was thinking we could consider the following names.

RestSourceStage       -> HttpHookSourceStage
RestClientSourceStage -> HttpRequestSourceStage
WriteToRestStage      -> HttpRequestSinkStage
RestServerSinkStage   -> HttpHookSinkStage

"Hook" meaning "I'm awaiting a request" and "Request" meaning "I'm performing a request"

I know HttpRequestSinkStage differs from "WriteTo*", but it's very explicit about what it's doing, and creates good uniformity between the different HTTP-related stages.

edit:

After discussing, we landed on

RestSourceStage       -> HttpServerSourceStage
RestClientSourceStage -> HttpClientSourceStage
WriteToRestStage      -> HttpClientSinkStage
RestServerSinkStage   -> HttpServerSinkStage

mdemoret-nv
mdemoret-nv approved these changes Aug 23, 2023
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets make an issue to add C++ implementations using the REST client loader

morpheus/stages/input/rest_source_stage.py Outdated Show resolved Hide resolved
morpheus/stages/input/rest_source_stage.py Outdated Show resolved Hide resolved
morpheus/stages/output/rest_server_sink_stage.py Outdated Show resolved Hide resolved
morpheus/stages/output/rest_server_sink_stage.py Outdated Show resolved Hide resolved
tests/test_rest_client_source_stage_pipe.py Outdated Show resolved Hide resolved
@dagardner-nv
Copy link
Contributor Author

Lets make an issue to add C++ implementations using the REST client loader

Done #1150

@dagardner-nv
Copy link
Contributor Author

/merge

@dagardner-nv dagardner-nv changed the title Adding REST sources & sinks Adding HTTP sources & sinks Aug 24, 2023
@dagardner-nv
Copy link
Contributor Author

/merge

1 similar comment
@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 913fdfb into nv-morpheus:branch-23.11 Aug 24, 2023
@dagardner-nv dagardner-nv deleted the david-rest-source-sink branch August 24, 2023 17:30
tzemicheal added a commit to tzemicheal/Morpheus that referenced this pull request Aug 25, 2023
@tzemicheal
Adding HTTP sources & sinks (nv-morpheus#977)
93f232b 
* Adds four new stages (nv-morpheus#864):
  1. HttpServerSourceStage - Spins up a rest server to listen for incoming POSTS
  2. HttpClientSourceStage - Source that polls a remote HTTP server for incoming data
  3. HttpClientSinkStage - Sink that POSTS data to a remote HTTP server
  4. HttpServerSinkStage - Sink that statrs a rest server listening for GET requests
* Fix `TensorIndex` usage for the `stop_after` feature of the KafkaSourceStage

fixes nv-morpheus#864

Currently `HttpServerSourceStage` is the only one of the four with a C++ impl. The rest server itself is written in C++ with Python bindings and is used by both C++ & Python stages.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: nv-morpheus#977
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA]: Create REST Source/Sink Stages
3 participants
@dagardner-nv @cwharris @mdemoret-nv