Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement HTTP request/response source #839

Open
kerinin opened this issue Oct 31, 2023 · 0 comments
Open

Implement HTTP request/response source #839

kerinin opened this issue Oct 31, 2023 · 0 comments

Comments

@kerinin
Copy link
Collaborator

kerinin commented Oct 31, 2023

Idea is something along the lines of the code block shown below. Things to consider:

  1. Support both "read-only sources" (things which always respond 200 and are used in defining other responses)
  2. Support "read/write sources" (things which compute responses using kaskada)

It's OK if this only works with the new partitioned execution code paths (eg., using the Source / Destination traits).

# this produces `{ request_id: i64, data: <schema> }`
source = kd.sources.Http(
  schema = ...,
  # Optional response. This allows for requests that append data.
  # If not specified, then a later `run_and_respond` is necessary.
  respond_with = 200,
)

output = kd.record({
  request_id: source.request_id,
  x_plus_y = source.data.x + source.data.y,
)}

source.run_and_respond(
  # Use Kaskada to define responses.
  respond_with = output,
  # Report a warning if the output is not available
  # for a request within this timelimit.
  # this may happen if the request gets buffered in
  # a shift, etc.
  timelimit = 1s
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant