Skip to content

Conversation

@elisalimli
Copy link
Contributor

@elisalimli elisalimli commented Mar 11, 2024

What does this PR do?

image

Fixes #39 #53

Steps to test

  1. Run docker-compose up -d to spin up kafka, kafka-ui and redis
  2. Start python server
  3. Run cd ./service/kafka/ & python3 ./consume.py

@elisalimli elisalimli marked this pull request as draft March 11, 2024 06:51
@elisalimli elisalimli requested a review from homanp March 11, 2024 16:29
@elisalimli elisalimli marked this pull request as ready for review March 11, 2024 16:29
@teocns
Copy link

teocns commented Mar 11, 2024

Hey @elisalimli, I am just taking a look at ray which can simplify the integration

Also worth reading distributed kafka consumers with ray

@teocns
Copy link

teocns commented Mar 11, 2024

I am testing this in a Kubernetes cluster and provide feedback along the way;

@elisalimli
Copy link
Contributor Author

Hey @elisalimli, I am just taking a look at ray which can simplify the integration

Also worth reading distributed kafka consumers with ray

@teocns I haven't looked at it yet. What would be the benefit of using Ray instead of just Kafka?

@elisalimli
Copy link
Contributor Author

@homanp we can deploy this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Offload ingestion to background tasks

2 participants