Feature/topic route #200
Feature/topic route #200
Conversation
| raise TypeError("Need one of query_param, shard_param, or exact key") | ||
|
|
||
| @wraps(fun) | ||
| async def get( |
There was a problem hiding this comment.
We support only get so no other HTTP operations will be supported here. Could you add that note as a comment
There was a problem hiding this comment.
Thanks @patkivikram!
Have you seen this?
#191
I believe it adds a support for other methods... I can add another unit test to verify this... Makes sense?
There was a problem hiding this comment.
Is there a reason we were only supporting get before? Should #191 be reversed? @patkivikram
There was a problem hiding this comment.
Honestly, I don't think it's a problem, but I am not sure what's the use case for other methods... I mean I get why we are using GET for table_route, to get table values from other workers when the state is distributed, but I don't see how POST is useful, because you can't really modify table values outside of stream operations... And if you send something to a stream, well, then you can send it from any worker, the send doesn't have to come from a specific worker. But maybe there is another use case that I am not aware of.
There was a problem hiding this comment.
correct we don't need PUT/POST/DELETE which is why I asked you to document
Codecov Report
@@ Coverage Diff @@
## master #200 +/- ##
=======================================
Coverage 94.46% 94.46%
=======================================
Files 100 100
Lines 10643 10684 +41
Branches 1199 1204 +5
=======================================
+ Hits 10054 10093 +39
- Misses 524 526 +2
Partials 65 65
Continue to review full report at Codecov.
|
Description
This pull request adds support for a topic_route decorator which behaves exactly like table_route but for a topic. i.e. it routes the web request to the worker that processes the partition of a given topic and key.
This is especially useful for monitoring live topic data or intermediate results for stateless workers that don't store data in a table.
Implementation
I didn't want to change any of the existing interfaces so in some areas I just duplicated the table_route code.
The additional information of the topic partition distribution is stored in the user_data as external_topic_distribution.
Testing
To test this feature I slightly modified the word_count.py example
Run two workers:
faust -A word_count.app --datadir=word-counts-data/worker0 worker --web-port 8000 --web-host localhost -l info
faust -A word_count.app --datadir=word-counts-data/worker1 worker --web-port 8001 --web-host localhost -l info
And test the last value:
curl localhost:8000/last/fox/
curl localhost:8001/last/fox/
Expect the same result from both workers
{"fox": {"the": 0, "quick": 0, "brown": 18222, "fox": 16000}}
Repeat this for other words: the quick, brown... and see the last values for each worker
Please let me know if you have any questions on the implementation or ideas on how to improve this...