File Rate Limiter #751
File Rate Limiter #751
Conversation
Codecov Report
@@ Coverage Diff @@
## staging #751 +/- ##
===========================================
- Coverage 30.01% 29.26% -0.75%
===========================================
Files 68 77 +9
Lines 7653 8002 +349
===========================================
+ Hits 2297 2342 +45
- Misses 5014 5316 +302
- Partials 342 344 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. |
| } | ||
|
|
||
| func requestField(r *http.Request, key string) (string, bool) { | ||
| if r.Form != nil { |
There was a problem hiding this comment.
We can get URL query data from r.Form as well, so don't have to call the r.URL.Query below, and the code could be like:
if err := r.ParseForm(); err != nil {
return "", err
}
return r.Form.Get(key), nilThere was a problem hiding this comment.
I opted to just moved TryParseForm() and GetField() from handler package to common and reuse that GetField() instead.
config/0chain_blobber.yaml
Outdated
| @@ -69,6 +69,7 @@ block_worker: http://198.18.0.98:9091 | |||
|
|
|||
| handlers: | |||
| rate_limit: 0 # 10 per second . it can't too small one if a large file is download with blocks | |||
| file_rate_limit: 1000 # 100 files per second | |||
There was a problem hiding this comment.
1000 is too fast for both upload/download by default, the blobber would be under DDoS, so perhaps 100. And if the upload/download process get affected, then we might need to optimize the block size to reduce the requests number.
There was a problem hiding this comment.
Make sense. Adjust the chunk size instead of creating too many API requests for upload, and block size for download.
Changes
File Rate Limiter on upload and download.
When
file_rate_limiter= 0, rate limiter is disabled.As part of changes,
TryParseForm()andGetField()were moved to common pkgFixes
Resolves #671
Tests
Tasks to complete before merging PR:
Associated PRs (Link as appropriate):