Rotating proxy functionality #139
Conversation
andreademasi
changed the title
andreademasi
force-pushed
the
feature/rotating-proxy
branch
3 times, most recently
from
43c5403
to
78a5cfd
Compare
andreademasi
force-pushed
the
feature/rotating-proxy
branch
from
0489e11
to
2d0a200
Compare
|
You somehow changed the line-endings think, which marked the whole file as changed, can you fix that, so only your actual changes are marked. |
andreademasi
force-pushed
the
feature/rotating-proxy
branch
from
03cc3f6
to
1af5fe1
Compare
|
@Linus045 Fixed |
| logger.error(e) | ||
|
|
||
| # Merging old proxies with new ones | ||
| _list = list(proxy_res[:-1].split("\n") | _proxy_list.keys()) |
There was a problem hiding this comment.
If the try..except above triggers proxy_res will be an invalid value and therefore cause another exception here when trying proxy_res[:-1] wouldn't it?
There was a problem hiding this comment.
I'll add a return statement in the except clause
andreademasi
force-pushed
the
feature/rotating-proxy
branch
2 times, most recently
from
20c5f7c
to
8a98ba8
Compare
|
|
||
|
|
||
| def init_proxy(): | ||
| threading.Thread(target=lambda: _every(60 * 10, _fetch_proxies)).start() |
There was a problem hiding this comment.
we might wanna make the poll delay also configurable via the config.
But for now its fine, after some more testing we can think about it again.
| proxy = get_proxy() | ||
| logger.debug(f"Using proxy: {proxy}") | ||
| try: | ||
| response = self.http_client.get(request_url, proxies={"http": "socks5://" + proxy}) |
There was a problem hiding this comment.
Another point we should consider is what happens when the request via the proxy fails or returns status code 429.
In that case we probably wanna remove the proxy or don't use it for some time (e.g. 5min).
But I don't mind implementing that in another PR considering that this is an opt-in feature via the config anyway.
| threads: list[threading.Thread] = [] | ||
| try: | ||
| proxy_res = requests.get( | ||
| "https://www.proxyscan.io/api/proxy?last_check=180&limit=20&type=socks5&format=txt&ping=1000" |
There was a problem hiding this comment.
we might also wanna consider a different last_check value, but thats something we can adjust after more testing as well.
(For reference: https://www.proxyscan.io/api)
|
|
||
| if rotating_proxy_is_ready(): | ||
| proxy = get_proxy() | ||
| logger.debug(f"Using proxy: {proxy}") |
There was a problem hiding this comment.
I think this should be a logger.info combined with the first message e.g.:
when using proxy: "Getting Binance announcements [Using proxy: 127.0.0.1]"
when no valid proxy exists: "Getting Binance announcements [No proxy available]"
when the proxy feature is disabled: "Getting Binance announcements"
That being said, it should be implemented in a new PR and not here.
I was planning on reworking the console logs at some point anyway so we should consider this then as well.
|
I'm thinking it might be useful to filter proxies to countries close to japan (where Binance's servers are located) like China, Thailand, Singapore, South Korea, Hong Kong, and of course Japan. @Linus045 what are your thoughts on this? |
I agree, but I think this should be done in a new PR. |
|
This looks good to me. |
This PR addresses issues regarding rate limiting by using a socks5 rotating proxy derived from a list of proxies fetched from proxyscan.io.
May fix #137 and #115
Main issue is proxy quality, some of them are slow or not working.
TODO:
config.yml