Add proxyUrl to impit.fetch()

[garysassano]

I'd like to be able to use a proxyUrl within impit.fetch() similarly to Bun's fetch() or gotScraping. Currently you can only set proxyUrl on a Impit instance, which means you are limited to either use a single proxy or a rotating proxy endpoint. By being able to use the proxyUrl on the fetch() method directly you would instead be able to change the proxy used for each single request.

[barjin]

This is intended behaviour, as reusing the same Impit instance allows for HTTP connection pooling (reusing the underlying TLS / TCP resources for multiple subsequent requests).

HTTP connection pooling (and other resource-sharing notions, e.g. HTTP/2 internals) are not possible when using different proxies for different requests (two requests going to the same endpoint cannot use the same TCP connection if they are told to go through different proxies).

We'd prefer to communicate this idea in our interface too - the fact that you have to create a new Impit instance when switching proxies shows what's actually happening under the hood (Impit has to open a new socket, go through the TLS handshake etc.)

[garysassano]

I've been thinking about this proxy issue and creating a new Impit instance for every single request would honestly use way too much memory - it's just not practical for any serious scraping work.

The thing with those rotating proxy endpoints that many providers offer is that they're quite limiting. You're stuck with just one provider's proxy pool, can't mix proxies from different providers, and have minimal control over how proxies are selected (beyond maybe picking a country).

What works better for me is a simpler approach. I keep all my proxy URLs in a text file, parse it when starting up the scraper, and convert them to URL objects. Then I rotate through them sequentially rather than randomly. I prefer sequential rotation because it gives me predictability of how often each proxy gets used over time. This is particularly useful when working with a limited set of proxies and you're trying to avoid hitting the same proxy too frequently.

This is exactly why being able to pass a proxy URL directly to the fetch() method would be so much more flexible than needing to create a new instance every time you want to change proxies. It would make implementing this kind of rotation strategy much more straightforward and efficient.

[barjin]

I turned this into a discussion (not closing this), since we're definitely open to interface improvement ideas.

Would e.g. something like httpx's Client interface work for you? E.g.

import { Impit, fetch } from 'impit'

// Persistent client reusing network resources
const client = new Impit();
await client.fetch();
await client.fetch();

// Ad-hoc client creation (each top-level `fetch` call creates a new one-use `Impit`)
await fetch()
await fetch()
 

[garysassano]

I think this approach could work. Another idea (though I haven’t tested it) is to instantiate a separate Impit object for each available proxy at startup, then cycle through the array of Impit instances. I'm unsure about the memory overhead this might introduce.