Question on config

[9mm]

Hey, I had some specific questions about the config settings:

• So pool_size is the actual number of simultaneous connections that are kept open for each host:port combo, correct? ... so if I hit 3 urls 20 times each very rapidly, and I set pool_size to 30, that is OK because its 30 for each url (90 total) and not 30 overall
• pool_timeout ... does this mean after this many milliseconds, connections are closed for that host:port combo in the pool?
• pool_max_overflow what does this mean? The only thing I can imagine is that if you have a pool size of 1, and you create 5 requests quickly, it will "queue" them up, up to 5, and after that it would error or something? From what I can tell, however, is I can make 20 requests really fast when pool_size is andpool_max_overflow` is 2 and it still requests lots of connections without errors or anything. I just dont really understand what this does

Furthermore i actually dont even know its actually using my pool setting. If i put it some garbage for the pool name like :asdf it still works, it doesnt raise an error. I would assume its working because the atom is the same, but 🤷🏻‍♂️

[9mm]

I am trying to make 15K requests in 1 minute to 3 different hosts... I keep getting connection pool timeouts but ive set it to 99999 and I still get them. I guess I have no idea what this setting really does. Its very important I hit all URLs without timing out, if i set a timeout it should be on the request level, as I will fine tune the timeout for the different hosts and what their performance is like.

Its almost like its not reading my config?

resp = MachineGun.get!(url, [{"Accept", "*/*"}, {"User-Agent", "Test/1.0"}], %{request_timeout: @request_timeout, pool_group: :feeds})

And in config/config.ex

config :machine_gun, feeds: %{
  pool_size: 2,         # Poolboy size [1]
  pool_max_overflow: 2, # Poolboy max_overflow [1]
  pool_timeout: 1000000,
  conn_opts: %{}        # Gun connection options [2]
}

[petrohi]

Pool grows each time when new request arrives and all connections are busy serving other requests. Pool grows to pool_size without closing idle connections. Then it grows to pool_size + pool_max_overflow, which is maximum size, but now it will close idle connections to get back to pool_size.

You may define special pool configuration, like you did with feeds, and use it by referencing with pool_group option. Dedicated pool is maintained per each host:port:group combination.

pool_timeout is time you allow to wait for next available connection. When pool is at maximum size you may start seeing {:error, :pool_timeout} errors. You can specify :infinity pool timeout so that your request will wait forever for available connection.

[9mm]

Thanks so much for your reply.. All what you wrote make sperfect sense. Am i putting it on the wrong spot or something, I still get pool timeouts

[9mm]

This is in config/config.exs at the bottom

config :machine_gun, feeds: %{
  pool_size: 2,         # Poolboy size [1]
  pool_max_overflow: 2, # Poolboy max_overflow [1]
  pool_timeout: :infinity,
  conn_opts: %{}        # Gun connection options [2]
}

[9mm]

And heres copy paste exactly how I use it in anoher part of my elixir project

resp = MachineGun.get!(url, [{"Accept", "*/*"}, {"User-Agent", "Test/1.0"}], %{request_timeout: @request_timeout, pool_group: :feeds})

[petrohi]

Let me try this. Maybe you found a bug.

[9mm]

Thanks man! Happy thanksgiving :D

[9mm]

Hey just wondering if you had a chance to check that out. Cheers!

[petrohi]

You are right--there is an issue.

Right now pool_timeout can only be passed inside opts map to the request function (along with request_timeout).

I will fix by allowing to specify pool_timeout and request_timeout in the config and then allowing to override them in opts passed to the request function.

[9mm]

Awesome, thanks!