Req adapter follow redirect option incompatible

[adamcstephens]

Environment

• Elixir & Erlang versions (elixir --version):

Erlang/OTP 27 [erts-15.0] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

Elixir 1.17.2 (compiled with Erlang/OTP 27)

• ExAws version mix deps |grep ex_aws

* ex_aws 2.5.5 (Hex package) (mix)
  locked at 2.5.5 (ex_aws) ed7ee39c
* ex_aws_sqs 3.4.0 (Hex package) (mix)
  locked at 3.4.0 (ex_aws_sqs) b5044822

• HTTP client version. IE for hackney do mix deps | grep hackney

* req 0.5.6 (Hex package) (mix)
  locked at 0.5.6 (req) cfaa8e72

Current behavior

Attempting to use the Req backend fails as :follow_redirect is not an http option. This error is from Broadway SQS.

    ** (EXIT) an exception was raised:
        ** (ArgumentError) unknown option :follow_redirect. Did you mean :follow_redirects?
            (req 0.5.6) lib/req/request.ex:1169: Req.Request.validate_options/2
            (req 0.5.6) lib/req.ex:512: Req.merge/2
            (req 0.5.6) lib/req.ex:1064: Req.request/2
            (ex_aws 2.5.5) lib/ex_aws/request/req.ex:22: ExAws.Request.Req.request/5
            (ex_aws 2.5.5) lib/ex_aws/instance_meta.ex:18: ExAws.InstanceMeta.request/3
            (ex_aws 2.5.5) lib/ex_aws/instance_meta.ex:76: ExAws.InstanceMeta.task_role_credentials/1
            (ex_aws 2.5.5) lib/ex_aws/instance_meta.ex:91: ExAws.InstanceMeta.security_credentials/1
            (ex_aws 2.5.5) lib/ex_aws/config/auth_cache.ex:132: ExAws.Config.AuthCache.refresh_auth_now/2
    (elixir 1.17.1) lib/gen_server.ex:1128: GenServer.call/3

• config

config :ex_aws,
  http_client: ExAws.Request.Req

Expected behavior

Expect to successfully connect to SQS.

[GRoguelon]

Hi,

I got the same error. I had to rollback to our custom Req implemenation. Also the way it's implemented, you need to have :hackney and Req in your project besides the dependencies being declared as optional.

Otherwise you get this Elixir warning at compilation:

==> ex_aws
Compiling 28 files (.ex)
    warning: Req.request/1 is undefined (module Req is not available or is yet to be defined)
    │
 22 │     |> Req.request()
    │            ~
    │
    └─ (ex_aws 2.5.5) lib/ex_aws/request/req.ex:22:12: ExAws.Request.Req.request/5

    warning: :hackney.request/5 is undefined (module :hackney is not available or is yet to be defined)
    │
 21 │     case :hackney.request(method, url, headers, body, opts) do
    │                   ~
    │
    └─ (ex_aws 2.5.5) lib/ex_aws/request/hackney.ex:21:19: ExAws.Request.Hackney.request/5

Generated ex_aws app

[jlgeering]

this is the culprit:

ex_aws/lib/ex_aws/instance_meta.ex

Line 107 in 821722c

defaults = [follow_redirect: true]

=> a default http option from hackney, that does not exist in req, and that cannot be change by setting something in the config

we might be able to work around this by convincing req to accept this option: https://hexdocs.pm/req/Req.Request.html#register_options/2 (somewhere in ExAws.Request.Req ? or maybe even add a custom Req step that transforms hackney options to req options?)

I'm too tired to investigate more tonight, I hope that someone else will have fixed the issue by tomorrow morning :-)

[jlgeering]

my current workaround:

# config.exs
config :ex_aws,
  access_key_id: [:instance_role],
  secret_access_key: [:instance_role],
  http_client: MyHttpClient
  
# my_http_client.ex
# copied from https://github.com/ex-aws/ex_aws/blob/main/lib/ex_aws/request/req.ex
defmodule MyHttpClient do
  @behaviour ExAws.Request.HttpClient

  @moduledoc """
  Configuration for `m:Req`.

  Options can be set for `m:Req` with the following config:

      config :ex_aws, :req_opts,
        receive_timeout: 30_000

  The default config handles setting the above.
  """

  @default_opts [receive_timeout: 30_000]

  @impl true
  def request(method, url, body \\ "", headers \\ [], http_opts \\ []) do
    [method: method, url: url, body: body, headers: headers, decode_body: false]
    |> Keyword.merge(Application.get_env(:ex_aws, :req_opts, @default_opts))
    |> Keyword.merge(Keyword.delete(http_opts, :follow_redirect)) # the temporary fix!
    |> Req.request()
    |> case do
      {:ok, %{status: status, headers: headers, body: body}} ->
        {:ok, %{status_code: status, headers: headers, body: body}}

      {:error, reason} ->
        {:error, %{reason: reason}}
    end
  end
end