TypeError: unhashable type: 'list' on set_recipient_status

[danieltellez]

Hi,

With:
django-anymail==8.1
django-mailer==2.1

and using SendInBlue as ESP, I am getting this error on set_recipient_status: TypeError: unhashable type: 'list'

I've patched the code of this function as this:

anymail/message.py

    def set_recipient_status(self, recipients):
        self.recipients.update(recipients)
        recipient_statuses = self.recipients.values()
        message_ids = [recipient.message_id for recipient in recipient_statuses]
        message_ids = [item for sublist in message_ids if type(sublist) == list for item in sublist]
        self.message_id = set(message_ids)
        if len(self.message_id) == 1:
            self.message_id = self.message_id.pop()  # de-set-ify if single message_id
        self.status = set([recipient.status for recipient in recipient_statuses])

I think this isn't the best way to resolve it, so I'll thank you very much if anyone can give me a clue why is this happening. Is it maybe some misconfiguration?

[medmunds]

@danieltellez for this error to occur, something has gone very wrong earlier in the code. So you are correct, patching set_recipient_status is not the best way to resolve this. We should try to find the root problem and fix it there.

The message_id for each recipient should always be a single scalar value, never a list. This means something is going wrong in the code that parses recipient status out of SendinBlue's API response: anymail.backends.sendinblue.EmailBackend.parse_recipient_status. There's not a whole lot to that code; it just grabs the "messageId" field from the API response JSON, and assigns that messageId to every recipient of the message.

For that messageId to end up a list, SendinBlue would have to be returning an array as the value of the messageId field, rather than a single string. That wouldn't match their API documentation. (We also have live integration tests that run weekly, to catch undocumented changes to ESP API behavior. The tests that ran last Thursday didn't catch any problems like that.)

So how could this be happening?

• It's possible you've found some combination of message options and settings that make SendinBlue return an undocumented API response format. It would be helpful to see your code that constructs and sends the message, and also all of your ANYMAIL, MAILER, and EMAIL_BACKEND settings.py entries (changing API keys to "[secret]" before posting).

• It's possible you've found a bug in Anymail. It would be helpful to see the full Python stack trace when the error occurs.

• Either way, it would be helpful to see the actual SendinBlue API call and response that is confusing Anymail. The easiest way to get that is to enable DEBUG_API_REQUESTS: in your settings.py, set ANYMAIL = { "DEBUG_API_REQUESTS": True, ... other settings }, and then send the message again. Look for output starting "===== Anymail API request" and "----- Response". (Be sure to change the "api-key" header value to "[secret]" before posting here.)

[danieltellez]

Thank you for your response and help, Mike !

These are my settings:

ANYMAIL = {
    "SENDINBLUE_API_KEY": [secret]"
}
EMAIL_BACKEND = 'mailer.backend.DbBackend'
MAILER_EMAIL_BACKEND = "anymail.backends.sendinblue.EmailBackend"
DEFAULT_FROM_EMAIL = "xxx"
SERVER_EMAIL = "xxx"

This the full stack trace when I execute Anymail send_email command:

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/mailer/management/commands/send_mail.py", line 28, in handle
    send_all()
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/mailer/engine.py", line 245, in send_all
    connection, action_taken = error_handler(connection, message, err)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/mailer/engine.py", line 146, in handle_delivery_exception
    six.reraise(*sys.exc_info())
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/mailer/engine.py", line 233, in send_all
    email.send()
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/django/core/mail/message.py", line 284, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/anymail/backends/base.py", line 94, in send_messages
    sent = self._send(message)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/anymail/backends/base_requests.py", line 58, in _send
    return super()._send(message)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/anymail/backends/base.py", line 130, in _send
    message.anymail_status.set_recipient_status(recipient_status)
  File "/home/vinoteo/.virtualenvs/vinoteo/lib/python3.8/site-packages/anymail/message.py", line 123, in set_recipient_status
    self.message_id = set([recipient.message_id for recipient in recipient_statuses])
TypeError: unhashable type: 'list'

This sentence seems very interesting to me:

The message_id for each recipient should always be a single scalar value, never a list.

This has been the very first thing it seems wrong to me. When I checked the AnymayRecipientStatus, I got this: AnymailRecipientStatus(['<160975180562.601475.17757706999439156974@vinoteo>']. So it looks like, as you said, the Sendinblue response is wrong on parse_recipient_status.

I'll try, in a while, enabling DEBUG_API_REQUESTS, so I can show you the output.

[medmunds]

Thanks for adding the detail, Daniel. There's nothing unusual about those settings, so I'm fairly certain you've discovered a way to provoke undocumented behavior in Sendinblue's API. If so (and ideally if we can get a reproducible case to add to the tests), we should add a workaround in Anymail's Sendinblue EmailBackend.

Per Sendinblue's Send a transactional email API docs (and every example I've ever tried, including with multiple recipients, cc's, using templates or not, etc.), a successful send API call returns JSON in the form:

{
  "messageId": "<201798300811.5787683@relay.domain.com>"
}

The relevant part of Anymail's Sendinblue response parsing just grabs "messageId" from the parsed JSON and sticks it in the AnymailRecipientStatus… effectively:

try:
    message_id = parsed_response['messageId']
except (KeyError, TypeError) as err:
    raise AnymailRequestsAPIError(...)
status = AnymailRecipientStatus(message_id=message_id, status="queued")

So there are only two ways I can think that you'd end up with message_id as a list: either a very weird bug in Requests' json parsing (seems unlikely), or the Sendinblue API returning an array in the JSON response:

{
  "messageId": [
    "<201798300811.5787683@relay.domain.com>",
    "<201798300812.5787684-this-is-not-documented-behavior@relay.domain.com>"
  ]
}

Once you get DEBUG_API_REQUESTS going, if you see an array in the response like that, then the next question is: how is your send different from the tests I've tried? There may be some combination of message options or account settings that leads to different behavior? (Are you adding any custom headers, or sending to a special address like a Sendinblue marketing campaign list, or ???)