Encoding issues encountered while trying to send multi-part emails

[duncanmorrissey]

Describe the bug
When the SDK builds a multipart/form-data request to POST /v3/grants/:id/messages/send, the "message" part (JSON) is left in UTF-8 encoding.
HTTParty (used under-the-hood) expects every multipart field to be ASCII-8BIT/BINARY; with a UTF-8 part it raises
ArgumentError: invalid byte sequence in UTF-8 during HTTParty::Request::Body#generate_multipart.

To Reproduce

```ruby
require "nylas"

nylas = Nylas::Client.new(api_key: ENV["NYLAS_API_KEY"])

payload = {
  to:   [{ email: "someone@example.com", name: "Someone" }],
  subject: "Multipart test",
  body: "hello",
  attachments: [
    # a pdf attachment ≥ 3 MB to force multipart
    Nylas::FileUtils.attach_file_request_builder("/path/to/big.pdf")
  ]
}

nylas.messages.send(identifier: ENV["GRANT_ID"], request_body: payload)

Stack trace (shortened):

…/gems/httparty-0.23.1/lib/httparty/request/body.rb:54:in `block in generate_multipart'
…/nylas-6.5.0/lib/nylas/handler/http_client.rb:157:in `httparty_execute'

Expected behavior
Nylas::Messages#send should succeed without raising, regardless of attachment size or type

SDK Version:

• nylas-ruby 6.5.0
• Ruby 3.2.5
• httparty 0.23.1

Additional context

A local monkey-patch in my rails app fixed the issue by forcing the "message" part to be binary

module Nylas::FileUtils
  class << self
    alias_method :_orig_build_form_request, :build_form_request
    def build_form_request(request_body)
      form_data, files = _orig_build_form_request(request_body)
      form_data["message"] = form_data["message"].dup.force_encoding(Encoding::BINARY)
      [form_data, files]
    end
  end
end

A similar change inside FileUtils.build_form_request (or at the call-site before passing to HTTParty) would permanently resolve the error.

[AaronDDM]

Hey. Thank you for this report and sorry that you are running into this. I'll investigate and see what is going on.

[AaronDDM]

@duncanmorrissey I am unable to reproduce this locally, would you happen to have some reproducible code that I could look at?

[duncanmorrissey]

Hi @AaronDDM - this script reproduces the error for me locally:

#!/usr/bin/env ruby
# reproduce_multipart_encoding_bug.rb
#
# Prerequisites
#   • Ruby 3.2.5
#   • ENV["NYLAS_API_KEY"]      – your app-level API key
#   • ENV["NYLAS_TEST_GRANT_ID"] – any live grant you can send mail from
#
# Expected: Encoding::CompatibilityError raised inside
#           httparty/request/body.rb:54 (generate_multipart)

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"
  gem "nylas",   "6.5.0"
  gem "httparty", "0.23.1"   # 0.23.0 and 0.22.0 show the same stack-trace
end

require "nylas"
require "securerandom"

NYLAS_API_KEY  = ENV.fetch("NYLAS_API_KEY")
GRANT_ID       = ENV.fetch("NYLAS_TEST_GRANT_ID")

client = Nylas::Client.new(api_key: NYLAS_API_KEY)
big_file = File.open("/Users/YourUser/Downloads/example\ \(1\).jpg", "rb") # this is an 8mb jpg file so it's handled as multipart

payload = {
  from:    [{email: "your_email@example.com",    name: "Sender name"}], # TODO swap in your email
  to:      [{email: "other_email@example.com", name: "Recipient Name"}],
  subject: "Multipart bug demo #{SecureRandom.hex(4)}",
  body:    "<p>Hello UTF-8 world&nbsp; – rocket 🚀</p>",                    # <-- Ruby treats as UTF-8 because richtext included
  attachments: [{ filename: "big.jpg", content: big_file, size: big_file.size, content_type: "image/jpeg" }]
}

form_payload, opened_files = Nylas::FileUtils.handle_message_payload(payload)

begin
  client.messages.send(identifier: GRANT_ID, request_body: form_payload)
rescue => e
  puts "\n=== Reproduced exception ==="
  puts "#{e.class}: #{e.message}"
  puts e.backtrace.take(15).join("\n")
  puts "=============================\n\n"
ensure
  opened_files.each(&:close)
end

[AaronDDM]

Thank you, taking a look!

[AaronDDM]

Thanks @duncanmorrissey , I am able to reproduce this. Question however, are you not using the Nylas::FileUtils.attach_file_request_builder utility because you want more control over how you manage the attachment file?

Edit: not saying that it solves the problem - just curious. I'll fix the issue you reported. Thank you!

[AaronDDM]

Please take a look at the PR 🙏

[duncanmorrissey]

Hi @AaronDDM - appreciate you taking a look. I'm unfortunately still getting the same error locally when using this branch though. Do both the key and value of the hash need to be encoded as ASCII-8BIT? Do other values (subject) need the same fix? What additional information can I provide?

---

Question however, are you not using the Nylas::FileUtils.attach_file_request_builder utility because you want more control over how you manage the attachment file?

We are not using that utility because we do not have local files and the interface depends on local files. Our attachments are all in S3 and we can't put on our webserver to send.

We have a Rails Mailer that creates a MailMessage object, which files are attached to:

# pseudocode example
class OurMailer < ApplicationMailer
  def email(emailable:)
    # ...

    @emailable.attachments.each do |attachment|
      filename = attachment.blob.filename.to_s
      attachments[filename] = attachment.open do |file|
        {
          mime_type: attachment.content_type,
          content: file.read,
          filename: filename
        }
      end
    end

    # ...

    I18n.with_locale(locale) { mail(options) }
  end
end

We use that to create a MailMessage object

message_delivery = OurMailer.email(
  emailable: emailable_object,
  # ...
)

Nylas::FileUtils.handle_message_payload has an implicit requirement above 3mb that it also requires an actual file object so we are wrapping our streamed content into a "File-like" object for use here:

# Nylas::FileUtils.handle_message_payload has an implicit contract that above 3mb, the attachment objects
# passed are will be IO objects that can be read directly. This ensures that contract is met.
class GcNylas::InMemoryFile < StringIO
  attr_reader :original_filename, :content_type

  def initialize(data, filename:, content_type:)
    super(data)
    @original_filename = filename
    @content_type = content_type
  end

  # HTTParty / multipart‑post look for this to decide the "filename=" part
  def path
    original_filename
  end
  alias_method :to_path, :path
end

# this is passed as the `file_attachments` key of the handle_message_payload request body
def file_attachments
  @_file_attachments ||= @mail_message.attachments.map do |part|
    io = GcNylas::InMemoryFile.new(
      part.body.decoded,
      filename: part.filename.to_s,
      content_type: part.mime_type
    )

    {
      filename: io.original_filename,
      content_type: io.content_type,
      size: io.size,
      content: io
    }
  end
end

If there is a different approach you recommend, I'd love to hear!

[AaronDDM]

@duncanmorrissey
Sorry for the delay - I think I. originally went down the wrong path. It looks like the issue was that we simply were not handling cases where the content was directly being provided into the payload. If you used the utility, we would create a File object and that's why it worked well.

Now I am using StringIO to convert the content binary into an object useable by HTTParty.

Can you let me know if this fix works for you?