Headers should not use charset specified in Content-Type:

[ConradIrwin]

Headers in email are defined to be ASCII only (or UTF-8 if you're experimenting with RFC 5335). They should not be any other character encoding.

At the moment the mail gem decodes headers that appear after the Content-Type header with the same charset as specified there. This is wrong, and causes strange errors with some character encodings:

pry> Mail.new("Content-Type: text/plain; charset=iso-2022-jp\r\nSubject: Test\r\n\r\nTest\r\n").encoded
Encoding::CompatibilityError: incompatible encoding regexp match (US-ASCII regexp with ISO-2022-JP string)
from /home/conrad/.rvm/gems/ruby-1.9.3-p194@foo/bundler/gems/mail-de4eec71b0b2/lib/mail/fields/unstructured_field.rb:179:in `gsub!'

[jeremy]

Agreed. The Content-Type charset doesn't mean the other headers use that charset.

[jpastuszek]

I see this too. Custom headers after Content-Type: text/plain; charset="iso-2022-jp" header causes:

       incompatible encoding regexp match (US-ASCII regexp with ISO-2022-JP string)
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/fields/unstructured_field.rb:195:in `gsub!'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/fields/unstructured_field.rb:195:in `encode_crlf'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/fields/unstructured_field.rb:155:in `fold'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/fields/unstructured_field.rb:103:in `wrapped_value'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/fields/unstructured_field.rb:69:in `do_encode'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/fields/unstructured_field.rb:51:in `encoded'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/field.rb:189:in `method_missing'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/header.rb:209:in `block in encoded'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/header.rb:208:in `each'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/header.rb:208:in `encoded'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/message.rb:1807:in `encoded'
     # /Users/wcc/.rvm/gems/ruby-1.9.3-p429/gems/mail-2.6.1/lib/mail/message.rb:1875:in `to_s'

[igorcalabria]

So, we could force UTF-8 encoding for the header? It's backwards compatible with ascii and it solves this problem I've been having with some annoying mail providers that uses Latin-1 on its headers. Currently I'm using this patch:

module Ext
  Mail::Header.class_eval do
    def initialize(header_text = nil, charset = nil)
      @charset = charset
      if header_text
        self.raw_source = header_text.force_encoding(Encoding::BINARY).to_crlf.lstrip
        split_header
      end
    end
  end
end

I'm using binary because I can deal with whatever these a*holes send to me later. Anyway, using UTF-8 and replacing invalid chars makes sense too.

[fatihorhan]

@igorcalabria One of those a*holes would be Microsoft with Exchange 2013. It send headers in Windows-1254 charset. It was doing this with "text/plain" mails so came up with this hack:

if msg.multipart?
    if msg.text_part and msg.text_part.charset.downcase != "utf-8"
        part = msg.text_part
        raw_source = msg.header.raw_source
        raw_source = raw_source.force_encoding(part.charset).encode("UTF-8")
        msg.header.instance_variable_set "@raw_source", raw_source
        msg.header.send :split_header
    end
end