Encode::MIME::Header: Rewrite both decoder and encoder

[pali]

Encoder should be now fully compliant of RFC 2047. Decoder is less strict
to be able to decode strings generated by old versions of this module.

To enforce correct strict mode, set package variable
$Encode::MIME::Header::STRICT_DECODE to 1.

[dankogai]

Thank you!

[dracos]

Thanks so much for this, I was just today wondering about trying to fix some of the issues with Encode::MIME::Header and found this had just already been done :) Looking forward to it being released.

[nwellnhof]

First of all, thanks for the effort. I am the author of Email::MIME::RFC2047 and invested quite some time trying to make it robust and RFC-conformant. I ran some tests from my module against your code and the only problems I found are the handling of invalid encoded words. Examples:

Encoded words that aren't separated by whitespace like text=?iso-8859-1?q?text?= should not be decoded, but your code does. From the RFC:

Ordinary ASCII text and 'encoded-word's may appear together in the same header field. However, an 'encoded-word' that appears in a header field defined as '*text' MUST be separated from any adjacent 'encoded-word' or 'text' by 'linear-white-space'.

and

Any message or body part header field defined as '*text', or any user-defined header field, should be parsed as follows: Beginning at the start of the field-body and immediately following each occurrence of 'linear-white-space', each sequence of up to 75 printable characters (not containing any 'linear-white-space') should be examined to see if it is an 'encoded-word' according to the syntax rules in section 2. Any other sequence of printable characters should be treated as ordinary ASCII text.

Then there are encoded words with invalid contents like =?iso-8859-1?b?----?=? My module tries hard to keep such words intact but your code seems to replace them with the empty string. Section 6.3 of the RFC says:

A mail reader need not attempt to display the text associated with an 'encoded-word' that is incorrectly formed. However, a mail reader MUST NOT prevent the display or handling of a message because an 'encoded-word' is incorrectly formed.

This doesn't rule out to completely remove invalid encoded words, but I think it's safer to keep them. Some client code might treat an empty email header as an error condition.

[pali]

@nwellnhof: local $Encode::MIME::Header::STRICT_DECODE = 1;

[nwellnhof]

Great, this works. (Encoded words with invalid contents are still stripped, though.)

[pali]

@nwellnhof How should be correctly decoded string "=?iso-8859-1?b?----?=" ? I think RFC 2047 does not specify it. If you have good idea I can fix it in STRICT_DECODE.

[nwellnhof]

I would simply leave such strings undecoded.

[pali]

Ok, that could make sense in STRICT_DECODE. I will prepare some patches for it...

[pali]

Implemented in #68