Filename decoding issue for zip files archived by macOS

[benoitsan]

Summary

Zip entries filenames are not decoded correctly when reading the content of a zip file produced by macOS.

Steps to Reproduce

1. on macOS 10.13.4, make a zip file archive.zip with 2 files "Benoît.txt" and "pic👨‍👩‍👧‍👦🎂.jpg"
2. Read the path entries of this zip archive on iOS 11.3.1:

let archive = Archive(url: /urlto:archive.zip/, accessMode: .read)!
for entry in archive {
  print("entry: \(entry.path)")
}

Expected Results

It should log "Benoît.txt" and "pic👨‍👩‍👧‍👦🎂.jpg"

Actual Results

It logs "Benoît.txt" and "pic👨‍👩‍👧‍👦🎂.jpg"

Regression & Version

• I'm using ZIPFoundation on the master branch (e44ff6b)

• The workaround to fix this issue is to force to use the UTF-8 encoder in the path getter of the Entry struct in Entry.Swift

public var path: String {
  ...
  let encoding = String.Encoding.utf8 //isUTF8 ? String.Encoding.utf8 : codepage437
  return String(data: self.centralDirectoryStructure.fileNameData, encoding: encoding) ?? ""
}

Related Link

• archive.zip to reproduce this issue. https://www.dropbox.com/s/5tuf7ll4hapwgp1/utfissue.zip?dl=0

[weichsel]

Thank you for the detailed bug report.

I am able to reproduce the problem, but so far I have no proper solution.
The problem seems to be that the Archive Utility on macOS does not set Bit 11 of the "General Purpose Bit Field" as required by APPENDIX D of the ZIP specification:

If general purpose bit 11 is set, the
filename and comment must support The Unicode Standard, Version 4.1.0 or
greater using the character encoding form defined by the UTF-8 storage
specification.

Archive Utility seems to leave the Bit unset and implicitly treats all paths as UTF-8 - Also during unzipping. That's why the paths are intact when you perform a zip/unzip roundtrip with only Archive Utility.

I am looking for input on how to solve this.
Some things I considered so far:

• Check if Archive Utility stores some metadata in one of the CD extra fields
• Perform some sort of encoding detection
• Ignore the spec and also default to UTF-8

[benoitsan]

Thank you Thomas for the details.

It seems to be a common issue with zip files. I found some interesting topics here:

• Unicode character support in filenames when reading a zip created in OSX gildas-lormeau/zip.js#131
• https://stackoverflow.com/questions/13261347/correctly-decoding-zip-entry-file-names-cp437-utf-8-or

I looked in some other zip libraries to see how they handle the filename encoding and I didn't find any real solution. Some libs strictly use the Bit 11, some other libs force the encoding as UTF-8.

I'm not sure there is a good solution without drawback.

PS: This issue is not a real one for using ZIPFoundation. I found it during my tests and reported it.

[weichsel]

Thanks for the links Benoit - Some very good discussion there.

I tried to come up with a solution today and experimented with stringEncoding(for:encodingOptions:convertedString:usedLossyConversion:) but abandoned that approach for 2 reasons:

• That method basically performs an educated guess and might return a wrong encoding
• It is not available on Linux

Currently I think the best approach would be to stick with the strict implementation but additionally provide a well documented utf8EncodedPath property which ignores Bit 11 of the "General Purpose Bit Field" and always decodes the path using UTF-8.

[benoitsan]

I'm not sure what would be the best way to deal with this encoding issue.

PS: If I'm not wrong the bit 11 for UTF encoding was added in 2006. That could explain why it's missing in the macOS archived files…

In my opinion, adding a utf8EncodedPath property is an improvment but the value can only be trusted if you know that the zip file is encoded in UTF-8. It's still quite fragile.

Using an encoding detection for UTF-8 if the bit 11 is missing seems to be the best way to improve the current API. It would allow to get a better chance to have the original filename.

They do this in ZipLib: http://community.sharpdevelop.net/forums/t/21873.aspx

 public static string ConvertToStringExt(int flags, byte[ data)
    {
        if ( data == null ) {
            return string.Empty;    
        }
        // Compliant ZIP files set this flag if the filename uses UTF8
        var useUtf8 = (flags & (int)GeneralBitFlags.UnicodeText) != 0;
        // Some programs, such as OSX zip, don't set the flag, but use UTF8 anyway,
        // so a best guess must be made of the filename encoding
        if (!useUtf8) {
            useUtf8 = Unicode.Utf8Checker.IsUtf8(data, data.Length);
        }
        // Is the filename UTF8 encoded? 
        if (useUtf8) {
            return Encoding.UTF8.GetString(data, 0, data.Length);
        }
        else {
            return ConvertToString(data, data.Length);
        }
    }
           

You said that you got wrong results when using +[NSString(NSStringEncodingDetection) stringEncodingForData:encodingOptions:convertedString:usedLossyConversion:]

Did you try to limit the detection to NSUTF8StringEncoding by using these keys?

FOUNDATION_EXPORT NSStringEncodingDetectionOptionsKey const NSStringEncodingDetectionSuggestedEncodingsKey           API_AVAILABLE(macos(10.10), ios(8.0), watchos(2.0), tvos(9.0));   // NSArray of NSNumbers which contain NSStringEncoding values; if this key is not present in the dictionary, all encodings are weighted the same

FOUNDATION_EXPORT NSStringEncodingDetectionOptionsKey const NSStringEncodingDetectionUseOnlySuggestedEncodingsKey    API_AVAILABLE(macos(10.10), ios(8.0), watchos(2.0), tvos(9.0));   // NSNumber boolean value; if this key is not present in the dictionary, the default value is NO

Since the lib must work on Linux, you could also implement you own UTF-8 checker. There is an implementation here

[colincameron]

I'm having this same issue with encoding of filenames in ZIP files created on macOS.

String.init(data:encoding:) is failable, so would it be an option to try UTF-8 encoding if bit 8 is unset, and fallback to Code page 437 if it fails?

The question is, is there any circumstance where a CP437 filename would be valid UTF-8, and decode correctly?

From reading the docs on CP437 (link to table) and UTF-8, and some experimentation in a playground, here's how characters would decode as UTF-8:

• 0x01-0x1F - Unicode control characters. Decode successfully. Mostly as non-printable characters, except 0x09-0x0c (horizontal tab, line feed, vertical tab, form feed)
• 0x20-0x7E - Characters match.
• 0x7F - Unicode DEL - renders as non-printable character.
• 0x80-0xFF - Fail to decode, except 0xA9 (⌐ - reverse not sign in CP437) which renders as non-printable when followed by a character in the range 0x01-0x7F (the following character prints correctly).

The following combinations of characters that form Unicode code points decode:

• 0xC2-0xDF followed by 0x80-0xBF - This would be a box drawing character, followed by an accented character, fraction, some punctuation, or another box drawing character in CP437.
• 0xE0-0xEF followed by 0x80-0xBF twice - This would be a greek character, followed by an accented latin character, punctuation, or box drawing character (twice).
• 0xF8-0xFF followed by 0x80-0xBF thrice - Mathematical symbols, then as above.

The only issues that would arise from assuming UTF-8 would come from use of the dingbats in 0x01-0x1F, or certain multi-character combinations of box drawing characters, greek letters, or accented latin characters. Any other string will either fail to decode or match. I would be extremely surprised if these situations would ever arise in a filename.

Of course, doing this would be going against the spec, but surely the situation in this issue is far more common than any decoding error that would come up in assuming UTF-8 with a CP437 fallback.

[weichsel]

Thanks for your research Colin.

Prioritizing UTF-8 over CP437 by default would lead to decoding issues in real world situations. Greek letters in file names are not uncommon for example.
I've seen several apps that use the β symbol to indicate beta status in their app names.

The only deterministic way to solve this problem I can think of, is some sort of opt-in mechanism that lets API clients decide if they want to override the encoding that is defined in the archive. Probably we should add an optional stringEncoding parameter to all methods that handle paths:

• Archive.extract
• and the FileManager unzipItem: extension method

[colincameron]

Thanks for the reply

Prioritizing UTF-8 over CP437 by default would lead to decoding issues in real world situations. Greek letters in file names are not uncommon for example.
I've seen several apps that use the β symbol to indicate beta status in their app names.

Just to clarify, decoding errors would only crop up with greek characters followed by two accented characters. A rare situation, but still technically possible.

Perhaps an optional encoding parameter in the appropriate places would be a good solution, which on failure would fall back to UTF-8 if bit 11 is set, and CP437 if not - the current behaviour.

As far as I can see, the relevant methods are:

• FileManager.unzipItem()
• Entry.path - as this is a property, an alternative method Entry.path(preferredEncoding:) or similar would be needed
• Archive.subscript() - not sure how this can be addressed, as there's no way of passing any additional parameters. What about a new property preferredPathEncoding that can be set on an Archive object?

It doesn't look like either Archive.extract method actually uses the path at all (I'm just reading the code though, you wrote it 🙂).

If you're interested, I could put together a PR that covers these changes.

[weichsel]

Hi Colin,

A PR that implements user defined path decoding would be great.
The changes you are suggesting look good to me.
You are right, both extract: methods get the path passed in from the outside - so they don't need the preferredEncoding parameter.

I think we can get the feature implemented by:

• adding an optional preferredEncoding: parameter to the Archive initializer. (Including documentation that this only affects path decoding)
• store the preferredEncoding as property of Archive
• ... take that encoding into account in the subscript implementation
• add path(using: String.Encoding) to Entry (The existing path getter can use that new method with the encoding it derived from the metadata)
• add an optional preferredEncoding: parameter to the unzip method in the FileManager extensions. This should get forwarded to Archive.

I am currently on a tight schedule but let me know if I can help/test anything. I can also provide documentation and unit tests.

[marinofaggiana]

Hi @weichsel I have an issue with the preferredEncoding in Archive, with .utf8 the fileName in archive.addEntry(with: fileName, relativeTo:path) do not encode correctly the fileName ... or am I wrong ?

For example the ä it should be c3 a4 but in zip file the ä is 61 cc 88

let str = String(describing: "ä".cString(using: String.Encoding.utf8))

Thanks

[weichsel]

Hi @marinofaggiana,
There are several ways to encode umlauts in UTF-8.
ZIPFoundation obtains the filenames via FileManager.fileSystemRepresentation which seems to return an a followed by an ¨. This uses multiple code points to represent a single character (61 cc 88). The same character can be represented by a single codepoint (which would be encoded as the byte sequence: c3 a4).
The standard library provides a method that can be used to normalize strings to only use precomposed characters: precomposedStringWithCanonicalMapping .

I think writing the un-composed version is fine but when investigating the phenomenon you pointed out, I noticed some other path encoding problems within the library:

• Archives created with ZIPFoundaiton currently always set the 11th bit of the generalPurposeBitFlag field to 1 to indicate UTF-8 encoded paths.
• ... but FileManager.fileSystemRepresentation returns just an array of bytes without making any guarantees about the used encoding. (The encoding is an implementation detail of the file system and e.g. APFS and several other file systems do not use a canonical encoding at all). I think the proper way to handle this would be to always encode the path as UTF-8 when adding entries.
• The preferredEncoding flag is a bit misleading because it does not affect the way paths are stored. It is only used when deocding the paths in an existing archive. It can be useful to enforce an encoding for archives where the general purpose bit 11 is not properly set. We should either rename that parameter or at least better document it.

I will re-open this issue in the hope for some more input - Finding a sane path through this file system/string encoding maze is 🤯.