Possible encoding problem with Repository.file_status #687

Activity

1. tmr232 commented on Apr 12, 2017

   

   tmr232

   on Apr 12, 2017

   Last edited by tmr232

   Contributor

   More actions

   @wme-at-contact-de Does it happen with pygit-0.25.0 as well?
2. Ari-E-S commented on Jul 23, 2017

   

   Ari-E-S

   on Jul 23, 2017

   Last edited by Ari-E-S

   More actions

   Hi!

   I think I'm running into the same problem using Python 3.6.2 and pygit2 (0.26.0) on MacOS
   I whipped up a test script to duplicate the issue using the official repo for Thumbor (master branch)

   ~ $ cat test.py
   #!/usr/bin/env python3
   import pygit2
   git_status_flags = {
           pygit2.GIT_STATUS_CURRENT:          ('CURRENT', 0),
           pygit2.GIT_STATUS_INDEX_NEW:        ('INDEX_NEW', 1),
           pygit2.GIT_STATUS_INDEX_MODIFIED:   ('INDEX_MOD', 1),
           pygit2.GIT_STATUS_INDEX_DELETED:    ('INDEX_DEL', 1),
           pygit2.GIT_STATUS_WT_NEW:           ('WT_NEW', 1),
           pygit2.GIT_STATUS_WT_MODIFIED:      ('WT_MOD', 1),
           pygit2.GIT_STATUS_WT_DELETED:       ('WT_DEL', 1),
           pygit2.GIT_STATUS_IGNORED:          ('IGNORED', 0), # Flags for ignored files
           pygit2.GIT_STATUS_CONFLICTED:       ('CONFLICTED', 1),
           }
   
   def process_git_repo(path):
       result = {}
       repo = pygit2.Repository(pygit2.discover_repository(path))
       for filepath, flags in repo.status().items():
           for status_flag in git_status_flags.keys():
               if git_status_flags[status_flag][1] and flags == status_flag:
                   print([git_status_flags[status_flag][0]], filepath)
       return result
   
   if __name__ == '__main__':
       process_git_repo("./thumbor")
   

   Running on a fresh clone of the Thumbor master branch

   ~ $ git clone git@github.com:thumbor/thumbor.git
   Cloning into 'thumbor'...
   remote: Counting objects: 11947, done.
   remote: Total 11947 (delta 0), reused 0 (delta 0), pack-reused 11947
   Receiving objects: 100% (11947/11947), 38.57 MiB | 2.23 MiB/s, done.
   Resolving deltas: 100% (8124/8124), done.
   

   Should return empty but doesn't

   ~ $ ./test.py
   ['WT_DEL'] tests/fixtures/images/alabama1_ap620é.jpg
   ~ $ ls -l thumbor/tests/fixtures/images/alabama1_ap620é.jpg
   -rw-r--r-- 1 asalvo staff 5319 Jul 23 17:11 thumbor/tests/fixtures/images/alabama1_ap620é.jpg
   

   The same test for encoding shows UTF-8

   >>> sys.getfilesystemencoding()
   'utf-8'
   
3. jdavid

   added a commit that references this issue on Aug 12, 2026

   tests: add regression tests for status_file with non-ASCII paths

   ...

   89c4fa0
4. jdavid commented on Aug 12, 2026

   

   jdavid

   on Aug 12, 2026

   Member

   More actions

   The original bug reported here no longer reproduces on the platforms pygit2 currently supports.

   • Windows: Python switched to UTF-8 filesystem encoding in Python 3.6 (PEP 529), so the old mbcs failure with characters like ä or U+00A0 is gone.
   • macOS: The operating system moved from HFS+ to APFS, which is normalization-preserving. The classic scenario — a Linux-created repo with NFC paths checked out on HFS+ and returned as NFD — no longer occurs.

   I also added regression tests covering Repository.status_file() with non-ASCII paths in test/test_status.py (e.g. täst_é.txt, U+00A0, and both NFC/NFD forms of café.txt).

   That said, there is still an underlying correctness issue: APIs like status_file() convert paths via filesystem encoding (pgit_borrow_fsdefault) rather than treating them as the raw UTF-8 bytes that Git stores. On modern Windows/macOS this happens to work, but it is semantically wrong for repository-internal paths. A proper fix would be to use UTF-8 (with surrogateescape) for string paths and accept bytes for raw path bytes, scoped only to Git-internal APIs while keeping filesystem-path APIs unchanged.

   So the reported symptom is effectively fixed, but the code still has a latent correctness issue worth tracking.

   The analysis and answer above were done with the help of Kimi Code. I keep the issue open to keep record of the correctness problem.
5. jdavid

   added 2 commits that reference this issue on Aug 13, 2026

   Fix status_file (and index/tree path handling) for non-ASCII paths

   ...

   This issue will close once commit 41174c4 is merged into the 'master' branch.

   41174c4

   Fix status_file (and index/tree path handling) for non-ASCII paths

   ...

   This issue will close once commit 5ea06be is merged into the 'master' branch.

   5ea06be
6. jdavid

   added a commit that references this issue on Sep 14, 2026

   Fix status_file (and index/tree path handling) for non-ASCII paths

   ...

   This issue will close once commit 8dba95a is merged into the 'master' branch.

   8dba95a