Using exiv2.exe on msvc and mingw with greek letters on path

[Sparsh-Bansal]

Describe the bug

exiv2 can't read exif/meta information from filename having Chinese/Greek characters in it .
Command: exiv2 .\Εκκρεμότητες\VB5_4591.jpg
It says: Failed to open the file
Also on using pyexiv2 python wrapper for exiv2, Used different encodings like utf-8, GBK, ISO-8859-1, but still the same error -> Failed to open the data source: No such file or directory (errno = 2)

Example of such words -> Βάπτιση, Εκκρεμότητες, Κάλια Μανινάκη

[kmilos]

In order to expect any assistance, please also provide more details about your environment: OS, output of exiv2 -vV, where you got the exiv2 binary, or how you built it yourself...

[Sparsh-Bansal]

I just downloaded the exiv2 (exiv2-0.27.4-2019msvc64) binary for windows from here
Output of exiv2 -vV:

exiv2=0.27.4
platform=windows
compiler=MSVC
bits=64
dll=1
debug=0
cplusplus=201402
version=13.28 (2019/x64)
date=Jun 15 2021
time=15:10:10
processpath=D:\exiv2-0.27.4-2019msvc64\exiv2-0.27.4-2019msvc64\bin
package_name=exiv2
curl=0
executable=D:\exiv2-0.27.4-2019msvc64\exiv2-0.27.4-2019msvc64\bin\exiv2.exe
library=C:\WINDOWS\SYSTEM32\ntdll.dll
library=C:\WINDOWS\System32\KERNEL32.DLL
library=C:\WINDOWS\System32\KERNELBASE.dll
library=C:\WINDOWS\System32\ucrtbase.dll
library=C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll
library=C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll
library=C:\WINDOWS\SYSTEM32\MSVCP140.dll
library=D:\exiv2-0.27.4-2019msvc64\exiv2-0.27.4-2019msvc64\bin\exiv2.dll
library=C:\WINDOWS\System32\PSAPI.DLL
library=C:\WINDOWS\System32\WS2_32.dll
library=C:\WINDOWS\System32\RPCRT4.dll
library=C:\WINDOWS\System32\SHELL32.dll
library=C:\WINDOWS\System32\msvcp_win.dll
library=C:\WINDOWS\System32\USER32.dll
library=C:\WINDOWS\System32\win32u.dll
library=C:\WINDOWS\System32\GDI32.dll
library=C:\WINDOWS\System32\gdi32full.dll
library=C:\WINDOWS\System32\IMM32.DLL
have_inttypes=0
have_libintl=0
have_lensdata=1
have_iconv=0
have_memory=1
have_lstat=0
have_regex=0
have_regex_h=0
have_stdbool=1
have_stdint=1
have_stdlib=0
have_strlib=0
have_strerror_r=0
have_strings_h=0
have_mmap=0
have_munmap=0
have_sys_stat=1
have_unistd_h=0
have_sys_mman=0
have_libz=1
have_xmptoolkit=1
adobe_xmpsdk=0
have_bool=0
have_strings=0
have_sys_types=1
have_unistd=0
have_unicode_path=0
enable_bmff=0
enable_video=0
enable_webready=0
enable_nls=0
use_curl=0
use_ssh=0
config_path=C:\Users\spars\exiv2.ini
xmlns=DICOM:http://ns.adobe.com/DICOM/
xmlns=GPano:http://ns.google.com/photos/1.0/panorama/
xmlns=Iptc4xmpCore:http://iptc.org/std/Iptc4xmpCore/1.0/xmlns/
xmlns=Iptc4xmpExt:http://iptc.org/std/Iptc4xmpExt/2008-02-29/
xmlns=MP:http://ns.microsoft.com/photo/1.2/
xmlns=MPRI:http://ns.microsoft.com/photo/1.2/t/RegionInfo#
xmlns=MPReg:http://ns.microsoft.com/photo/1.2/t/Region#
xmlns=MicrosoftPhoto:http://ns.microsoft.com/photo/1.0/
xmlns=acdsee:http://ns.acdsee.com/iptc/1.0/
xmlns=album:http://ns.adobe.com/album/1.0/
xmlns=asf:http://ns.adobe.com/asf/1.0/
xmlns=audio:http://www.audio/
xmlns=aux:http://ns.adobe.com/exif/1.0/aux/
xmlns=bmsp:http://ns.adobe.com/StockPhoto/1.0/
xmlns=creatorAtom:http://ns.adobe.com/creatorAtom/1.0/
xmlns=crs:http://ns.adobe.com/camera-raw-settings/1.0/
xmlns=crss:http://ns.adobe.com/camera-raw-saved-settings/1.0/
xmlns=dc:http://purl.org/dc/elements/1.1/
xmlns=dcterms:http://purl.org/dc/terms/
xmlns=digiKam:http://www.digikam.org/ns/1.0/
xmlns=dwc:http://rs.tdwg.org/dwc/index.htm
xmlns=exif:http://ns.adobe.com/exif/1.0/
xmlns=exifEX:http://cipa.jp/exif/1.0/
xmlns=expressionmedia:http://ns.microsoft.com/expressionmedia/1.0/
xmlns=iX:http://ns.adobe.com/iX/1.0/
xmlns=jp2k:http://ns.adobe.com/jp2k/1.0/
xmlns=jpeg:http://ns.adobe.com/jpeg/1.0/
xmlns=kipi:http://www.digikam.org/ns/kipi/1.0/
xmlns=lr:http://ns.adobe.com/lightroom/1.0/
xmlns=mediapro:http://ns.iview-multimedia.com/mediapro/1.0/
xmlns=mwg-kw:http://www.metadataworkinggroup.com/schemas/keywords/
xmlns=mwg-rs:http://www.metadataworkinggroup.com/schemas/regions/
xmlns=pdf:http://ns.adobe.com/pdf/1.3/
xmlns=pdfaExtension:http://www.aiim.org/pdfa/ns/extension/
xmlns=pdfaField:http://www.aiim.org/pdfa/ns/field#
xmlns=pdfaProperty:http://www.aiim.org/pdfa/ns/property#
xmlns=pdfaSchema:http://www.aiim.org/pdfa/ns/schema#
xmlns=pdfaType:http://www.aiim.org/pdfa/ns/type#
xmlns=pdfaid:http://www.aiim.org/pdfa/ns/id/
xmlns=pdfx:http://ns.adobe.com/pdfx/1.3/
xmlns=pdfxid:http://www.npes.org/pdfx/ns/id/
xmlns=photoshop:http://ns.adobe.com/photoshop/1.0/
xmlns=plus:http://ns.useplus.org/ldf/xmp/1.0/
xmlns=png:http://ns.adobe.com/png/1.0/
xmlns=rdf:http://www.w3.org/1999/02/22-rdf-syntax-ns#
xmlns=stArea:http://ns.adobe.com/xmp/sType/Area#
xmlns=stDim:http://ns.adobe.com/xap/1.0/sType/Dimensions#
xmlns=stEvt:http://ns.adobe.com/xap/1.0/sType/ResourceEvent#
xmlns=stFnt:http://ns.adobe.com/xap/1.0/sType/Font#
xmlns=stJob:http://ns.adobe.com/xap/1.0/sType/Job#
xmlns=stMfs:http://ns.adobe.com/xap/1.0/sType/ManifestItem#
xmlns=stRef:http://ns.adobe.com/xap/1.0/sType/ResourceRef#
xmlns=stVer:http://ns.adobe.com/xap/1.0/sType/Version#
xmlns=tiff:http://ns.adobe.com/tiff/1.0/
xmlns=video:http://www.video/
xmlns=wav:http://ns.adobe.com/xmp/wav/1.0/
xmlns=xml:http://www.w3.org/XML/1998/namespace
xmlns=xmp:http://ns.adobe.com/xap/1.0/
xmlns=xmpBJ:http://ns.adobe.com/xap/1.0/bj/
xmlns=xmpDM:http://ns.adobe.com/xmp/1.0/DynamicMedia/
xmlns=xmpG:http://ns.adobe.com/xap/1.0/g/
xmlns=xmpGImg:http://ns.adobe.com/xap/1.0/g/img/
xmlns=xmpMM:http://ns.adobe.com/xap/1.0/mm/
xmlns=xmpNote:http://ns.adobe.com/xmp/note/
xmlns=xmpRights:http://ns.adobe.com/xap/1.0/rights/
xmlns=xmpT:http://ns.adobe.com/xap/1.0/t/
xmlns=xmpTPg:http://ns.adobe.com/xap/1.0/t/pg/
xmlns=xmpidq:http://ns.adobe.com/xmp/Identifier/qual/1.0/```

[clanmills]

I think there are two issues which may be causing this:

1. This build has not set -DEXIV2_ENABLE_UNICODE=ON
   I know this from the output have_unicode_path=0 (and because I did the build)

2. The command-line program exiv2 isn't linked to use wmain() and I think you need that to use UNICODE paths.

The sample program sample/exifprint.cpp was modified to use wmain() and other MSVC wide support (tchar, etc). Please build exiv2 with -DEXIV2_ENABLE_UNICODE=1 and see if bin/exifprint.exe can read the metadata in your image when using a Greek path.

Background Explanation

The exiv2 library is passed the path and opens it with good old fopen(). When ENABLE_UNICODE is set and you will be using _wfopen.

I'm a native english speaker and don't understand Greek. Write a simple version of wc to count the bytes in a file using fopen(). When you know how to get that to behave correctly with your Greek path, you should link/build exiv2 in a similar manner.

748 rmills@rmillsm1:~/temp $ cat wc.cpp
#include <stdio.h>

int main(int argc, const char* argv[])
{
    int result = 0;
    if ( argc != 2 ) {
        printf("Syntax: %s path\n",argv[0]);
        result = 1;
    } else {
        FILE* f = fopen (argv[1],"rb") ;
        if ( !f) {
           fprintf(stderr,"Unable to open %s\n",argv[1]);
           result = 2 ;
        } else {
            int wc = 0 ;
            while ( fgetc(f) != EOF ) wc++ ;
            printf("%d\n",wc);
        }
    }
    return result ;
}
749 rmills@rmillsm1:~/temp $ make wc
c++     wc.cpp   -o wc
750 rmills@rmillsm1:~/temp $ ls -l wc.cpp
-rw-r--r--@ 1 rmills  staff  480 13 Nov 06:20 wc.cpp
751 rmills@rmillsm1:~/temp $ /usr/bin/wc wc.cpp
      21      70     480 wc.cpp
752 rmills@rmillsm1:~/temp $ ./wc wc.cpp
480
753 rmills@rmillsm1:~/temp $ 

[clanmills]

I am 100% certain about this. I attach a WIN_UNICODE build of v0.27.5. The build of exifprint.exe understands the Greek path. The build of exiv2.exe does not. This is because exiv2.cpp has not had the treatment given to exifprint.cpp to work with msvc/unicode support (tchar etc).

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin>exiv2 -vV | grep uni
have_unistd_h=0
have_unistd=0
have_unicode_path=1

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin>mkdir Εκκρεμότητες

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin>cd Εκκρεμότητες

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin\Εκκρεμότητες>curl -LO http://clanmills.com/Stonehenge.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6597k  100 6597k    0     0  2650k      0  0:00:02  0:00:02 --:--:-- 2657k

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin\Εκκρεμότητες>..\exiv2.exe -g original/i Stonehenge.jpg
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:38:54
Exif.Photo.SubSecTimeOriginal                Ascii       3  00

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin\Εκκρεμότητες>cd ..

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin>exiv2.exe -g original/i Εκκρεμότητες\Stonehenge.jpg
????e╡?t?te?\Stonehenge.jpg: Failed to open the file

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin>exifprint Εκκρεμότητες\Stonehenge.jpg
Exif.Image.Make                              0x010f Ascii      18  NIKON CORPORATION
Exif.Image.Model                             0x0110 Ascii      12  NIKON D5300
...
Exif.Thumbnail.YCbCrPositioning              0x0213 Short       1  1

C:\Users\rmills\issue_1996\exiv2-0.27.5-2019msvc64\bin>

exiv2-0.27.5-2019msvc64-2021-11-13_07:02:00.zip

[kmilos]

We should probably build w/ unicode/wide char support by default on Windows.

[clanmills]

I agree, @kmilos

I was wrongly of the believe that when #define EXV_UNICODE_PATH was set at compile time (by the CMake option -DEXIV2_ENABLE_WIN_UNICODE), the library was totally wchar and char was not available. I discovered when documenting this in my book that the API is augmented with the wchar variants for ImageFactory::open et al.

About 2015, I modified samples/exifprint.cpp to use the wchar interface when EXV_UNICODE_PATH is used. I added that in response to a bug raised by a Canadian user. I think that may have been something similar to this Greek issue.

I didn't upgrade exiv2.cpp at that time because I didn't want to distribute builds with exclusive wchar API because I wrongly thought the UNICODE=1 and UNICODE=0 libraries were incompatible.

You are correct. We should always set EXV_UNICODE_PATH when building on MSVC.

The other part of the ugly twins (unicode and wmain) is that we should:

1. port the tchar magic from samples/exifprint.cpp into exiv2.cpp and every sample applications
2. add appropriate tests to the test suite
3. review/revise README.md concerning EXV_UNICODE_PATH

I know @postscript-dev will be interested in this matter. However he may not have time to work on this.

@Sparsh-Bansal are you willing to work on porting the tchar magic into exiv2.cpp and other samples?

[clanmills]

I've updated the release script svn://dev.exiv2.org/svn/team/contrib/buildserver/build.sh release as follows:

1. Add option --unicode to set -DEXIV2_ENABLE_WIN_UNICODE=1
2. Hardwire -DEXV_UNICODE_PATH=1 for MSVC builds
3. Hardwire -DEXV_DYNAMIC_RUN_TIME=1 for MSVC builds (evades an expat link issue with static builds)

Please note that this is only used by the release scripts. Changes have not been made to CMakeLists.txt and/or README.md

Setting -DEXIV2_ENABLE_WIN_UNICODE=1 changes the build of the library and the sample exifprint. In this issue, the user's issue concerns the exiv2 command-line program which will require exiv2.cpp to be modified. exiv2.cpp must use tchar and other msvc/unicode magic similar to that found in samples/exifprint.cpp

[postscript-dev]

I have thought a little about this recently.

It has concerned me that we aren't running the CI on the Unicode versions of the functions. I don't know of any problems but I think that it is worth testing to make sure that everything works. One of the limiting factors to this has been that the exiv2 sample program, which we use to test, did not support Unicode. Once the exiv2 sample conversion has taken place, I think that it would be helpful to run the CI on both the Unicode and non-Unicode versions on Windows. Any tests which makes use of the Unicode features, could be skipped when running the non-Unicode Windows version. Both exiv2 --verbose --version --grep unicode and exiv2 --verbose --version --grep platform could help create Python variables which can be used to skip tests as appropriate.

Perhaps the website API docs could be generated from the Windows Unicode build. Currently only the non-Unicode functions are displayed on exiv2.org and I think that it would be useful to include the Unicode versions too.

Another problem that occurred to me this morning is that the lens configuration file doesn't supports Unicode. I haven't looked extensively at it, but I think that there are two issues. First that the path could contain Unicode characters and second that the contents of the file could. The path can be fixed by using _wgetcwd() but the INIReader class doesn't support Unicode. Unless someone want to fix this, I could add something to the manpage to document the current limitations.

I realise that this a substantial list of problems and I am sorry that I don't have time to contribute code to solve them. I had hoped to look at some of this, sometime next year.