Can't be built on macOS

[HiromuHota]

Fugashi cannot be built hence cannot be installed on macOS.

$ pip install fugashi
Collecting fugashi
  Using cached https://files.pythonhosted.org/packages/2e/88/156c51c78ee4ccfd54000e720f0c9814d073993b4e1f5d400d01416ddb6d/fugashi-0.1.10.tar.gz
Requirement already satisfied: Cython in /Users/hiromu/miniconda3/envs/fonduer-dev/lib/python3.7/site-packages (from fugashi) (0.29.16)
Building wheels for collected packages: fugashi
  Building wheel for fugashi (setup.py) ... error
  ERROR: Complete output from command /Users/hiromu/miniconda3/envs/fonduer-dev/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-install-adei_m45/fugashi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-wheel-7supwlz0 --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_ext
  cythoning fugashi/fugashi.pyx to fugashi/fugashi.c
  /Users/hiromu/miniconda3/envs/fonduer-dev/lib/python3.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-install-adei_m45/fugashi/fugashi/fugashi.pyx
    tree = Parsing.p_module(s, pxd, full_module_name)
  building 'fugashi' extension
  creating build/temp.macosx-10.7-x86_64-3.7
  creating build/temp.macosx-10.7-x86_64-3.7/fugashi
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/hiromu/miniconda3/envs/fonduer-dev/include -arch x86_64 -I/Users/hiromu/miniconda3/envs/fonduer-dev/include -arch x86_64 -I/usr/local/Cellar/mecab/0.996/include -I/Users/hiromu/miniconda3/envs/fonduer-dev/include/python3.7m -c fugashi/fugashi.c -o build/temp.macosx-10.7-x86_64-3.7/fugashi/fugashi.o
  In file included from fugashi/fugashi.c:598:
  /usr/local/Cellar/mecab/0.996/include/mecab.h:380:47: warning: this function declaration is not a prototype [-Wstrict-prototypes]
    MECAB_DLL_EXTERN const char*   mecab_version();
                                                ^
                                                 void
  /usr/local/Cellar/mecab/0.996/include/mecab.h:520:54: warning: this function declaration is not a prototype [-Wstrict-prototypes]
    MECAB_DLL_EXTERN mecab_lattice_t *mecab_lattice_new();
                                                       ^
                                                        void
  fugashi/fugashi.c:4828:13: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
    __pyx_t_5 = mecab_nbest_sparse_tostr(__pyx_v_self->c_tagger, __pyx_t_3, __pyx_t_4);
              ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  fugashi/fugashi.c:9121:26: warning: code will never be executed [-Wunreachable-code]
                  module = PyImport_ImportModuleLevelObject(
                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  4 warnings generated.
  creating build/lib.macosx-10.7-x86_64-3.7
  gcc -bundle -undefined dynamic_lookup -L/Users/hiromu/miniconda3/envs/fonduer-dev/lib -arch x86_64 -L/Users/hiromu/miniconda3/envs/fonduer-dev/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.7/fugashi/fugashi.o -L/usr/local/Cellar/mecab/0.996/lib -lmecab -lstdc++ -o build/lib.macosx-10.7-x86_64-3.7/fugashi.cpython-37m-darwin.so
  clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
  ld: library not found for -lstdc++
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for fugashi
  Running setup.py clean for fugashi
Failed to build fugashi
Installing collected packages: fugashi
  Running setup.py install for fugashi ... error
    ERROR: Complete output from command /Users/hiromu/miniconda3/envs/fonduer-dev/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-install-adei_m45/fugashi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-record-y_d6t5au/install-record.txt --single-version-externally-managed --compile:
    ERROR: running install
    running build
    running build_ext
    skipping 'fugashi/fugashi.c' Cython extension (up-to-date)
    building 'fugashi' extension
    creating build/temp.macosx-10.7-x86_64-3.7
    creating build/temp.macosx-10.7-x86_64-3.7/fugashi
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/hiromu/miniconda3/envs/fonduer-dev/include -arch x86_64 -I/Users/hiromu/miniconda3/envs/fonduer-dev/include -arch x86_64 -I/usr/local/Cellar/mecab/0.996/include -I/Users/hiromu/miniconda3/envs/fonduer-dev/include/python3.7m -c fugashi/fugashi.c -o build/temp.macosx-10.7-x86_64-3.7/fugashi/fugashi.o
    In file included from fugashi/fugashi.c:598:
    /usr/local/Cellar/mecab/0.996/include/mecab.h:380:47: warning: this function declaration is not a prototype [-Wstrict-prototypes]
      MECAB_DLL_EXTERN const char*   mecab_version();
                                                  ^
                                                   void
    /usr/local/Cellar/mecab/0.996/include/mecab.h:520:54: warning: this function declaration is not a prototype [-Wstrict-prototypes]
      MECAB_DLL_EXTERN mecab_lattice_t *mecab_lattice_new();
                                                         ^
                                                          void
    fugashi/fugashi.c:4828:13: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
      __pyx_t_5 = mecab_nbest_sparse_tostr(__pyx_v_self->c_tagger, __pyx_t_3, __pyx_t_4);
                ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    fugashi/fugashi.c:9121:26: warning: code will never be executed [-Wunreachable-code]
                    module = PyImport_ImportModuleLevelObject(
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    4 warnings generated.
    creating build/lib.macosx-10.7-x86_64-3.7
    gcc -bundle -undefined dynamic_lookup -L/Users/hiromu/miniconda3/envs/fonduer-dev/lib -arch x86_64 -L/Users/hiromu/miniconda3/envs/fonduer-dev/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.7/fugashi/fugashi.o -L/usr/local/Cellar/mecab/0.996/lib -lmecab -lstdc++ -o build/lib.macosx-10.7-x86_64-3.7/fugashi.cpython-37m-darwin.so
    clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
    ld: library not found for -lstdc++
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "/Users/hiromu/miniconda3/envs/fonduer-dev/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-install-adei_m45/fugashi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-record-y_d6t5au/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/6j/mctnhv6n2zx4zf8c657hbr2w0000gn/T/pip-install-adei_m45/fugashi/

I installed Mecab using Homebrew

$ brew info mecab
mecab: stable 0.996 (bottled)
Yet another part-of-speech and morphological analyzer
https://taku910.github.io/mecab/
Conflicts with:
  mecab-ko (because both install mecab binaries)
/usr/local/Cellar/mecab/0.996 (20 files, 4.2MB) *
  Poured from bottle on 2019-03-25 at 14:19:36
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/mecab.rb
==> Analytics
install: 7,671 (30 days), 24,244 (90 days), 50,937 (365 days)
install-on-request: 653 (30 days), 2,051 (90 days), 10,150 (365 days)
build-error: 0 (30 days)

[polm]

Thanks for the bug report. I had a bit of trouble understanding what's going on here because the actual error doesn't seem to be in the output above; the only detail is this:

ld: library not found for -lstdc++

It looks like libstdc++ is a little weird on OSX, and at some point was deprecated in favor of libc++? I have attempted a fix that sets compiler flags to use the other library based on this StackOverflow post. I'm not sure it'll work since it seems to refer to a newer version of OSX than you're using - it looks like you have 10.7, but this is for 10.9+. However, it seems like libstdc++ should be normally usable before 10.9 so I'm still not entirely sure what's going on...

I have pushed an attempted fix to the osx-fix branch. Could you check that out, try a python setup.py install, and tell me what the output is? If it doesn't work you can try removing the -mmacosx-version-min=10.9 bits from setup.py.

[HiromuHota]

Thank you for your quick reply.

it looks like you have 10.7, but this is for 10.9+

For the record, my macOS version is 10.15.4.

I tried the osx-fix branch and failed.

$ python setup.py install
running install
running bdist_egg
running egg_info
writing fugashi.egg-info/PKG-INFO
writing dependency_links to fugashi.egg-info/dependency_links.txt
writing requirements to fugashi.egg-info/requires.txt
writing top-level names to fugashi.egg-info/top_level.txt
reading manifest file 'fugashi.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'fugahsi/mecab.pxd'
writing manifest file 'fugashi.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_ext
skipping 'fugashi/fugashi.c' Cython extension (up-to-date)
building 'fugashi' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/hiromu/miniconda3/envs/fonduer-dev/include -arch x86_64 -I/Users/hiromu/miniconda3/envs/fonduer-dev/include -arch x86_64 -I/usr/local/Cellar/mecab/0.996/include -I/Users/hiromu/miniconda3/envs/fonduer-dev/include/python3.7m -c fugashi/fugashi.c -o build/temp.macosx-10.7-x86_64-3.7/fugashi/fugashi.o -std=c++11 -mmacosx-version-min=10.9
error: invalid argument '-std=c++11' not allowed with 'C'
error: command 'gcc' failed with exit status 1

The error message says '-std=c++11' not allowed with 'C', so I modified the setup.py as in #17 and build passed and installed without an error.

$ python setup.py install
running install
running bdist_egg
running egg_info
writing fugashi.egg-info/PKG-INFO
writing dependency_links to fugashi.egg-info/dependency_links.txt
writing requirements to fugashi.egg-info/requires.txt
writing top-level names to fugashi.egg-info/top_level.txt
reading manifest file 'fugashi.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'fugahsi/mecab.pxd'
writing manifest file 'fugashi.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_ext
skipping 'fugashi/fugashi.c' Cython extension (up-to-date)
creating build/bdist.macosx-10.7-x86_64/egg
copying build/lib.macosx-10.7-x86_64-3.7/fugashi.cpython-37m-darwin.so -> build/bdist.macosx-10.7-x86_64/egg
creating stub loader for fugashi.cpython-37m-darwin.so
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fugashi.py to fugashi.cpython-37.pyc
creating build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying fugashi.egg-info/PKG-INFO -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying fugashi.egg-info/SOURCES.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying fugashi.egg-info/dependency_links.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying fugashi.egg-info/requires.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying fugashi.egg-info/top_level.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
writing build/bdist.macosx-10.7-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.fugashi.cpython-37: module references __file__
creating 'dist/fugashi-0.1.10-py3.7-macosx-10.7-x86_64.egg' and adding 'build/bdist.macosx-10.7-x86_64/egg' to it
removing 'build/bdist.macosx-10.7-x86_64/egg' (and everything under it)
Processing fugashi-0.1.10-py3.7-macosx-10.7-x86_64.egg
removing '/Users/hiromu/miniconda3/envs/fonduer-dev/lib/python3.7/site-packages/fugashi-0.1.10-py3.7-macosx-10.7-x86_64.egg' (and everything under it)
creating /Users/hiromu/miniconda3/envs/fonduer-dev/lib/python3.7/site-packages/fugashi-0.1.10-py3.7-macosx-10.7-x86_64.egg
Extracting fugashi-0.1.10-py3.7-macosx-10.7-x86_64.egg to /Users/hiromu/miniconda3/envs/fonduer-dev/lib/python3.7/site-packages
fugashi 0.1.10 is already the active version in easy-install.pth

Installed /Users/hiromu/miniconda3/envs/fonduer-dev/lib/python3.7/site-packages/fugashi-0.1.10-py3.7-macosx-10.7-x86_64.egg
Processing dependencies for fugashi==0.1.10
Searching for Cython==0.29.16
Best match: Cython 0.29.16
Adding Cython 0.29.16 to easy-install.pth file
Installing cygdb script to /Users/hiromu/miniconda3/envs/fonduer-dev/bin
Installing cython script to /Users/hiromu/miniconda3/envs/fonduer-dev/bin
Installing cythonize script to /Users/hiromu/miniconda3/envs/fonduer-dev/bin

Using /Users/hiromu/miniconda3/envs/fonduer-dev/lib/python3.7/site-packages
Finished processing dependencies for fugashi==0.1.10

[HiromuHota]

The test also passed.

$ pytest fugashi/tests/test_basic.py 
============================================================== test session starts ==============================================================
platform darwin -- Python 3.7.3, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: /Users/hiromu/workspace/fugashi
plugins: monkeytype-1.0.1
collected 10 items                                                                                                                              

fugashi/tests/test_basic.py .........s                                                                                                    [100%]

========================================================= 9 passed, 1 skipped in 0.04s ==========================================================

[polm]

The error message says '-std=c++11' not allowed with 'C', so I modified the setup.py as in #17 and build passed and installed without an error.

Great, thank you! I'll merge that, clean it up a bit (the code should probably be in fugashi_util.py), and do a release.

For the record, my macOS version is 10.15.4.

Huh, OK. I assumed it was 10.7 based on the following line:

creating build/temp.macosx-10.7-x86_64-3.7

But I guess that's a version number for something else.

[polm]

Released a new version where this should be fixed, let me know if you continue to have issues.

[HiromuHota]

Thank you. Now I'm able to build it without an error.

creating build/temp.macosx-10.7-x86_64-3.7

This was actually puzzling me too, but now I have a better view of what was happening.
According to pandas-dev/pandas#24274,

current 64/32b universal builds of python, which target macOS 10.7

I was in a conda environment when I was building fugashi==0.1.10, which may have targeted macOS 10.7.
As the above PR did, if macOS 10.9 is targeted where possible, this error does not happen.

[polm]

Ah, that makes sense about Anaconda - thanks for the explanation!