iojs 2.1.0 & pkg-config output issues

[alferpal]

Platform: Arch Linux which as some of you will know defaults to python 3.

To solve that, before using configure we use this in our packages to change references from python to python2

  find -type f -exec sed \
    -e 's_^#!/usr/bin/env python$_&2_' \
    -e 's_^\(#!/usr/bin/python2\).[45]$_\1_' \
    -e 's_^#!/usr/bin/python$_&2_' \
    -e 's_^\( *exec \+\)python\( \+.*\)$_\1python2\2_'\
    -e 's_^\(.*\)python\( \+-c \+.*\)$_\1python2\2_'\
    -e "s_'python'_'python2'_" -i {} \;
  find test/ -type f -exec sed 's_python _python2 _' -i {} \;

./configure \
    --prefix=/usr \
    --shared-openssl \
    --shared-zlib \
    --with-intl=system-icu \
    --without-npm 
  make

This used to work with 2.0.x and now this is thrown:

creating  ./icu_config.gypi
{ 'target_defaults': { 'cflags': [],
                       'default_configuration': 'Release',
                       'defines': [],
                       'include_dirs': [],
                       'libraries': [ '-L/usr/lib',
                                      '-lz',
                                      '-lssl',
                                      '-lcrypto',
                                      '',
                                      '-licui18n',
                                      '-licuuc',
                                      '-licudata']},
  'variables': { 'host_arch': 'x64',
                 'icu_gyp_path': 'tools/icu/icu-system.gyp',
                 'icu_small': 'false',
                 'node_install_npm': 'false',
                 'node_prefix': '/usr',
                 'node_shared_http_parser': 'false',
                 'node_shared_libuv': 'false',
                 'node_shared_openssl': 'true',
                 'node_shared_zlib': 'true',
                 'node_tag': '',
                 'node_use_dtrace': 'false',
                 'node_use_etw': 'false',
                 'node_use_lttng': 'false',
                 'node_use_openssl': 'true',
                 'node_use_perfctr': 'false',
                 'openssl_no_asm': 0,
                 'python2': '/usr/bin/python2',
                 'target_arch': 'x64',
                 'uv_parent_path': '/deps/uv/',
                 'uv_use_dtrace': 'false',
                 'v8_enable_gdbjit': 0,
                 'v8_enable_i18n_support': 1,
                 'v8_no_strict_aliasing': 1,
                 'v8_optimized_debug': 0,
                 'v8_random_seed': 0,
                 'v8_use_snapshot': 1,
                 'want_separate_host_toolset': 0}}
creating  ./config.gypi
creating  ./config.mk
Traceback (most recent call last):
  File "tools/gyp_node.py", line 57, in <module>
    run_gyp(gyp_args)
  File "tools/gyp_node.py", line 18, in run_gyp
    rc = gyp.main(args)
  File "./tools/gyp/pylib/gyp/__init__.py", line 526, in main
    return gyp_main(args)
  File "./tools/gyp/pylib/gyp/__init__.py", line 502, in gyp_main
    params, options.check, options.circular_check)
  File "./tools/gyp/pylib/gyp/__init__.py", line 129, in Load
    params['parallel'], params['root_targets'])
  File "./tools/gyp/pylib/gyp/input.py", line 2736, in Load
    variables, includes, depth, check, True)
  File "./tools/gyp/pylib/gyp/input.py", line 386, in LoadTargetBuildFile
    includes, True, check)
  File "./tools/gyp/pylib/gyp/input.py", line 258, in LoadOneBuildFile
    aux_data, includes, check)
  File "./tools/gyp/pylib/gyp/input.py", line 296, in LoadBuildFileIncludesIntoDict
    subdict_path, include)
  File "./tools/gyp/pylib/gyp/input.py", line 2131, in MergeDicts
    MergeDicts(to[k], v, to_file, fro_file)
  File "./tools/gyp/pylib/gyp/input.py", line 2191, in MergeDicts
    MergeLists(to[list_base], v, to_file, fro_file, is_paths, append)
  File "./tools/gyp/pylib/gyp/input.py", line 2048, in MergeLists
    to_item = MakePathRelative(to_file, fro_file, item)
  File "./tools/gyp/pylib/gyp/input.py", line 2023, in MakePathRelative
    if item[-1] == '/':
IndexError: string index out of range while reading includes of /dev/shm/iojs/src/iojs-v2.1.0/node.gyp while trying to load /dev/shm/iojs/src/iojs-v2.1.0/node.gyp

I've been trying to look at the commits on the project, changes on the tools directory and so but I can't find what can trigger this because my knowledge of python and the whole build tool-chain are almost 0. I don't know how all the files call each other and where to track this.

Any hint?

Sorry if this is not the kind of thing to ask. And thank you in advance

[jbergstroem]

Looks like an empty string in your libraries. Can you pass me full output of your pkg-config outputs? (zlib, openssl, icu-i18n). Whitespace is extra important.

[bnoordhuis]

Does the configure step succeed when you remove the --shared-openssl --shared-zlib --with-intl=system-icu switches? If that is the case, can you check which combinations work and which don't?

[jbergstroem]

@bnoordhuis by the order of things its either in openssl or icu-i18n. I'm suspecting icu. The real problem is that we shouldn't really parse output from pkg-config in the first place. Do you perhaps know why gyp expects the libraries (and includedirs) to be a list?

[alferpal]

Yeah, sorry, should have tried that before >,<
I usually compare all deps in the iojs deps folder against the one in the arch's repos to see if there's any potential mismatch, can't see why didn't this time.

Problematic swith: --with-intl=system-icu

And I've never used pkg-config in my life, so I don't know if I've run it correctly:

0 akai@Arch-Akai: /dev/shm/iojs 23:52 ---> pkg-config zlib
0 akai@Arch-Akai: /dev/shm/iojs 23:53 ---> pkg-config openssl 
0 akai@Arch-Akai: /dev/shm/iojs 23:53 ---> pkg-config icu-i18n
0 akai@Arch-Akai: /dev/shm/iojs 23:53 --->

There seems to be no output

[jbergstroem]

@alferpal this is what you want: pkg-config --libs-only-l zlib (and so on). Possibly also provide the output from --libs-only-L if there is any. Again, important to note that we're probably looking for unwanted whitespace here.

[bnoordhuis]

Do you perhaps know why gyp expects the libraries (and includedirs) to be a list?

Some of the generators process them as individual elements. I know the ninja generator does it to escape them properly.

[jbergstroem]

@bnoordhuis ok. Not much to do other than adding more cases to parsing input.

[alferpal]

@jbergstroem :

This is the output:

0 akai@Arch-Akai: /dev/shm/iojs 23:59 ---> pkg-config --libs-only-l zlib
-lz 
0 akai@Arch-Akai: /dev/shm/iojs 0:06 ---> pkg-config --libs-only-l openssl
-lssl -lcrypto 
0 akai@Arch-Akai: /dev/shm/iojs 0:07 ---> pkg-config --libs-only-l icu-il8n
Package icu-il8n was not found in the pkg-config search path.
Perhaps you should add the directory containing `icu-il8n.pc'
to the PKG_CONFIG_PATH environment variable
No package 'icu-il8n' found

[alferpal]

I forgot the one with -L

0 akai@Arch-Akai: /dev/shm/iojs 0:14 ---> pkg-config --libs-only-L zlib 
-L/usr/lib 
0 akai@Arch-Akai: /dev/shm/iojs 0:15 ---> pkg-config --libs-only-L openssl 

0 akai@Arch-Akai: /dev/shm/iojs 0:15 ---> pkg-config --libs-only-L icu-i18n 

0 akai@Arch-Akai: /dev/shm/iojs 0:15 ---> 

[jbergstroem]

@alferpal I think you have a typo for pkg-config --libs-only-l icu-i18n (you had icu-il8n)

[alferpal]

Hard to spot that one, and was in fact going crazy searching a reason because in fact that file existed in my system. Correct output ( I hope)

0 akai@Arch-Akai: /dev/shm/iojs 0:23 ---> pkg-config --libs-only-l icu-i18n
-licui18n -licuuc -licudata 
0 akai@Arch-Akai: /dev/shm/iojs 0:23 ---> 

[jbergstroem]

@alferpal could you gist/pastebin your zlib, openssl and icu-i18n .pc files? (don't paste here, will be too much noise) -- I can probably figure it out with them. Preferably use something like wgetpaste so copy pasting won't modify whitespace.

[alferpal]

http://sprunge.us/VJVZ
http://sprunge.us/gYIM
http://sprunge.us/SQUj

I expect they're in the right format...

[jbergstroem]

@alferpal I can reproduce. The problem is icu. I'll post a patch shortly.

[jbergstroem]

@alferpal try this patch (safeguarding against empty strings):

diff --git configure configure
index 4a30973..d303846 100755
--- configure
+++ configure
@@ -846,10 +846,12 @@ def configure_intl(o):
       sys.exit(1)
     (libs, cflags, libpath) = pkgicu
     # libpath provides linker path which may contain spaces
-    o['libraries'] += [libpath]
+    if libpath:
+      o['libraries'] += [libpath]
     # safe to split, cannot contain spaces
     o['libraries'] += libs.split()
-    o['cflags'] += cflags.split()
+    if cflags:
+      o['cflags'] += cflags.split()
     # use the "system" .gyp
     o['variables']['icu_gyp_path'] = 'tools/icu/icu-system.gyp'
     return

[jbergstroem]

I should probably rewrite this to use configure_library instead..

[alferpal]

Your patch solves the problem. I can get past configure with the icu flag.

[jbergstroem]

Fixed and committed in c5a1009. I'll drive including this in all minor releases that needs a fix. Thanks for the bug report, @alferpal.