python3 system tests fail to use gRPC

[dhermes]

@tseaver Can you confirm? I'm seeing basic imports failing.

For example:

======================================================================
ERROR: test_log_struct (logging_.TestLogging)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/logging_.py", line 149, in test_log_struct
    entries, _ = _retry_backoff(_has_entries, logger.list_entries)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/logging_.py", line 37, in _retry_backoff
    from grpc.beta.interfaces import StatusCode
ImportError: cannot import name 'StatusCode'

[tseaver]

@dhermes Looks like they need to dump the PYTHONPATH=_testing bit.

[dhermes]

But grpcio successfully installs in the Python 3 environment otherwise?

UPDATE: Or you're saying that the GAX version wouldn't be used if the import failed?

[tseaver]

I've run the Py3k system tests with grpcio installed (and the gax-* libraries), but suppressing having _testing on the PYTHONPATH.

[dhermes]

On my machine pip3 install grpcio fails with

could not find grpc_python_plugin (protoc plugin for GRPC Python)

I could put grpc_python_plugin on my path but that won't help Travis.

[tseaver]

I'm beginning to think we should back out all the "just pretend grpc works" stuff we've done (make it a pure extra), disabled by default. I also want to shoot the _testing directory in the head.

[dhermes]

The _testing directory was just there for unit tests (so imports didn't break), never intended to be used in system tests. The real culprit is grpcio.

[tseaver]

I think unit tests don't really need it, either: the bigtable tests could just use `unittest2.skipUnless' like the GAX-related tests in logging and pubsub.

[dhermes]

At the time it was added, there was no Travis environment where grpcio installed, so that was the only way. I'm fine with your suggestion given the current state of things.

[tseaver]

W00t! https://pypi.python.org/pypi/grpcio/1.0.0rc1/#downloads has wheels for Python 3.4 on Linux/Mac/Windows!

[dhermes]

I'm surprised they're at / near a 1.0

[tseaver]

I wonder if pinning the RC would unbreak Travis system-tests3 and Appveyor.

[nathanielmanistaatgoogle]

Apologies for the gap in 3.4-on-Windows.

[daspecster]

@tseaver https://ci.appveyor.com/project/daspecster/gcloud-python-utjkc/build/job/ibd6g2k1a4gkumjo

Installed c:\users\appveyor\appdata\local\temp\1\easy_install-idvc8i\grpcio-1.0.0rc1\.eggs\markupsafe-0.23-py2.7-win32.egg
warning: no files found matching '*.h' under directory 'src\python\grpcio\grpc'
warning: no files found matching '*.python' under directory 'src\python\grpcio\grpc'
warning: no previously-included files matching '*.so' found under directory 'src\python\grpcio\grpc\_cython'
warning: no previously-included files matching '*.pyd' found under directory 'src\python\grpcio\grpc\_cython'
warning: no files found matching 'src\python\grpcio\precompiled.py'
cygrpc.c
include\grpc/impl/codegen/compression_types.h(38) : fatal error C1083: Cannot open include file: 'stdbool.h': No such file or directory
a.c

[soltanmm]

@daspecster Looks like Appveyor is trying to build from source on Windows for Python 2.7.

https://pypi.python.org/packages/3f/81/b14dc4e143d1eb0efabb0c8a4d88fafc6798d6392e26867f4e45a44d617a/grpcio-1.0.0rc1.tar.gz#md5=ce2462a64aa86670696a50c6d947d5e6

It should be downloading the wheels...

[dhermes]

I think the issue is that pip install grpcio won't grab he RC version so it's trying a pre-1.0 release. Either that or your wheel doesn't support Windows?

[soltanmm]

Note that here you'll need to use the --pre flag to get 1.0.0rc1 unless you want to be grabbing it during nosetests as part of the command's potentially problematic gathering of (constrained to egg or .tar.gz?) dependencies (AFAICT).

[daspecster]

Thanks @soltanmm, I actually just forced the version with pip install grpcio==1.0.0rc1 for now to try it in my branch.

https://ci.appveyor.com/project/daspecster/gcloud-python-pjqjn

[daspecster]

Some success and some more failures :(
https://ci.appveyor.com/project/daspecster/gcloud-python-pjqjn/build/job/1gpl7vqkrafyvjcu

[dhermes]

64-bit 3.4 fails with

    include\grpc/impl/codegen/compression_types.h(38) : fatal error C1083: Cannot open include file: 'stdbool.h': No such file or directory
    creating Users
    creating Users\appveyor

while both versions of 3.5 fail with an OpenSSL issue

[soltanmm]

IIRC grpcio can be built within a MinGW 64-bit Windows environment with vanilla Python 3.4 (NOT e.g. MSYS2's) if configured to use the GCC toolchain via e.g. -c mingw32. grpcio-tools will fail with linker errors that might be alleviated by compiling with TDM-GCC (haven't tried that yet) or finishing up C++98-ifying the plug-in code enough for MSVC 2010 to handle it.

It looks like y'all just need grpcio, not grpcio-tools, though. Is that the case?

EDIT: Turns out I recalled incorrectly and that grpcio also fails with linker errors.

[tseaver]

@soltanmm Correct: users of gcloud-python only need grpcio: developers may need grpcio-tools for regenerating proto-based files, but we don't have any Windows-based developers, and we don't do that task via Windows CI.

[tseaver]

Via #2065, #2085.