Release a Windows wheel?

[ankostis]

Is it possible to build it also on Windows?
Maybe even use Appveyor for automated builds? (but I have no experience in building native libraries on Windows).

[martindurant]

There is a windows conda package, is that any use to you?
Am I right in thinking that, in order to build a wheel for windows myself, I would need a windows system?

[mrocklin]

The limitation here might be Numba, which requires LLVM. I recommend asking @seibert if an LLVM Windows wheel is feasible. Generally wheels weren't intended to package non-python dependencies super well.

…

On Wed, Mar 22, 2017 at 8:43 AM, Martin Durant ***@***.***> wrote: There is a windows conda package, is that any use to you? Am I right in thinking that, in order to build a wheel for windows myself, I would need a windows system? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#110 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszAsWxcQykWZaQbCdspLKOXJS80_9ks5roRdjgaJpZM4Mk9W8> .

[ankostis]

Am I right in thinking that, in order to build a wheel for windows myself, I would need a windows system?

Only at the beginning, to make it work - afterwards, probably Appveyor can produce the wheels.

There is a windows conda package, is that any use to you?

Nope.
I'm using vanilla python 3.6 (actually as packaged by WinPython distribution).

The limitation here might be Numba, which requires LLVM.

Gohlke packages numba for Windows](http://www.lfd.uci.edu/~gohlke/pythonlibs/#numba),
which is included also in WinPython, so I suppose there shouldn't be any problem.

[martindurant]

I have installed a windows VM and can look into this when I have the time (not immediately); although I am certain there are people with more windows experience than myself that might help.

[stonebig]

to create a wheel, I would do:

• use a python distro with numpy already installed,
• unzip "fastparquet-master" from github and launch "python setup.py bdist_wheel"

numba package requires llvmlite package, which includes LLVM. so:

• there is no special thing to do (at least on Windows10/VS2015+/python-3.5/3.6)
• I suspect fastparquet is not compatible with older Python versions on Windows (because of Visual Studio 2010 not being C++11 friendly)

[martindurant]

@stonebig , are you happy to try this, to see if you can build something pip-installable?

[stonebig]

I did, from April 15th tree. As I compiled from Python-3.6.1, the 3.6 wheels may not be convenient for python-3.6.0 (see https://bugs.python.org/issue29943).

result there: https://github.com/stonebig/winpython_afterdoc/tree/master/various_wheels/fastparquet

pandas or scikit-learn team do know how to generate wheel from appveyor better than me. you may ask them

nota : this doesn't work for pyarrow

[martindurant]

OK, I'll be sure to give it a go when next I do a release, which shouldn't be too far off.

[stonebig]

checking my own dog food, I fall directly on this:

from fastparquet import ParquetFile
pf = ParquetFile(r'C:\WinPython\a\fastparquet-master20170415\fastparquet-master\test-data\snappy-nation.impala.parquet')
df = pf.to_pandas()

Traceback (most recent call last):

  File "<ipython-input-1-f3f64ac893d5>", line 1, in <module>
    from fastparquet import ParquetFile

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\fastparquet\__init__.py", line 14, in <module>
    from .thrift_structures import parquet_thrift

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\fastparquet\thrift_structures.py", line 7, in <module>
    parquet_thrift = thriftpy.load(THRIFT_FILE, module_name="parquet_thrift")  # pylint: disable=invalid-name

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\thriftpy\parser\__init__.py", line 30, in load
    include_dir=include_dir)

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\thriftpy\parser\parser.py", line 496, in parse
    url_scheme))

ThriftParserError: ThriftPy does not support generating module with path in protocol 'c'

I'm lost in thoughts : what Thriftpy is supposed to do here ?

[stonebig]

same error the other way:

import pandas as pd
import numpy as np
df = pd.DataFrame({ 'A' : 1.,
                     'B' : pd.Timestamp('20130102'),
                     'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
                     'D' : np.array([3] * 4,dtype='int32'),
                     'E' : pd.Categorical(["test","train","test","train"]),
                      'F' : 'foo' })
from fastparquet import write
write('outfile.parq', df)
write('outfile2.parq', df, row_group_offsets=[0, 10000, 20000],
      compression='GZIP', file_scheme='hive')
      
Traceback (most recent call last):

  File "<ipython-input-1-680fbbc82549>", line 10, in <module>
    from fastparquet import write

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\fastparquet\__init__.py", line 14, in <module>
    from .thrift_structures import parquet_thrift

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\fastparquet\thrift_structures.py", line 7, in <module>
    parquet_thrift = thriftpy.load(THRIFT_FILE, module_name="parquet_thrift")  # pylint: disable=invalid-name

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\thriftpy\parser\__init__.py", line 30, in load
    include_dir=include_dir)

  File "C:\WinPython\basedir36\buildQt5\winpython-64bit-3.6.x.1\python-3.6.1.amd64\lib\site-packages\thriftpy\parser\parser.py", line 496, in parse
    url_scheme))

ThriftParserError: ThriftPy does not support generating module with path in protocol 'c'

I can compile fastpartquet, but I still have no exact clue how it must be used

[stonebig]

there is a workaround here : Thriftpy/thriftpy#234 (comment)

[stonebig]

after applying the suggested one-liner, this works :

import pandas as pd
import numpy as np
df = pd.DataFrame({ 'A' : 1.,
                     'B' : pd.Timestamp('20130102'),
                     'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
                     'D' : np.array([3] * 4,dtype='int32'),
                     'E' : pd.Categorical(["test","train","test","train"]),
                      'F' : 'foo' })
from fastparquet import write
write('outfile.parq', df) 
      

and this also:

from fastparquet import ParquetFile
pf = ParquetFile(r'C:\WinPython\a\fastparquet-master20170415\fastparquet-master\test-data\snappy-nation.impala.parquet')
df = pf.to_pandas()

[martindurant]

That's a horrible workaround! :)
Actually thriftpy.parser.parser was partially rewritten recently - you could try downgrading it one or more minor versions.

[stonebig]

you mean I should redo a wheel from Thriftpy github master tree ? I was using version Thriftpy-0.3.9 ...
ok, I re-did a Thriftpy-0.3.9 wheel with the one-liner.

if Thriftpy is a dead project, what shall we use ?

[martindurant]

It looks like the code in 0.3.8 was different. There are a couple of further commits on that part of the code since 0.3.9 too, so it maybe that your issue is fixed in master.