Support Hugging Face Transformers

[conker84]

Is there any plan to support Hugging Face Transformers? They are the next (maybe current) big thing in Python
I'm trying to make it work right now but I'm getting the various errors and one of them is:

com.oracle.truffle.api.CompilerDirectives$ShouldNotReachHere: writeByte not implemented

Traceback (most recent call last):
  File "pip", line 8, in <module 'pip'>
  File "main.py", line 70, in main
  File "base_command.py", line 101, in main
  File "base_command.py", line 221, in _main
  File "base_command.py", line 167, in exc_logging_wrapper
  File "req_command.py", line 247, in wrapper
  File "install.py", line 369, in run
  File "resolver.py", line 92, in resolve
  File "resolvers.py", line 481, in resolve
  File "resolvers.py", line 348, in resolve
  File "resolvers.py", line 172, in _add_to_criteria
  File "structs.py", line 151, in __bool__
  "functools", line 590, in wrapper
  File "found_candidates.py", line 155, in __bool__
  File "found_candidates.py", line 143, in <genexpr>
  File "found_candidates.py", line 44, in _iter_built
  File "factory.py", line 279, in iter_index_candidate_infos
  "functools", line 566, in wrapper
  File "package_finder.py", line 889, in find_best_candidate
  "functools", line 566, in wrapper
  File "package_finder.py", line 830, in find_all_candidates
  File "sources.py", line 134, in page_candidates
  File "package_finder.py", line 790, in process_project_url
  File "collector.py", line 577, in fetch_response
  File "collector.py", line 481, in _get_index_content
  File "collector.py", line 138, in _get_simple_response
  File "sessions.py", line 600, in get
  File "session.py", line 518, in request
  File "sessions.py", line 587, in request
  File "sessions.py", line 745, in send
  File "models.py", line 899, in content
  File "models.py", line 816, in generate
  File "response.py", line 573, in stream
  File "response.py", line 516, in read
  File "filewrapper.py", line 96, in read
  File "filewrapper.py", line 76, in _close
  File "controller.py", line 353, in cache_response
  File "controller.py", line 274, in _cache_set
  File "serialize.py", line 70, in dumps
  File "__init__.py", line 38, in packb
  File "fallback.py", line 883, in pack
  File "fallback.py", line 862, in _pack
  File "fallback.py", line 968, in _pack_map_pairs
  File "fallback.py", line 862, in _pack
  File "fallback.py", line 968, in _pack_map_pairs
com.oracle.truffle.api.CompilerDirectives$ShouldNotReachHere: writeByte not implemented

[msimacek]

Could you please post the package name and what command fails for you?

[conker84]

Hi sure:
I executed this from graalpython venv:
venv/bin/pip install transformers

[msimacek]

I tried it and I got a different error - it couldn't build tokenizers dependency because it uses rust. We currently don't support rust packages because Sulong, which we use to execute native dependencies, doesn't support passing rust objects to native libraries (which is necessary to support rust standard library). The issue is being worked by the Sulong team.

[conker84]

Yes I got your same error at first then the CLI suggested upgrading pip so I did
venv/bin/pip install --upgrade pip

and then I got the last error, so I thought that by upgrading pip the last error was the real one.
oki thanks
Do you know where I can check the progress on this issue? A GitHub issue maybe?

[msimacek]

There's no github issue, but I can keep this one updated. But there's more things that will need to happen for this package to work - for example, we currently don't support recent numpy versions that are required by the package. That's also being worked on.

[conker84]

@msimacek any update on this?

[msimacek]

The numpy update is almost finished. I have no update from Sulong team on the Rust issue.

[conker84]

Do you know if it is in their (Sulong team) interest to fix it?

[msimacek]

Yes, it is. And they are working on it. It's just a very complex change.

[conker84]

thank you so much for the info @msimacek

[conker84]

Hi @msimacek do you have any update on this?

[msimacek]

No, still in progress

[conker84]

@msimacek hi! Any progress on this?

[msimacek]

Hi, we now support newer numpy, which was one of the items that was blocking this. The rust issue is still unsolved.

[conker84]

@msimacek Hi! Do you know the status of the rust issue? Thank you so much!