Docker train failure: ModuleNotFoundError: No module named 'setuptools._distutils' #3295

lissyx · 2020-08-31T11:57:56Z

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 331, in run
    resolver.resolve(requirement_set)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
    reqs = backend.get_requires_for_build_wheel()
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/wrappers.py", line 152, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/wrappers.py", line 255, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py", line 63, in _build_backend
    obj = import_module(mod_path)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/dist-packages/setuptools/__init__.py", line 5, in <module>
    import distutils.core
  File "/tmp/pip-build-env-4floway6/overlay/lib/python3.6/site-packages/_distutils_hack/__init__.py", line 82, in create_module
    return importlib.import_module('._distutils', 'setuptools')
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools._distutils'

The text was updated successfully, but these errors were encountered:

DewiBrynJones · 2020-08-31T13:53:42Z

FWIW I'm getting this error as well when trying to rebuild DeepSpeech from my docker scripts for 0.7.4

https://github.com/techiaith/docker-deepspeech-cy/blob/deepspeech-0.7.4/Makefile#L19

it used to build! If I change to 0.8.2 or latest on master, the same bug appears.

DewiBrynJones · 2020-08-31T14:00:37Z

It's an issue at pypa setuptools...

pypa/setuptools#2353

Hopefully this doesn't require any changes or workarounds on DeepSpeech's side.

lissyx · 2020-08-31T14:13:50Z

FWIW I'm getting this error as well when trying to rebuild DeepSpeech from my docker scripts for 0.7.4

Unfortunately, we can't upgrade this release. Also, forcing setuptools==50.0.0 get past this error for me, but the deepspeech_training package does not get installed, it seems.

DewiBrynJones · 2020-08-31T19:39:41Z

It builds successfully for me if you instead specify the last version of setuptools before today's 50.0.0 release:

RUN pip3 install --upgrade pip==20.0.2 wheel==0.34.2 setuptools==49.6.0

lissyx · 2020-08-31T20:08:08Z

Yeah there are a lot of reports of bad breakages because of 50.0.0. Unfortunately we wont be able to issue that kind of fix for older release so i prefer to wait and see for pypa/setuptools feedback on all those issues.

lissyx · 2020-09-01T17:14:19Z

People report that SETUPTOOLS_USE_DISTUTILS=stdlib env var helps.

DewiBrynJones · 2020-09-01T19:12:37Z

People report that SETUPTOOLS_USE_DISTUTILS=stdlib env var helps.

in the case of Docker, that would still need a change in the Dockerfile (you can't set and pass an environment variable in a docker build command) . So it's not a fix for older releases or even the current 0.8.2 stable release. For the next release, you might as well update the setuptools version number (?)

lissyx · 2020-09-02T08:46:54Z

People report that SETUPTOOLS_USE_DISTUTILS=stdlib env var helps.

in the case of Docker, that would still need a change in the Dockerfile (you can't set and pass an environment variable in a docker build command) . So it's not a fix for older releases or even the current 0.8.2 stable release. For the next release, you might as well update the setuptools version number (?)

There's so much mess, I really don't get. Some of what seems to be one of the maintainer's of the project seems to imply there is no bug here, but seeing how this explodes everywhere it looks suspicious.

As to the setuptools version, in my testing it was not enough.

lissyx · 2020-09-02T08:47:28Z

This really comes at a very bad time, we have so much to do that is equally important, I can't spare the time to try and debug pip/setuptools as well ...

lissyx · 2020-09-02T09:06:53Z

@DewiBrynJones Seems like 50.0.3 works

lissyx · 2020-09-02T09:07:14Z

Except the deepspeech_training package does not get installed

DewiBrynJones · 2020-09-02T09:08:41Z

Great!

I got the 0.8.2 release to build and train models with the following workaround to use 49.6.3:

I take my own copy of Dockerfile.train.tmpl and update the setuptools version: https://github.com/techiaith/docker-deepspeech-cy/blob/deepspeech-0.8.2/Dockerfile.train.tmpl#L44
Then copy into cloned DS and build: https://github.com/techiaith/docker-deepspeech-cy/blob/deepspeech-0.8.2/Makefile#L21

I could try and debug pip/setuptools the latest on master (?) if it's any help (although I'm no expert)

lissyx · 2020-09-02T09:14:28Z

It looks like we are facing https://github.com/pypa/setuptools/issues/2350:

Step 16/22 : RUN DS_NODECODER=y DS_NOTENSORFLOW=y pip3 install --upgrade .
 ---> Using cache
 ---> 5c5492c4b3b1
Step 17/22 : RUN find / -type f | grep deepspeech_training
 ---> Running in dc44939c8fac
/usr/local/lib/python3.6/dist-packages/deepspeech_training/GRAPH_VERSION
/usr/local/lib/python3.6/dist-packages/deepspeech_training/VERSION
/usr/local/lib/python3.6/dist-packages/deepspeech_training/__init__.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/__pycache__/__init__.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/__pycache__/evaluate.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/__pycache__/train.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/evaluate.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/train.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__init__.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/__init__.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/audio.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/augmentations.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/check_characters.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/checkpoints.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/config.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/downloader.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/evaluate_tools.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/feeding.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/flags.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/gpu.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/helpers.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/importers.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/logging.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/sample_collections.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/stm.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/taskcluster.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/__pycache__/text.cpython-36.pyc
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/audio.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/augmentations.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/check_characters.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/checkpoints.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/config.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/downloader.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/evaluate_tools.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/feeding.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/flags.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/gpu.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/helpers.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/importers.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/logging.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/sample_collections.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/stm.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/taskcluster.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training/util/text.py
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/INSTALLER
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/LICENSE
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/METADATA
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/RECORD
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/REQUESTED
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/WHEEL
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/direct_url.json
/usr/local/lib/python3.6/dist-packages/deepspeech_training-0.9.0a7.dist-info/top_level.txt
/DeepSpeech/training/deepspeech_training/GRAPH_VERSION
/DeepSpeech/training/deepspeech_training/VERSION
/DeepSpeech/training/deepspeech_training/__init__.py
/DeepSpeech/training/deepspeech_training/evaluate.py
/DeepSpeech/training/deepspeech_training/train.py
/DeepSpeech/training/deepspeech_training/util/__init__.py
/DeepSpeech/training/deepspeech_training/util/audio.py
/DeepSpeech/training/deepspeech_training/util/augmentations.py
/DeepSpeech/training/deepspeech_training/util/check_characters.py
/DeepSpeech/training/deepspeech_training/util/checkpoints.py
/DeepSpeech/training/deepspeech_training/util/config.py
/DeepSpeech/training/deepspeech_training/util/downloader.py
/DeepSpeech/training/deepspeech_training/util/evaluate_tools.py
/DeepSpeech/training/deepspeech_training/util/feeding.py
/DeepSpeech/training/deepspeech_training/util/flags.py
/DeepSpeech/training/deepspeech_training/util/gpu.py
/DeepSpeech/training/deepspeech_training/util/helpers.py
/DeepSpeech/training/deepspeech_training/util/importers.py
/DeepSpeech/training/deepspeech_training/util/logging.py
/DeepSpeech/training/deepspeech_training/util/sample_collections.py
/DeepSpeech/training/deepspeech_training/util/stm.py
/DeepSpeech/training/deepspeech_training/util/taskcluster.py
/DeepSpeech/training/deepspeech_training/util/text.py
Removing intermediate container dc44939c8fac

 ---> c4ad9c34f035
Step 16/22 : RUN DS_NODECODER=y DS_NOTENSORFLOW=y pip3 install --upgrade -e .
 ---> Using cache            
 ---> 807b068dd581                                                      
Step 17/22 : RUN find / -type f | grep deepspeech_training        
 ---> Running in 483fa03cf9ff                                         
/DeepSpeech/training/deepspeech_training/GRAPH_VERSION                                        
/DeepSpeech/training/deepspeech_training/VERSION                                              
/DeepSpeech/training/deepspeech_training/__init__.py                                       
/DeepSpeech/training/deepspeech_training/evaluate.py                  
/DeepSpeech/training/deepspeech_training/train.py                  
/DeepSpeech/training/deepspeech_training/util/__init__.py                  
/DeepSpeech/training/deepspeech_training/util/audio.py                                             
/DeepSpeech/training/deepspeech_training/util/augmentations.py                                  
/DeepSpeech/training/deepspeech_training/util/check_characters.py                                                                                                                                       
/DeepSpeech/training/deepspeech_training/util/checkpoints.py                                                                                                                                            
/DeepSpeech/training/deepspeech_training/util/config.py                                                                                                                                                 
/DeepSpeech/training/deepspeech_training/util/downloader.py                                      
/DeepSpeech/training/deepspeech_training/util/evaluate_tools.py                                                                                                                                         
/DeepSpeech/training/deepspeech_training/util/feeding.py                                                                                                                                                
/DeepSpeech/training/deepspeech_training/util/flags.py                                            
/DeepSpeech/training/deepspeech_training/util/gpu.py                                            
/DeepSpeech/training/deepspeech_training/util/helpers.py                                      
/DeepSpeech/training/deepspeech_training/util/importers.py                                        
/DeepSpeech/training/deepspeech_training/util/logging.py                                                                                                                                                
/DeepSpeech/training/deepspeech_training/util/sample_collections.py                               
/DeepSpeech/training/deepspeech_training/util/stm.py                                                                                                                                                    
/DeepSpeech/training/deepspeech_training/util/taskcluster.py                                  
/DeepSpeech/training/deepspeech_training/util/text.py                                                                                                                                                   
/DeepSpeech/training/deepspeech_training.egg-info/PKG-INFO                                     
/DeepSpeech/training/deepspeech_training.egg-info/SOURCES.txt           
/DeepSpeech/training/deepspeech_training.egg-info/dependency_links.txt          
/DeepSpeech/training/deepspeech_training.egg-info/requires.txt                     
/DeepSpeech/training/deepspeech_training.egg-info/top_level.txt               
Removing intermediate container 483fa03cf9ff                             
 ---> c3ba6f5990f0                                                           
Step 18/22 : RUN python3 util/taskcluster.py --source tensorflow --branch r1.15         --artifact convert_graphdef_memmapped_format  --target .
 ---> Running in 51030dd88d0d                                             
Training package is not installed. See training documentation.          
Traceback (most recent call last):                                    
  File "util/taskcluster.py", line 7, in <module>                         
    from deepspeech_training.util import taskcluster as dsu_taskcluster     
ModuleNotFoundError: No module named 'deepspeech_training'                
The command '/bin/bash -c python3 util/taskcluster.py --source tensorflow --branch r1.15         --artifact convert_graphdef_memmapped_format  --target .' returned a non-zero code: 1

lissyx · 2020-09-02T09:15:28Z

49.6.3

I can't find this one.

lissyx · 2020-09-02T09:17:25Z

I got the 0.8.2 release to build and train models with the following workaround to use 49.6.3:

It seems to work here as well.

I'd welcome a PR against r0.8 and against master that:

bumps pip==20.2.2 version
bumps setuptools==49.6.0 version
updates the documentation as well

And we could make a 0.8.3 to release the fix as well.

lissyx · 2020-09-02T09:18:35Z

@DewiBrynJones I took the liberty to assign you the bug and I'm waiting on your PR since you have nailed that down quite well :)

DewiBrynJones · 2020-09-02T09:20:33Z

I took the liberty to assign you the bug and I'm waiting on your PR since you have nailed that down quite well :)

Ha ha! me and my big mouth. i'll get on it.

lissyx · 2020-09-02T09:26:47Z

I took the liberty to assign you the bug and I'm waiting on your PR since you have nailed that down quite well :)

Ha ha! me and my big mouth. i'll get on it.

No, it's really welcome, but if you can't do it I'll take care of it, no pressure here.

Fix for setuptools._distutils issue (#3295)

fix for #3295 on r0.8 branch

lissyx · 2020-09-02T13:56:48Z

Thanks @DewiBrynJones it's now fixed on both branches

lissyx · 2020-09-02T13:58:08Z

@DewiBrynJones Do you mind also sending a PR for master and for r0.8 to update our documentation: doc/TRAINING.rst?

DewiBrynJones · 2020-09-02T14:23:34Z

Here you are:

#3306

#3307

Cheers!

update after #3295

updating docs for #3295

lissyx · 2020-09-02T14:29:44Z

Thanks!

lissyx added bug help wanted labels Aug 31, 2020

lissyx added upstream-issue This bug is actually an upstream issue and removed help wanted labels Sep 1, 2020

lissyx assigned DewiBrynJones Sep 2, 2020

DewiBrynJones mentioned this issue Sep 2, 2020

Fix for setuptools._distutils issue (#3295) #3301

Merged

lissyx added a commit that referenced this issue Sep 2, 2020

Merge pull request #3301 from techiaith/master

b6f5ddf

Fix for setuptools._distutils issue (#3295)

lissyx added a commit that referenced this issue Sep 2, 2020

Merge pull request #3302 from techiaith/r0.8

b17d43a

fix for #3295 on r0.8 branch

lissyx added the documentation label Sep 2, 2020

lissyx added a commit that referenced this issue Sep 2, 2020

Merge pull request #3306 from techiaith/r0.8

9ff6df7

update after #3295

lissyx added a commit that referenced this issue Sep 2, 2020

Merge pull request #3307 from techiaith/master

91e7060

updating docs for #3295

lissyx closed this as completed Sep 2, 2020

reuben referenced this issue in reuben/STT Nov 16, 2020

fix for #3295

8f3a01d

reuben referenced this issue in reuben/STT Nov 16, 2020

updating docs for #3295

33414af

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker train failure: ModuleNotFoundError: No module named 'setuptools._distutils' #3295

Docker train failure: ModuleNotFoundError: No module named 'setuptools._distutils' #3295

lissyx commented Aug 31, 2020

DewiBrynJones commented Aug 31, 2020

DewiBrynJones commented Aug 31, 2020

lissyx commented Aug 31, 2020

DewiBrynJones commented Aug 31, 2020

lissyx commented Aug 31, 2020

lissyx commented Sep 1, 2020

DewiBrynJones commented Sep 1, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

DewiBrynJones commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

DewiBrynJones commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

DewiBrynJones commented Sep 2, 2020

lissyx commented Sep 2, 2020

Docker train failure: ModuleNotFoundError: No module named 'setuptools._distutils' #3295

Docker train failure: ModuleNotFoundError: No module named 'setuptools._distutils' #3295

Comments

lissyx commented Aug 31, 2020

DewiBrynJones commented Aug 31, 2020

DewiBrynJones commented Aug 31, 2020

lissyx commented Aug 31, 2020

DewiBrynJones commented Aug 31, 2020

lissyx commented Aug 31, 2020

lissyx commented Sep 1, 2020

DewiBrynJones commented Sep 1, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

DewiBrynJones commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

DewiBrynJones commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

lissyx commented Sep 2, 2020

DewiBrynJones commented Sep 2, 2020

lissyx commented Sep 2, 2020