Getting an infinite loop while using pyfakefs together with tqdm on MacOS

[introkun]

Describe the bug
Getting an infinite loop while using pyfakefs together with tqdm on MacOS.
I can see in the stdout:

Traceback (most recent call last):
  File "/Users/sergey/miniconda/envs/py385/lib/python3.8/multiprocessing/resource_tracker.py", line 191, in main
    cmd, name, rtype = line.strip().decode('ascii').split(':')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 0: ordinal not in range(128)

Switching to unittest module it solves the issue.

How To Reproduce
MacOS: Big Sur, 11.0.1 (20B29)
Python (in miniconda): 3.8.5

import tqdm
import time
import unittest
import pyfakefs
from pyfakefs.fake_filesystem_unittest import TestCase

class TestClass(TestCase):
	def setUp(self):
		self.setUpPyfakefs()
		pass

	def test_tqdm(self):
		with tqdm.tqdm(total=100, ascii=False) as pbar:
			for i in range(10):
				time.sleep(0.1)
				pbar.update(10)

if __name__ == "__main__":
	unittest.main()

Your enviroment

Python 3.8.5 (default, Sep  4 2020, 02:22:02)
[Clang 10.0.0 ]
pyfakefs 4.3.1

[mrbean-bremen]

Hm, I cannot reproduce this. I had your example program run in the different CI environment, and as you can see, there seems to be no problem. Maybe I misunderstood what you are doing?
Or there is another package installed that somehow causes the problem - can you show the output of pip list?

[introkun]

pip list

Package                  Version
------------------------ -------------------
asn1                     2.4.1
astroid                  2.4.2
attrs                    20.3.0
beautifulsoup4           4.9.3
cachetools               4.1.1
certifi                  2020.11.8
cffi                     1.14.4
chardet                  3.0.4
click                    7.1.2
colorama                 0.4.4
coverage                 5.3
coveralls                2.2.0
cryptography             3.2.1
docopt                   0.6.2
enum-compat              0.0.3
filelock                 3.0.12
Flask                    1.1.2
future                   0.18.2
google-api-core          1.23.0
google-api-python-client 1.12.8
google-auth              1.23.0
google-auth-httplib2     0.0.4
google-auth-oauthlib     0.4.2
googleapis-common-protos 1.52.0
httplib2                 0.18.1
idna                     2.10
iniconfig                1.1.1
invoke                   1.4.1
isort                    5.6.4
itsdangerous             1.1.0
Jinja2                   2.11.2
lazy-object-proxy        1.4.3
MarkupSafe               1.1.1
mccabe                   0.6.1
oauthlib                 3.1.0
packaging                20.4
Pillow                   8.0.1
pip                      20.2.4
pluggy                   0.13.1
protobuf                 3.14.0
py                       1.9.0
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pycparser                2.20
pycryptodome             3.9.9
pycryptoplus             0.0.1
pycurl                   7.43.0.6
pyfakefs                 4.3.1
pylint                   2.6.0
pyOpenSSL                19.1.0
pyparsing                2.4.7
PyQt5                    5.15.2
PyQt5-sip                12.8.1
pytest                   6.1.2
pytz                     2020.4
pyusb                    1.1.0
requests                 2.25.0
requests-oauthlib        1.3.0
rsa                      4.6
setuptools               50.3.1.post20201107
six                      1.15.0
soupsieve                2.0.1
toml                     0.10.2
tqdm                     4.53.0
Unidecode                1.1.1
uritemplate              3.0.1
urllib3                  1.26.2
Werkzeug                 1.0.1
wheel                    0.35.1
wrapt                    1.12.1
zstandard                0.14.0

[introkun]

Hm, I cannot reproduce this. I had your example program run in the different CI environment, and as you can see, there seems to be no problem. Maybe I misunderstood what you are doing?
Or there is another package installed that somehow causes the problem - can you show the output of pip list?

Yes it also works in other CI env (e.g. https://repl.it/join/xrhsucag-introkun).
But it didn't work on my laptop...

[mrbean-bremen]

Thanks - that will be difficult to reproduce. I don't have a Mac, so I can't reproduce this locally for this specific enviroment... Can you try the same test in a minimal virtualenv, that has only pyfakefs and tqdm? If that also fails, it would rule out that other packages are responsible.

[introkun]

https://monosnap.com/file/FIz6wR2EK8vUWHPdUwmVUN2P1HBoim

[introkun]

Thanks - that will be difficult to reproduce. I don't have a Mac, so I can't reproduce this locally for this specific enviroment... Can you try the same test in a minimal virtualenv, that has only pyfakefs and tqdm? If that also fails, it would rule out that other packages are responsible.

It's same on clean env:

Package    Version
---------- -------------------
certifi    2020.11.8
pip        20.2.4
pyfakefs   4.3.1
setuptools 50.3.1.post20201107
tqdm       4.53.0
wheel      0.35.1

[mrbean-bremen]

Ok, thanks - so I guess it has somehow to do with tqdm that uses multiprocessing. Do you use the same OS and Python version in your working CI environment?
I remember that there have been problems with multiprocessing before, but that had to do with some file system functions that could not be patched because system-specific functionality is used there (pyfakefs cannot patch these).

[introkun]

Adding 'encoding='utf-8'' to the python's resource_tracker.py solves the issue for me.

[mrbean-bremen]

The CI tests use MacOS 10.15.7 and Python 3.8.6 - maybe the code has changed from 3.8.5 to 3.8.6? I will check.

[mrbean-bremen]

Nope, nothing substantially has changed...
Interesting about the encoding - can you actually check what kind of resource causes the problem? Maybe something with your local installation is different?

[introkun]

can you actually check what kind of resource causes the problem

No idea how to do that. 'print(line)' doesn't work with error 'ValueError: I/O operation on closed file.'

[mrbean-bremen]

I see - locally I just don't get to this line, so I can't test it here. I currently don't have an idea how to proceed without being able to reproduce the problem, and I don't have more time today... I'll see if I can think of something - sorry I couldn't help.

[mrbean-bremen]

I tried both under Linux and Windows (as I wrote - don't have a Mac), with tqdm under Python 3.8, and the resource tracker code didn't get called in both cases. As I have no idea why and how it is called in your case, I'm giving up for the moment. Let me know if you have some new information.

[mrbean-bremen]

@introkun - did you get anywhere with this problem? I'm inclined to close the issue if we have no way to reproduce it.