AttributeError: Can't pickle local object 'http_server.<locals>.serve'

[jrave]

Gave this a spin on Windows with Python 3.5.2 and am getting the following errors:

C:\Users\Johannes\Documents\projects\trytls>trytls https python stubs\python3-urllib\run.py
platform: Windows
runner: trytls 0.2.1 (CPython 3.5.2, OpenSSL 1.0.2h)
stub: python 'stubs\python3-urllib\run.py'
 PASS support for TLS server name indication (SNI) [accept badssl.com:443]
 PASS expired certificate [reject expired.badssl.com:443]
 PASS wrong hostname in certificate [reject wrong.host.badssl.com:443]
 PASS self-signed certificate [reject self-signed.badssl.com:443]
 PASS SHA-256 signature [accept sha256.badssl.com:443]
 PASS 1000 subjectAltNames [accept 1000-sans.badssl.com:443]
 PASS incomplete chain of trust [reject incomplete-chain.badssl.com:443]
 PASS Superfish CA [reject superfish.badssl.com:443]
 PASS eDellRoot CA [reject edellroot.badssl.com:443]
 PASS DSDTestProvider CA [reject dsdtestprovider.badssl.com:443]
 PASS protect against Apple's TLS vulnerability CVE-2014-1266 [reject www.ssllabs.com:10443]
 PASS protect against the FREAK attack [reject www.ssllabs.com:10444]
 PASS protect against the Logjam attack [reject www.ssllabs.com:10445]
 PASS protect against FREAK attack (test server 1) [reject cve.freakattack.com:443]
 PASS protect against FREAK attack (test server 2) [reject cve2.freakattack.com:443]
Traceback (most recent call last):
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Johannes\AppData\Local\Programs\Python\Python35-32\Scripts\trytls.exe\__main__.py", line 9, in <module>
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\site-packages\trytls\runner.py", line 235, in main
    if not run(command, bundle):
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\site-packages\trytls\runner.py", line 186, in run
    for test, res in collect(args, tests):
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\site-packages\trytls\runner.py", line 101, in collect
    with env() as test:
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\site-packages\trytls\bundles\https.py", line 123, in local
    with http_server(certdata, keydata) as (host, port):
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\site-packages\trytls\bundles\https.py", line 110, in http_server
    process.start()
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'http_server.<locals>.serve'

C:\Users\Johannes\Documents\projects\trytls>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\multiprocessing\spawn.py", line 100, in spawn_main
    new_handle = steal_handle(parent_pid, pipe_handle)
  File "c:\users\johannes\appdata\local\programs\python\python35-32\lib\multiprocessing\reduction.py", line 81, in steal_handle
    _winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] The parameter is incorrect

Additional note:
Would be great to add to the documentation that you need openssl on the path.

[jviide]

Thanks for the report! I created pull requet #197 that potentially fixes this. I say "potentially" because I don't have a Windows setup handy to confirm the fix, but I managed to reproduce the issue and confirm the fix in a non-Windows environment by forcing the multiprocessing start method to spawn.

The root cause was the fact that trytls.bundles.https.http_server uses the multiprocessing module to run code in a separate process. This was fine in Unix, where multiprocessing creates the subprocesses by forking. On Windows the subprocesses are spawned, and the worker function/callable needs to be picklable, which it wasn't. Heres an simplified example demonstrating this on Python 3.5:

>>> import pickle
>>>
>>> def outer():
...     def inner():
...         pass
...     pickle.dumps(inner)
... 
>>> outer()
Traceback (most recent call last):
  ...
AttributeError: Can't pickle local object 'outer.<locals>.inner'

#197 hoists the function to the module top level, which should take care of the problem.

I also created ticket #198 about needing openssl on the path.

[Mamietti]

Closed by #197.