Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

problems running tests on master #1528

Closed
njsmith opened this issue Feb 18, 2019 · 17 comments
Closed

problems running tests on master #1528

njsmith opened this issue Feb 18, 2019 · 17 comments
Assignees
@robnagler

Comments

@njsmith
Copy link
Contributor

njsmith commented Feb 18, 2019

Using the vagrant image, running python setup.py tests in the sirepo source dir, on latest master (3d1d462), I get:

[py2;@v sirepo]$ python setup.py test
running test
running egg_info
writing requirements to sirepo.egg-info/requires.txt
writing sirepo.egg-info/PKG-INFO
writing top-level names to sirepo.egg-info/top_level.txt
writing dependency_links to sirepo.egg-info/dependency_links.txt
writing entry points to sirepo.egg-info/entry_points.txt
reading manifest file 'sirepo.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sirepo.egg-info/SOURCES.txt'
running build_ext
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.7.0, pluggy-0.4.0
rootdir: /home/vagrant/src/radiasoft/sirepo, inifile:
plugins: forked-0.2
collected 36 items                                                              

tests/application_data_test.py .

And then it seems to hang forever.

If I do pytest tests -v -s --fulltrace, and hit control-C after it's stuck, I get a ton of output, that suggests it's stuck in waitpid?

============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.7.0, pluggy-0.4.0 -- /home/vagrant/.pyenv/versions/2.7.14/envs/py2/bin/python2.7
cachedir: .cache
rootdir: /home/vagrant/src/radiasoft/sirepo, inifile:
plugins: forked-0.2
collected 36 items                                                              

tests/application_data_test.py::test_processed_image PASSEDsirepo/oauth.py:216:_init_tables creating user oauth database
^C

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

config = <_pytest.config.Config object at 0x7f918f319150>
doit = <function _main at 0x7f918f2e5f50>

    def wrap_session(config, doit):
        """Skeleton command line program"""
        session = Session(config)
        session.exitstatus = EXIT_OK
        initstate = 0
        try:
            try:
                config._do_configure()
                initstate = 1
                config.hook.pytest_sessionstart(session=session)
                initstate = 2
>               session.exitstatus = doit(config, session) or 0

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py:110: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config = <_pytest.config.Config object at 0x7f918f319150>
session = <Session 'sirepo'>

    def _main(config, session):
        """ default command line protocol for initialization, session,
        running tests and reporting. """
        config.hook.pytest_collection(session=session)
>       config.hook.pytest_runtestloop(session=session)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py:146: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_HookCaller 'pytest_runtestloop'>
kwargs = {'__multicall__': <_MultiCall 0 results, 0 meths, kwargs={'__multicall__': <_MultiCall 0 results, 0 meths, kwargs={...}>, 'session': <Session 'sirepo'>}>, 'session': <Session 'sirepo'>}

    def __call__(self, **kwargs):
        assert not self.is_historic()
>       return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:745: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.config.PytestPluginManager object at 0x7f918f2faa10>
hook = <_HookCaller 'pytest_runtestloop'>, methods = []
kwargs = {'__multicall__': <_MultiCall 0 results, 0 meths, kwargs={'__multicall__': <_MultiCall 0 results, 0 meths, kwargs={...}>, 'session': <Session 'sirepo'>}>, 'session': <Session 'sirepo'>}

    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

hook = <_HookCaller 'pytest_runtestloop'>, methods = []
kwargs = {'__multicall__': <_MultiCall 0 results, 0 meths, kwargs={'__multicall__': <_MultiCall 0 results, 0 meths, kwargs={...}>, 'session': <Session 'sirepo'>}>, 'session': <Session 'sirepo'>}

    self._inner_hookexec = lambda hook, methods, kwargs: \
>       _MultiCall(methods, kwargs, hook.spec_opts).execute()

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:334: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_MultiCall 0 results, 0 meths, kwargs={'__multicall__': <_MultiCall 0 results, 0 meths, kwargs={...}>, 'session': <Session 'sirepo'>}>

    def execute(self):
        all_kwargs = self.kwargs
        self.results = results = []
        firstresult = self.specopts.get("firstresult")
    
        while self.hook_impls:
            hook_impl = self.hook_impls.pop()
            try:
                args = [all_kwargs[argname] for argname in hook_impl.argnames]
            except KeyError:
                for argname in hook_impl.argnames:
                    if argname not in all_kwargs:
                        raise HookCallError(
                            "hook call must provide argument %r" % (argname,))
            if hook_impl.hookwrapper:
                return _wrapped_call(hook_impl.function(*args), self.execute)
>           res = hook_impl.function(*args)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:614: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

session = <Session 'sirepo'>

    def pytest_runtestloop(session):
        if (session.testsfailed and
                not session.config.option.continue_on_collection_errors):
            raise session.Interrupted(
                "%d errors during collection" % session.testsfailed)
    
        if session.config.option.collectonly:
            return True
    
        for i, item in enumerate(session.items):
            nextitem = session.items[i + 1] if i + 1 < len(session.items) else None
>           item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py:169: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_HookCaller 'pytest_runtest_protocol'>
kwargs = {'__multicall__': <_MultiCall 0 results, 1 meths, kwargs={'item': <Function 'test_anonymous_user'>, '__multicall__': <...extitem': <Function 'test_no_user'>}>, 'item': <Function 'test_anonymous_user'>, 'nextitem': <Function 'test_no_user'>}

    def __call__(self, **kwargs):
        assert not self.is_historic()
>       return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:745: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.config.PytestPluginManager object at 0x7f918f2faa10>
hook = <_HookCaller 'pytest_runtest_protocol'>
methods = [<_pytest.vendored_packages.pluggy.HookImpl instance at 0x7f918f2ca098>]
kwargs = {'__multicall__': <_MultiCall 0 results, 1 meths, kwargs={'item': <Function 'test_anonymous_user'>, '__multicall__': <...extitem': <Function 'test_no_user'>}>, 'item': <Function 'test_anonymous_user'>, 'nextitem': <Function 'test_no_user'>}

    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

hook = <_HookCaller 'pytest_runtest_protocol'>
methods = [<_pytest.vendored_packages.pluggy.HookImpl instance at 0x7f918f2ca098>]
kwargs = {'__multicall__': <_MultiCall 0 results, 1 meths, kwargs={'item': <Function 'test_anonymous_user'>, '__multicall__': <...extitem': <Function 'test_no_user'>}>, 'item': <Function 'test_anonymous_user'>, 'nextitem': <Function 'test_no_user'>}

    self._inner_hookexec = lambda hook, methods, kwargs: \
>       _MultiCall(methods, kwargs, hook.spec_opts).execute()

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:334: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_MultiCall 0 results, 1 meths, kwargs={'item': <Function 'test_anonymous_user'>, '__multicall__': <_MultiCall 0 results, 1 meths, kwargs={...}>, 'nextitem': <Function 'test_no_user'>}>

    def execute(self):
        all_kwargs = self.kwargs
        self.results = results = []
        firstresult = self.specopts.get("firstresult")
    
        while self.hook_impls:
            hook_impl = self.hook_impls.pop()
            try:
                args = [all_kwargs[argname] for argname in hook_impl.argnames]
            except KeyError:
                for argname in hook_impl.argnames:
                    if argname not in all_kwargs:
                        raise HookCallError(
                            "hook call must provide argument %r" % (argname,))
            if hook_impl.hookwrapper:
>               return _wrapped_call(hook_impl.function(*args), self.execute)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:613: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

wrap_controller = <generator object pytest_runtest_protocol at 0x7f918bda2870>
func = <bound method _MultiCall.execute of <_MultiCall 0 results, 1 meths, kwargs={'i...all 0 results, 1 meths, kwargs={...}>, 'nextitem': <Function 'test_no_user'>}>>

    def _wrapped_call(wrap_controller, func):
        """ Wrap calling to a function with a generator which needs to yield
        exactly once.  The yield point will trigger calling the wrapped function
        and return its _CallOutcome to the yield point.  The generator then needs
        to finish (raise StopIteration) in order for the wrapped call to complete.
        """
        try:
            next(wrap_controller)   # first yield
        except StopIteration:
            _raise_wrapfail(wrap_controller, "did not yield")
        call_outcome = _CallOutcome(func)
        try:
            wrap_controller.send(call_outcome)
            _raise_wrapfail(wrap_controller, "has second yield")
        except StopIteration:
            pass
>       return call_outcome.get_result()

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:254: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.vendored_packages.pluggy._CallOutcome instance at 0x7f9189a34368>

    def get_result(self):
        if self.excinfo is None:
            return self.result
        else:
            ex = self.excinfo
            if _py3:
                raise ex[1].with_traceback(ex[2])
>           _reraise(*ex)  # noqa

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:280: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.vendored_packages.pluggy._CallOutcome instance at 0x7f9189a34368>
func = <bound method _MultiCall.execute of <_MultiCall 0 results, 1 meths, kwargs={'i...all 0 results, 1 meths, kwargs={...}>, 'nextitem': <Function 'test_no_user'>}>>

    def __init__(self, func):
        try:
>           self.result = func()

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:265: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_MultiCall 0 results, 1 meths, kwargs={'item': <Function 'test_anonymous_user'>, '__multicall__': <_MultiCall 0 results, 1 meths, kwargs={...}>, 'nextitem': <Function 'test_no_user'>}>

    def execute(self):
        all_kwargs = self.kwargs
        self.results = results = []
        firstresult = self.specopts.get("firstresult")
    
        while self.hook_impls:
            hook_impl = self.hook_impls.pop()
            try:
                args = [all_kwargs[argname] for argname in hook_impl.argnames]
            except KeyError:
                for argname in hook_impl.argnames:
                    if argname not in all_kwargs:
                        raise HookCallError(
                            "hook call must provide argument %r" % (argname,))
            if hook_impl.hookwrapper:
>               return _wrapped_call(hook_impl.function(*args), self.execute)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:613: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

wrap_controller = <generator object pytest_runtest_protocol at 0x7f9189a0e910>
func = <bound method _MultiCall.execute of <_MultiCall 0 results, 1 meths, kwargs={'i...all 0 results, 1 meths, kwargs={...}>, 'nextitem': <Function 'test_no_user'>}>>

    def _wrapped_call(wrap_controller, func):
        """ Wrap calling to a function with a generator which needs to yield
        exactly once.  The yield point will trigger calling the wrapped function
        and return its _CallOutcome to the yield point.  The generator then needs
        to finish (raise StopIteration) in order for the wrapped call to complete.
        """
        try:
            next(wrap_controller)   # first yield
        except StopIteration:
            _raise_wrapfail(wrap_controller, "did not yield")
        call_outcome = _CallOutcome(func)
        try:
            wrap_controller.send(call_outcome)
            _raise_wrapfail(wrap_controller, "has second yield")
        except StopIteration:
            pass
>       return call_outcome.get_result()

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:254: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.vendored_packages.pluggy._CallOutcome instance at 0x7f9189a343f8>

    def get_result(self):
        if self.excinfo is None:
            return self.result
        else:
            ex = self.excinfo
            if _py3:
                raise ex[1].with_traceback(ex[2])
>           _reraise(*ex)  # noqa

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:280: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.vendored_packages.pluggy._CallOutcome instance at 0x7f9189a343f8>
func = <bound method _MultiCall.execute of <_MultiCall 0 results, 1 meths, kwargs={'i...all 0 results, 1 meths, kwargs={...}>, 'nextitem': <Function 'test_no_user'>}>>

    def __init__(self, func):
        try:
>           self.result = func()

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:265: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_MultiCall 0 results, 1 meths, kwargs={'item': <Function 'test_anonymous_user'>, '__multicall__': <_MultiCall 0 results, 1 meths, kwargs={...}>, 'nextitem': <Function 'test_no_user'>}>

    def execute(self):
        all_kwargs = self.kwargs
        self.results = results = []
        firstresult = self.specopts.get("firstresult")
    
        while self.hook_impls:
            hook_impl = self.hook_impls.pop()
            try:
                args = [all_kwargs[argname] for argname in hook_impl.argnames]
            except KeyError:
                for argname in hook_impl.argnames:
                    if argname not in all_kwargs:
                        raise HookCallError(
                            "hook call must provide argument %r" % (argname,))
            if hook_impl.hookwrapper:
                return _wrapped_call(hook_impl.function(*args), self.execute)
>           res = hook_impl.function(*args)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:614: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <Function 'test_anonymous_user'>

    @pytest.mark.tryfirst
    def pytest_runtest_protocol(item):
        if item.config.getvalue("forked"):
>           reports = forked_run_report(item)

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <Function 'test_anonymous_user'>

    def forked_run_report(item):
        # for now, we run setup/teardown in the subprocess
        # XXX optionally allow sharing of setup/teardown
        from _pytest.runner import runtestprotocol
        EXITSTATUS_TESTEXIT = 4
        import marshal
    
        def runforked():
            try:
                reports = runtestprotocol(item, log=False)
            except KeyboardInterrupt:
                py.std.os._exit(EXITSTATUS_TESTEXIT)
            return marshal.dumps([serialize_report(x) for x in reports])
    
        ff = py.process.ForkedFunc(runforked)
>       result = ff.waitfinish()

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <py._process.forkedfunc.ForkedFunc instance at 0x7f9189a340e0>
waiter = <built-in function waitpid>

    def waitfinish(self, waiter=os.waitpid):
>       pid, systemstatus = waiter(self.pid, 0)
E       KeyboardInterrupt

../../../.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/py/_process/forkedfunc.py:82: KeyboardInterrupt
========================== 1 passed in 32.96 seconds ===========================

I also tried running just the server_tests.py file, and it didn't freeze, but test_get_data_file failed. I guess this may be a second, unrelated issue:

[py2;@v sirepo]$ pytest tests/server_test.py -v
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.7.0, pluggy-0.4.0 -- /home/vagrant/.pyenv/versions/2.7.14/envs/py2/bin/python2.7
cachedir: .cache
rootdir: /home/vagrant/src/radiasoft/sirepo, inifile:
plugins: forked-0.2
collected 3 items                                                               

tests/server_test.py::test_basic PASSEDsirepo/runner/background.py:122:_start KSUlLQLW-0tOOOj5P-bunchReport1: started: pid=7766 cmd=['sirepo', 'elegant', 'run', '/home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/KSUlLQLW/elegant/0tOOOj5P/bunchReport']
sirepo/runner/background.py:50:_kill KSUlLQLW-0tOOOj5P-bunchReport1: kill 15 pid=7766
sirepo/runner/background.py:95:_sigchld_handler KSUlLQLW-0tOOOj5P-bunchReport1: waitpid pid=7766 status=15
sirepo/server.py:776:_simulation_error error in read_result: /home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/KSUlLQLW/elegant/0tOOOj5P/bunchReport: error: ERROR: Terminated unexpectedly

tests/server_test.py::test_get_data_file FAILEDsirepo/util.py:45:_raise srw: simulation not found by name: UndulatorRadiation
sirepo/uri_router.py:176:_dispatch find-by-name/srw/default/UndulatorRadiation: error: NotFound: 404 Not Found: The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.
  File "/home/vagrant/.pyenv/versions/py2/bin/pytest", line 11, in <module>
    sys.exit(main())
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/config.py", line 58, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 139, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 110, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 146, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 169, in pytest_runtestloop
    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py", line 35, in pytest_runtest_protocol
    reports = forked_run_report(item)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py", line 55, in forked_run_report
    ff = py.process.ForkedFunc(runforked)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/py/_process/forkedfunc.py", line 50, in __init__
    self._child(nice_level, child_on_start, child_on_exit)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/py/_process/forkedfunc.py", line 65, in _child
    retval = self.fun(*self.args, **self.kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py", line 50, in runforked
    reports = runtestprotocol(item, log=False)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 82, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 162, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 182, in call_runtest_hook
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 196, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 182, in <lambda>
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 112, in pytest_runtest_call
    item.runtest()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/python.py", line 1169, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/python.py", line 143, in pytest_pyfunc_call
    testfunction(**testargs)
  File "/home/vagrant/src/radiasoft/sirepo/tests/server_test.py", line 119, in test_srw
    pkeq(fc.get('/find-by-name/srw/default/UndulatorRadiation').status_code, 404)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 830, in get
    return self.open(*args, **kw)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/testing.py", line 200, in open
    follow_redirects=follow_redirects
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 803, in open
    response = self.run_wsgi_app(environ, buffered=buffered)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 716, in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 923, in run_wsgi_app
    app_rv = app(environ, start_response)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 172, in _dispatch
    return _dispatch_call(route.func, kwargs)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 182, in _dispatch_call
    resp = flask.make_response(func(**kwargs))
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/server.py", line 258, in api_findByName
    util.raise_not_found('{}: simulation not found by name: {}', simulation_type, simulation_name)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/util.py", line 25, in raise_not_found
    _raise('NotFound', *args, **kwargs)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/util.py", line 46, in _raise
    raise getattr(werkzeug.exceptions, exc)()

tests/server_test.py::test_srw PASSED

=================================== FAILURES ===================================
______________________________ test_get_data_file ______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/server_test.py", line 62, in test_get_data_file
    run.nextRequest
  File "/home/vagrant/src/radiasoft/pykern/pykern/pkcollections.py", line 56, in __getattr__
    return self.__getattribute__(name)
AttributeError: 'Dict' object has no attribute 'nextRequest'
===================== 1 failed, 2 passed in 13.36 seconds ======================

CC @robnagler
@njsmith njsmith mentioned this issue Feb 18, 2019
Merged
@robnagler robnagler self-assigned this Feb 18, 2019
@robnagler robnagler added the 1 label Feb 18, 2019
@robnagler
Copy link
Member

I can't reproduce in master or njs-sow-2. All tests run except runner_test.py (see comment).

Perhaps we can debug together. My schedule is open.

@njsmith
Copy link
Contributor Author

njsmith commented Feb 19, 2019

Unfortunately I was AFK during the day – going to take a look now, but I guess you're probably AFK now :-). If I don't get it tonight then maybe we can compare notes tomorrow...

@robnagler
Copy link
Member

Just leaving the kbd behind. I have a couple of meetings tomorrow, but ping me and we'll see. I didn't review the code, just tested, and tried to reproduce the issue.

@njsmith
Copy link
Contributor Author

njsmith commented Feb 19, 2019

OK, I built a new totally-vanilla vagrant image from scratch (just ran vagrant up, then installed redhat-dev and sirepo-dev, then ran pytest tests). And then I forgot about it for a while, which I guess was crucial :-). I get 3 failures:

[py2;@v2 sirepo]$ pytest tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.7.0, pluggy-0.4.0
rootdir: /home/vagrant/src/radiasoft/sirepo, inifile:
plugins: forked-1.0.1
collected 36 items                                                              

tests/application_data_test.py .
tests/beaker_compat_test.py F..
tests/blank_test.py .
tests/bluesky_test.py ...
tests/cookie_test.py .
tests/crystal_test.py ...
tests/exporter_test.py .
tests/importer_test.py ..
tests/oauth_test.py F
tests/server1_test.py ..
tests/server_test.py .F.
tests/srw_test.py ..
tests/template_common_test.py .
tests/uri_router_test.py ..
tests/pkcli/admin_test.py .
tests/template/elegant_import_test.py .
tests/template/elegant_rpn_test.py .
tests/template/elegant_test.py .
tests/template/shadow_generate_test.py .
tests/template/srw_generate_test.py ..
tests/template/srw_import_test.py .
tests/template/srw_magmeaszip_test.py ..

=================================== FAILURES ===================================
_____________________________ test_anonymous_user ______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/beaker_compat_test.py", line 54, in test_anonymous_user
    ('a', 'sros'),
  File "/home/vagrant/src/radiasoft/sirepo/tests/beaker_compat_test.py", line 44, in _test_cookie
    want_cookie=False,
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 81, in test_in_request
    fc = flask_client(cfg)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 52, in flask_client
    server.init()
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/server.py", line 657, in init
    uri_router.init(app, uwsgi)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 76, in init
    importlib.import_module('sirepo.' + m).init_apis(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 138, in init_apis
    _init_tables(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 217, in _init_tables
    _db.create_all()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 963, in create_all
    self._execute_for_all_tables(app, bind, 'create_all')
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 955, in _execute_for_all_tables
    op(bind=self.get_engine(app, bind), **extra)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 4200, in create_all
    ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2011, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1599, in _run_visitor
    visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 130, in traverse_single
    return meth(obj, **kw)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 752, in visit_metadata
    [t for t in tables if self._can_create_table(t)]
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 729, in _can_create_table
    self.connection, table.name, schema=effective_schema
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1357, in has_table
    connection, "table_info", table_name, schema=schema
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1799, in _get_table_pragma
    cursor = connection.execute(statement)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 974, in execute
    return self._execute_text(object_, multiparams, params)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1147, in _execute_text
    parameters,
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA table_info("user_t")'] (Background on this error at: http://sqlalche.me/e/e3q8)
______________________________ test_login_logout _______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/oauth_test.py", line 22, in test_login_logout
    'SIREPO_OAUTH_GITHUB_CALLBACK_URI': 'n/a',
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 52, in flask_client
    server.init()
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/server.py", line 657, in init
    uri_router.init(app, uwsgi)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 76, in init
    importlib.import_module('sirepo.' + m).init_apis(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 138, in init_apis
    _init_tables(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 217, in _init_tables
    _db.create_all()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 963, in create_all
    self._execute_for_all_tables(app, bind, 'create_all')
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 955, in _execute_for_all_tables
    op(bind=self.get_engine(app, bind), **extra)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 4200, in create_all
    ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2011, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1599, in _run_visitor
    visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 130, in traverse_single
    return meth(obj, **kw)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 752, in visit_metadata
    [t for t in tables if self._can_create_table(t)]
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 729, in _can_create_table
    self.connection, table.name, schema=effective_schema
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1357, in has_table
    connection, "table_info", table_name, schema=schema
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1799, in _get_table_pragma
    cursor = connection.execute(statement)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 974, in execute
    return self._execute_text(object_, multiparams, params)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1147, in _execute_text
    parameters,
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA table_info("user_t")'] (Background on this error at: http://sqlalche.me/e/e3q8)
______________________________ test_get_data_file ______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/server_test.py", line 78, in test_get_data_file
    raw_response=True,
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 142, in sr_get
    return _req(route_name, params, self.get, raw_response=raw_response)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 187, in _req
    resp = op(uri)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 830, in get
    return self.open(*args, **kw)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/testing.py", line 200, in open
    follow_redirects=follow_redirects
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 803, in open
    response = self.run_wsgi_app(environ, buffered=buffered)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 716, in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 923, in run_wsgi_app
    app_rv = app(environ, start_response)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 172, in _dispatch
    return _dispatch_call(route.func, kwargs)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 182, in _dispatch_call
    resp = flask.make_response(func(**kwargs))
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/server.py", line 152, in api_downloadDataFile
    filename, content, content_type = template.get_data_file(run_dir, model, frame, options=options)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/template/elegant.py", line 368, in get_data_file
    return _sdds(_report_output_filename('bunchReport'))
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/template/elegant.py", line 346, in _sdds
    '{}: invalid or empty output from sddsprintout'.format(path)
AssertionError: /home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/WuZ7gm5M/elegant/VEcL0HN7/bunchReport/elegant.bun: invalid or empty output from sddsprintout
----------------------------- Captured stderr call -----------------------------
sirepo/runner/background.py:122:_start WuZ7gm5M-VEcL0HN7-bunchReport1: started: pid=26097 cmd=['sirepo', 'elegant', 'run', '/home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/WuZ7gm5M/elegant/VEcL0HN7/bunchReport']
sirepo/runner/background.py:95:_sigchld_handler WuZ7gm5M-VEcL0HN7-bunchReport1: waitpid pid=26097 status=0
sirepo/template/elegant_common.py:61:subprocess_output ['sddsprintout', '-columns', '-spreadsheet=csv', '/home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/WuZ7gm5M/elegant/VEcL0HN7/bunchReport/elegant.bun']: exit=1 err=Error for sddsprintout:
Unable to read layout--no header lines found (SDDS_ReadLayout)
sirepo/uri_router.py:176:_dispatch download-data-file/elegant/VEcL0HN7/bunchReport1/-1/csv: error: AssertionError: /home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/WuZ7gm5M/elegant/VEcL0HN7/bunchReport/elegant.bun: invalid or empty output from sddsprintout
  File "/home/vagrant/.pyenv/versions/py2/bin/pytest", line 10, in <module>
    sys.exit(main())
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/config.py", line 58, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 139, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 110, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 146, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/main.py", line 169, in pytest_runtestloop
    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py", line 35, in pytest_runtest_protocol
    reports = forked_run_report(item)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py", line 55, in forked_run_report
    ff = py.process.ForkedFunc(runforked)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/py/_process/forkedfunc.py", line 50, in __init__
    self._child(nice_level, child_on_start, child_on_exit)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/py/_process/forkedfunc.py", line 65, in _child
    retval = self.fun(*self.args, **self.kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/pytest_forked/__init__.py", line 50, in runforked
    reports = runtestprotocol(item, log=False)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 82, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 162, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 182, in call_runtest_hook
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 196, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 182, in <lambda>
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/runner.py", line 112, in pytest_runtest_call
    item.runtest()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/python.py", line 1169, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
    self.result = func()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/_pytest/python.py", line 143, in pytest_pyfunc_call
    testfunction(**testargs)
  File "/home/vagrant/src/radiasoft/sirepo/tests/server_test.py", line 78, in test_get_data_file
    raw_response=True,
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 142, in sr_get
    return _req(route_name, params, self.get, raw_response=raw_response)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 187, in _req
    resp = op(uri)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 830, in get
    return self.open(*args, **kw)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/testing.py", line 200, in open
    follow_redirects=follow_redirects
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 803, in open
    response = self.run_wsgi_app(environ, buffered=buffered)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 716, in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/werkzeug/test.py", line 923, in run_wsgi_app
    app_rv = app(environ, start_response)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    re
==================== 3 failed, 33 passed in 948.61 seconds =====================

Those two sqlite failures seem to have like a 5+ minute timeouts on them before they fail, which I guess is why I thought it had frozen before!

Possibly the two sqlite locking-related errors are a side-effect of having my src/ dir on NFS? No idea about test_get_data_file though.

@njsmith
Copy link
Contributor Author

njsmith commented Feb 19, 2019

And for reference, here's the results on my original VM with the njs-sow-2 branch:

[py2;@v sirepo]$ pytest tests -v
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.7.0, pluggy-0.4.0 -- /home/vagrant/.pyenv/versions/2.7.14/envs/py2/bin/python2.7
cachedir: .cache
rootdir: /home/vagrant/src/radiasoft/sirepo, inifile:
plugins: forked-0.2
collected 37 items                                                              

tests/application_data_test.py::test_processed_image PASSED
tests/beaker_compat_test.py::test_anonymous_user FAILED
tests/beaker_compat_test.py::test_no_user PASSED
tests/beaker_compat_test.py::test_oauth_user PASSED
tests/blank_test.py::test_nothing PASSED
tests/bluesky_test.py::test_auth_hash PASSED
tests/bluesky_test.py::test_auth_hash_copy PASSED
tests/bluesky_test.py::test_auth_login PASSED
tests/cookie_test.py::test_1 PASSED
tests/crystal_test.py::test_calc_bragg_angle PASSED
tests/crystal_test.py::test_get_crystal_parameters PASSED
tests/crystal_test.py::test_get_crystal_parameters_str PASSED
tests/exporter_test.py::test_create_zip PASSED
tests/importer_test.py::test_import_json PASSED
tests/importer_test.py::test_import_zip PASSED
tests/oauth_test.py::test_login_logout FAILED
tests/runner_test.py::test_runner_myapp PASSED
tests/server1_test.py::test_1_serial_stomp PASSED
tests/server1_test.py::test_missing_cookies PASSED
tests/server_test.py::test_basic PASSED
tests/server_test.py::test_get_data_file FAILED
tests/server_test.py::test_srw PASSED
tests/srw_test.py::test_model_defaults PASSED
tests/srw_test.py::test_prepare_aux_files PASSED
tests/template_common_test.py::test_validate_safe_zip PASSED
tests/uri_router_test.py::test_not_found PASSED
tests/uri_router_test.py::test_uri_for_api PASSED
tests/pkcli/admin_test.py::test_purge_users PASSED
tests/template/elegant_import_test.py::test_importer PASSED
tests/template/elegant_rpn_test.py::test_rpn PASSED
tests/template/elegant_test.py::test_get_application_data PASSED
tests/template/shadow_generate_test.py::test_generate_python PASSED
tests/template/srw_generate_test.py::test_generate_all_optics PASSED
tests/template/srw_generate_test.py::test_generate_python PASSED
tests/template/srw_import_test.py::test_importer PASSED
tests/template/srw_magmeaszip_test.py::test_magnetic_measurements_zip_file PASSED
tests/template/srw_magmeaszip_test.py::test_validate_safe_zip PASSED

=================================== FAILURES ===================================
_____________________________ test_anonymous_user ______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/beaker_compat_test.py", line 54, in test_anonymous_user
    ('a', 'sros'),
  File "/home/vagrant/src/radiasoft/sirepo/tests/beaker_compat_test.py", line 44, in _test_cookie
    want_cookie=False,
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 81, in test_in_request
    fc = flask_client(cfg)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 52, in flask_client
    server.init()
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/server.py", line 703, in init
    uri_router.init(app, uwsgi)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 76, in init
    importlib.import_module('sirepo.' + m).init_apis(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 138, in init_apis
    _init_tables(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 217, in _init_tables
    _db.create_all()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 963, in create_all
    self._execute_for_all_tables(app, bind, 'create_all')
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 955, in _execute_for_all_tables
    op(bind=self.get_engine(app, bind), **extra)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 4005, in create_all
    tables=tables)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
    **kwargs).traverse_single(element)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 736, in visit_metadata
    [t for t in tables if self._can_create_table(t)])
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 711, in _can_create_table
    table.name, schema=effective_schema)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1181, in has_table
    connection, "table_info", table_name, schema=schema)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1577, in _get_table_pragma
    cursor = connection.execute(statement)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 942, in execute
    return self._execute_text(object, multiparams, params)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1104, in _execute_text
    statement, parameters
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA table_info("user_t")'] (Background on this error at: http://sqlalche.me/e/e3q8)
______________________________ test_login_logout _______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/oauth_test.py", line 22, in test_login_logout
    'SIREPO_OAUTH_GITHUB_CALLBACK_URI': 'n/a',
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/srunit.py", line 52, in flask_client
    server.init()
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/server.py", line 703, in init
    uri_router.init(app, uwsgi)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/uri_router.py", line 76, in init
    importlib.import_module('sirepo.' + m).init_apis(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 138, in init_apis
    _init_tables(app)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/oauth.py", line 217, in _init_tables
    _db.create_all()
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 963, in create_all
    self._execute_for_all_tables(app, bind, 'create_all')
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 955, in _execute_for_all_tables
    op(bind=self.get_engine(app, bind), **extra)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 4005, in create_all
    tables=tables)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
    **kwargs).traverse_single(element)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 736, in visit_metadata
    [t for t in tables if self._can_create_table(t)])
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 711, in _can_create_table
    table.name, schema=effective_schema)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1181, in has_table
    connection, "table_info", table_name, schema=schema)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1577, in _get_table_pragma
    cursor = connection.execute(statement)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 942, in execute
    return self._execute_text(object, multiparams, params)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1104, in _execute_text
    statement, parameters
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA table_info("user_t")'] (Background on this error at: http://sqlalche.me/e/e3q8)
______________________________ test_get_data_file ______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/server_test.py", line 62, in test_get_data_file
    run.nextRequest
  File "/home/vagrant/src/radiasoft/pykern/pykern/pkcollections.py", line 56, in __getattr__
    return self.__getattribute__(name)
AttributeError: 'Dict' object has no attribute 'nextRequest'
==================== 3 failed, 34 passed in 932.69 seconds =====================

So, the same three errors, though test_get_data_file has switched back to AttributeError: 'Dict' object has no attribute 'nextRequest', instead of AssertionError: /home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/WuZ7gm5M/elegant/VEcL0HN7/bunchReport/elegant.bun: invalid or empty output from sddsprintout

I just re-ran test_get_data_file on the clean VM, to check whether there's some randomness, and indeed, this time it gave me AttributeError: 'Dict' object has no attribute 'nextRequest'.

Conclusion so far: none of the failures can be attributed to my branch or to my having somehow screwed up my main VM, using NFS for src/ may cause some problems, and there's still a mystery about test_get_data_file.

@robnagler
Copy link
Member

NFS definitely causes problems. I debugged a speed problem this week on another developer's VM with NFS-mounted sources. lstat (git status) is an order of magnitude or two slower. It's good to know that this happens on Linux hosts, too, because I thought it was a Mac issue.

It's really odd that you got database is locked, because it is a fresh db unless you happen to be setting SIREPO_SRDB_ROOT or SIREPO_SERVER_DB_DIR. I don't think the tests clear those values, which is probably a bug. I wish people would print more context in error messages. This contains no information about what is locked:

OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA table_info("user_t")'] (Background on this error at: http://sqlalche.me/e/e3q8)

For the sdds error, you can run this command manually to see what is output:

sirepo/template/elegant_common.py:61:subprocess_output ['sddsprintout', '-columns', '-spreadsheet=csv', '/home/vagrant/src/radiasoft/sirepo/tests/server_work/db/user/WuZ7gm5M/elegant/VEcL0HN7/bunchReport/elegant.bun']: exit=1 err=Error for sddsprintout:

@njsmith
Copy link
Contributor Author

njsmith commented Feb 20, 2019

It's really odd that you got database is locked, because it is a fresh db unless you happen to be setting SIREPO_SRDB_ROOT or SIREPO_SERVER_DB_DIR. I don't think the tests clear those values, which is probably a bug.

I suspect it's something like: sqlite tries to lock the file. The lock command returns an error, meaning "this NFS mount doesn't have file-locking support". sqlite misinterprets the error as meaning "this file is already locked".

I just add nfs_version: 4 to my Vagrantfile. (Vagrant's default is to use NFS v3 for some reason. v4 has locking integrated into the protocol, rather than requiring some weird sideband protocol that always breaks.) Now those two tests pass, and the whole thing finishes in ~2 minutes instead of ~15:

[py2;@v sirepo]$ pytest tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.7.0, pluggy-0.4.0
rootdir: /home/vagrant/src/radiasoft/sirepo, inifile:
plugins: forked-0.2
collected 37 items                                                              

tests/application_data_test.py .
tests/beaker_compat_test.py ...
tests/blank_test.py .
tests/bluesky_test.py ...
tests/cookie_test.py .
tests/crystal_test.py ...
tests/exporter_test.py .
tests/importer_test.py ..
tests/oauth_test.py .
tests/runner_test.py .
tests/server1_test.py ..
tests/server_test.py .F.
tests/srw_test.py ..
tests/template_common_test.py .
tests/uri_router_test.py ..
tests/pkcli/admin_test.py .
tests/template/elegant_import_test.py .
tests/template/elegant_rpn_test.py .
tests/template/elegant_test.py .
tests/template/shadow_generate_test.py .
tests/template/srw_generate_test.py ..
tests/template/srw_import_test.py .
tests/template/srw_magmeaszip_test.py ..

=================================== FAILURES ===================================
______________________________ test_get_data_file ______________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/sirepo/tests/server_test.py", line 62, in test_get_data_file
    run.nextRequest
  File "/home/vagrant/src/radiasoft/pykern/pykern/pkcollections.py", line 56, in __getattr__
    return self.__getattribute__(name)
AttributeError: 'Dict' object has no attribute 'nextRequest'
==================== 1 failed, 36 passed in 135.52 seconds =====================

@njsmith
Copy link
Contributor Author

njsmith commented Feb 20, 2019

OK, I tracked down this last failure in test_get_data_file.

Here's what's happening:

  • We start the job, using the background runner. When starting it, we write "pending" into the job's status file.
  • The background runner forks, and in the child process, it writes "running" into the job's status file
  • It does this using pkio.write_text, which updates the file in-place by truncating it, and then writing the new text.
  • In the mean time, _simulation_run_status in server.py is reading the status file to figure out what's going on. Somehow, it manages to read it during the period when it's truncated, so the status it reads out is "". (This is a race condition under any circumstances, but I guess the reason I'm hitting it so much more reliably than you is probably something to do with NFS again.)
  • In _simulation_run_status, it does is_running = rep.job_status in _RUN_STATES. Here rep.job_status is "", so we have is_running = False
  • So _simulation_run_status ends up with is_processing = True and is_running = False, which makes it think it has some stale job running around, so calls runner.job_race_condition_reap to kill the job.
  • But that job we killed was in fact the real, correct job that our test was waiting for... so server.py tells the test that the job terminated unexpectedly, and the test isn't expecting that, so it crashes.

So it's a real bug, but given that (a) you're not hitting this in your production configuration, for whatever reason, and (b) this is exactly the code we're trying to get rid of (and this is why we're trying to get rid of it!), I don't think we should worry about this test failure.

@robnagler
Copy link
Member

Excellent debugging, thanks!

Yes, let's ignore the test failure.

I added checking nfs v4 on macs. I don't think it works, but it's what we use Linux-Linux, and it works much better, of course.

I'm in a crunch this week so I don't have time to review the code. I'll try to get the review done by this weekend.

@robnagler
Copy link
Member

There's an intermittent runner_test.py hang with a zombie sirepo process. The test process is in a loop with a rapid polling of runner.sock which no long exists:

select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=100000}) = 0 (Timeout)
stat("/home/vagrant/src/njsmith/sirepo/tests/runner_work/db/runner.sock", 0x7ffdb8c7df70) = -1 ENOENT (No such file or directory)
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=100000}) = 0 (Timeout)
stat("/home/vagrant/src/njsmith/sirepo/tests/runner_work/db/runner.sock",

Here's the ps:

 5655 pts/2    S+     0:00 bash test.sh
 5661 pts/2    S+     0:01 python setup.py test
 5887 pts/2    Sl+    0:00 python setup.py test
 5979 ?        Ssl    0:00 orted --hnp --set-sid --report-uri 23 --singleton-died-pipe 24 -mca state_novm_select 1
 5986 pts/2    Z+     0:00 [sirepo] <defunct>

orted is running, which happens because mpi4py starts it, but it is detached from the group so hard to say if it is the cause of the zombie. Here's

 4501 pts/0    Ss     0:00 -bash USER=vagrant LOGNAME=vagrant HOME=/home/vagrant PATH=/usr/local/bin:/usr/bin MAIL=/var/mail/vagrant SHELL=/bin/bash SSH_
 4843 pts/0    S+     0:00  \_ screen -r
 1145 pts/1    Ss     0:00 /bin/bash
 1405 pts/1    S+     0:03  \_ emacs
 1407 pts/2    Ss     0:00      \_ /bin/bash
 5655 pts/2    S+     0:00      |   \_ bash test.sh
 5661 pts/2    S+     0:01      |       \_ python setup.py test
 5887 pts/2    Sl+    0:00      |           \_ python setup.py test
 5986 pts/2    Z+     0:00      |               \_ [sirepo] <defunct>

The test is failing when run directly:

Traceback (most recent call last):
  File "/home/vagrant/src/njsmith/sirepo/tests/runner_test.py", line 86, in test_runner_myapp
    run.nextRequest
  File "/home/vagrant/src/radiasoft/pykern/pykern/pkcollections.py", line 56, in __getattr__
    return self.__getattribute__(name)
AttributeError: 'Dict' object has no attribute 'nextRequest'

@robnagler
Copy link
Member

The failure running individually was due to the 'complete' vs 'completed' that I had leftover. After I changed that, it worked, but the test still hangs from bash test.sh.

@njsmith
Copy link
Contributor Author

njsmith commented Feb 26, 2019

There's an intermittent runner_test.py hang with a zombie sirepo process

So this apparently is a difference between pytest tests and setup.py test. These seem to be the same except that if you do setup.py test then it sets PYTHONPATH to:

'PYTHONPATH': '/home/vagrant/src/radiasoft/sirepo:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/src/radiasoft/pykern:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages:/home/vagrant/.pyenv/versions/2.7.14/envs/py2/lib/python2.7/site-packages',

This is then inherited by our py3 process, and somehow things don't work out right when we try to run py3 with 122 copies of the py2 environment's site-packages added to its sys.path. If I add a del os.environ['PYTHONPATH'] at the top of the test, then setup.py test starts working again.

Maybe this makes sense to PJ Eby, but it sure doesn't make sense to me.

What environment does test.sh run in? Is it run after everything is installed into the local python environment, to check that it works-as-deployed? Is the goal to test the installed sirepo, or the sirepo in the source tree?

I'm trying to figure out which of these options is best:

  • modify test.sh, replacing setup.py test with pytest tests
  • modify test.sh, replacing setup.py test with PYTHONPATH=$PWD pytest tests
  • modify runner_test.py, to do the del os.environ['PYTHONPATH'] hack (which patches over the immediate problem, but breaks things if you want to set PYTHONPATH before running pytest)

@robnagler
Copy link
Member

I had to figure out why PYTHONPATH was being created so badly. I asked for these changes: pypa/setuptools#1709 They won't put it in, because I didn't follow procedures. That won't fix things for us right now.

I think the fix is del runner_env['PYTHONPATH'], because the entire python environment should be setup completely by pyenv exec.

test.sh runs in docker build. It performs validation, runs the javascript test, and deploys to pypi if it has the credentials to do so. Since you are calling pyenv exec directly, I don't think it can run in tox or other environments that modify PYTHONPATH. py3 and py2 have to be in agreement, and that's the only environment it needs to run in. That being said, test.sh runs pksetup.PKDeploy which runs tox, which will be a problem unless we skip this test, which we should in that case.

One thing I did realize is that the SIREPO_SRDB_ROOT setting is probably not enough. We should cascade the entire config environment. pkconfig could do that, but that's for another time.

@robnagler
Copy link
Member

This definitely fixes the python setup.py test hanging problem:

diff --git a/tests/runner_test.py b/tests/runner_test.py
index c2303bbf..ce950f6e 100644
--- a/tests/runner_test.py
+++ b/tests/runner_test.py
@@ -21,6 +21,7 @@ def test_runner_myapp():

     # Check if the py3 environment is set up
     py3_env = dict(os.environ)
+    del py3_env['PYTHONPATH']
     py3_env['PYENV_VERSION'] = 'py3'
     returncode = subprocess.call(
         ['pyenv', 'exec', 'sirepo', '--help'], env=py3_env

@jaraco
Copy link

jaraco commented Apr 5, 2019

I'm trying to figure out which of these options is best:
modify test.sh, replacing setup.py test with pytest tests

That option seems best to me. We're looking to deprecate setup.py <anything>, including test.

If you're relying on the setup_requires support that setup.py test offers, that's also deprecated. Our best recommendation if you need to install packages to run tests is to use tox.

@robnagler
Copy link
Member

Thanks for the notification @jaraco. I didn't know setup.py <anything> was deprecated.

I wonder if there's a plan for a command to create documentation, deployment, etc. per pypa/setuptools#1684 (comment). Is the plan to rely on tox for everything?

@jaraco
Copy link

jaraco commented Apr 6, 2019 •
edited
Loading

I wouldn’t say they’re deprecated yet, only discouraged. Tox may be suitable and it’s a model I’ve used for docs with success.

I guess it’s fair to say there isn’t a plan, just intentions and exploration and experimentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@robnagler robnagler

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jaraco @njsmith @robnagler