Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a --pdb option to stestr run #14

Closed
mtreinish opened this issue Jan 3, 2017 · 8 comments · Fixed by #276
Closed

Add a --pdb option to stestr run #14

mtreinish opened this issue Jan 3, 2017 · 8 comments · Fixed by #276

Comments

@mtreinish
Copy link
Owner

Right now if you want to run with pdb stestr breaks that. The way to workaround this is to call subunit.run or testtools.run directly. We point this out in the stestr development docs:

http://stestr.readthedocs.io/en/latest/developer_guidelines.html#running-the-tests

However there is no reason we can't just fast path stestr to call the testr runner directly if we want to use pdb. We probably don't want to store results in the repository in that case.

For an example of this being done, we added this to os-testr: https://github.com/openstack/os-testr/blob/master/os_testr/ostestr.py#L197-L210 we probably could steal that logic and dump it into stestr.commands.run.

@mtreinish
Copy link
Owner Author

So we added the - -no-discover option already which does the same thing and calls subunit.run directly. All we need to do here is add to the docs around pdb usage to tell people what to do.

@mtreinish
Copy link
Owner Author

I actually don't know if --no-discover will actually work for using pdb. It still subprocesses out which might cause an issue with pdb. Someone will need to test it works correctly with --no-discover before we call it implemented and just document how to do it.

@aspiers
Copy link

aspiers commented Feb 18, 2019

I just tested it and it works:

$ stestr --test-path=./nova/tests/functional run -n nova.tests.functional.test_servers.TraitsTrackingTests.test_resource_provider_traits
> /home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/placement/requestlog.py(44)__call__()
-> accept = environ.get('HTTP_ACCEPT')
(Pdb)

In contrast:

$ stestr --test-path=./nova/tests/functional run TraitsTrackingTests
> /home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/placement/requestlog.py(44)__call__()
-> accept = environ.get('HTTP_ACCEPT')
(Pdb)
{0} nova.tests.functional.test_servers.TraitsTrackingTests.test_resource_provider_traits [4.816931s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/mock/mock.py", line 1724, in _inner
        return f(*args, **kw)
      File "nova/tests/functional/test_servers.py", line 2047, in test_resource_provider_traits
        self._delete_trait('CUSTOM_FOO')
      File "nova/tests/functional/integrated_helpers.py", line 445, in _delete_trait
        return self.placement_api.delete('/traits/%s' % trait, version='1.6')
      File "nova/tests/functional/fixtures.py", line 45, in delete
        self.fixture._fake_delete(None, url, **kwargs))
      File "nova/tests/functional/fixtures.py", line 158, in _fake_delete
        headers={'x-auth-token': self.token})
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 366, in delete
        return self.request(url, 'DELETE', **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
        return self.session.request(url, method, **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/session.py", line 814, in request
        resp = send(**kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/keystoneauth1/session.py", line 903, in _send_request
        resp = self.session.request(method, url, **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
        resp = self.send(prep, **send_kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
        r = adapter.send(request, **kwargs)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/requests/adapters.py", line 449, in send
        timeout=timeout
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
        chunked=chunked)
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/urllib3/connectionpool.py", line 377, in _make_request
        httplib_response = conn.getresponse(buffering=True)
      File "/usr/lib64/python2.7/httplib.py", line 1118, in getresponse
        response = self.response_class(*args, **kwds)
      File "/usr/lib64/python2.7/httplib.py", line 368, in __init__
        self.fp = sock.makefile('rb')
      File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/functional/lib/python2.7/site-packages/wsgi_intercept/__init__.py", line 449, in makefile
        app_result = self.app(environ, start_response)
    wsgi_intercept.WSGIAppError: BdbQuit() at /usr/lib64/python2.7/bdb.py:68

@dshcherb
Copy link

dshcherb commented Jun 27, 2019

-n or --no-discover works for me too for executing a single test case with traces set up:

source .tox/py37/bin/activate
stestr run --no-discover unit_tests.test_actions.TestKeystoneSAMLMellonActions                   
> /home/foobar/src/openstack/charm-keystone-saml-mellon/unit_tests/test_actions.py(45)test_get_sp_metadata()
     44 
---> 45         service_name = 'keystone-foobar-mellon'
     46         self.service_name.return_value = service_name
tox -e py37 -- -n unit_tests.test_actions.TestKeystoneSAMLMellonActions                     
py37 installed: asn1crypto==0.24.0,attrs==19.1.0,backcall==0.1.0,cffi==1.12.3,charmhelpers==0.19.13,charms.openstack==0.0.1.dev1,charms.reactive==1.2.1,cliff==2.15.0,cmd2==0.9.13,colorama==0.4.1,coverage==4.5.3,cryptography==2.7,decorator==4.4.0,entrypoints==0.3,extras==1.0.0,fixtures==3.0.0,flake8==3.7.7,future==0.17.1,ipdb==0.12,ipython==7.5.0,ipython-genutils==0.2.0,jedi==0.14.0,Jinja2==2.10.1,linecache2==1.0.0,lxml==4.3.4,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,netaddr==0.7.19,nose==1.3.7,parso==0.5.0,pbr==5.3.1,pexpect==4.7.0,pickleshare==0.7.5,pkg-resources==0.0.0,prettytable==0.7.2,prompt-toolkit==2.0.9,ptyprocess==0.6.0,pyaml==19.4.1,pycodestyle==2.5.0,pycparser==2.19,pyflakes==2.1.1,Pygments==2.4.2,pyparsing==2.4.0,pyperclip==1.7.0,python-mimeparse==1.6.0,python-subunit==1.3.0,PyYAML==5.1.1,six==1.12.0,stestr==2.3.1,stevedore==1.30.1,Tempita==0.5.2,testtools==2.3.0,traceback2==1.4.0,traitlets==4.3.2,unittest2==1.1.0,voluptuous==0.11.5,wcwidth==0.1.7
py37 run-test-pre: PYTHONHASHSEED='0'
py37 runtests: commands[0] | stestr run -n unit_tests.test_actions.TestKeystoneSAMLMellonActions
> /home/foobar/src/openstack/charm-keystone-saml-mellon/unit_tests/test_actions.py(45)test_get_sp_metadata()
     44 
---> 45         service_name = 'keystone-foobar-mellon'
     46         self.service_name.return_value = service_name

@mtreinish
Copy link
Owner Author

Ok cool, then lets document this as the recommended way to run tests with pdb in the stestr man page and close the issue.

@aspiers
Copy link

aspiers commented Jun 27, 2019

@mtreinish commented on June 27, 2019 2:39 PM:

Ok cool, then lets document this as the recommended way to run tests with pdb in the stestr man page and close the issue.

Just as a reminder, even with -n there are issues: #186 (comment)

@mtreinish
Copy link
Owner Author

@aspiers when you get a second can you try running your pdb workflow with: #271 The code there isn't quite ready to merge yet (it's been a wish list feature I've been working on and off for some time in the background that I finally got running) but it launches things in parallel using multiprocessing.Process and passing the subunit stream as a pipe instead of a subprocess and reading stdout. I'm hoping this will have better support for parallel pdb usage than what was there before.

@aspiers
Copy link

aspiers commented Sep 9, 2019

After adding import pdb; pdb.set_trace() to a unit testcase and fixing the TypeError: 'sys.version_info' object is not callable issue, I still see issues:

nova $ stestr run --dynamic -n nova.tests.unit.virt.libvirt.test_designer
/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py:374: UserWarning: WARNING: The dynamic scheduler is still experimental. You might encounter issues while using it
  warnings.warn("WARNING: The dynamic scheduler is still experimental. "
'Popen' object has no attribute 'read'
Traceback (most recent call last):
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 237, in take_action
    all_attachments=all_attachments, dynamic=args.dynamic)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 410, in run_command
    return run_tests()
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 407, in run_tests
    all_attachments=all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 234, in load
    all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 267, in _load_case
    case.run(result)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/testtools/testsuite.py", line 158, in run
    for test, route_code in tests:
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 208, in make_tests
    stream, non_subunit_name='stdout')
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/v2.py", line 272, in __init__
    self.source = subunit.make_stream_binary(source)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1278, in make_stream_binary
    return _unwrap_text(stream)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1298, in _unwrap_text
    if type(stream.read(0)) is unicode_type:
AttributeError: 'Popen' object has no attribute 'read'
Traceback (most recent call last):
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/bin/stestr", line 10, in <module>
    sys.exit(main())
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/cli.py", line 118, in main
    return cli.run(argv)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 237, in take_action
    all_attachments=all_attachments, dynamic=args.dynamic)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 410, in run_command
    return run_tests()
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/run.py", line 407, in run_tests
    all_attachments=all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 234, in load
    all_attachments)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 267, in _load_case
    case.run(result)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/testtools/testsuite.py", line 158, in run
    for test, route_code in tests:
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/src/stestr/stestr/commands/load.py", line 208, in make_tests
    stream, non_subunit_name='stdout')
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/v2.py", line 272, in __init__
    self.source = subunit.make_stream_binary(source)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1278, in make_stream_binary
    return _unwrap_text(stream)
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/subunit/__init__.py", line 1298, in _unwrap_text
    if type(stream.read(0)) is unicode_type:
AttributeError: 'Popen' object has no attribute 'read'
nova $ Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: oslo_context.context, urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
--- Logging error ---
Traceback (most recent call last):
  File "/home/adam/.pyenv/versions/3.6.1/lib/python3.6/logging/__init__.py", line 996, in emit
    self.flush()
  File "/home/adam/.pyenv/versions/3.6.1/lib/python3.6/logging/__init__.py", line 976, in flush
    self.stream.flush()
BrokenPipeError: [Errno 32] Broken pipe
Call stack:
  File "/home/adam/SUSE/cloud/OpenStack/git/nova/.tox/py36/lib/python3.6/site-packages/oslo_log/log.py", line 203, in logging_excepthook
    getLogger(product_name).critical('Unhandled error', **extra)
Message: 'Unhandled error'
Arguments: ()
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

mtreinish added a commit that referenced this issue Oct 7, 2019
This commit adds a new flag --pdb to stestr running a test id (in a
similar fashion to --no-discover) for when pdb is going to be invoked
from the tests. Pdb has trouble running in situations where another
process is launched (which is pretty much all of stestr, including with
--no-discover). This commit adds a --pdb flag so that you can launch
the test runner without any additional processes being launched. This
will run the tests in a process capture the subunit output to an buffer
and after the tests finish executing will pass that to the load command
so it can write it to the repository and generate subunit-trace or any
other output format (per the set flags). The tradeoff here is that the
output will not be real time from the test instead it will only be
processed until after the test executes.

One thing to note is that if you're using a fixture to capture stdout
that will sometimes take the output from pdb so you will be in the
debugger but get no output. If you plan to debug a test suite with a
fixture make sure to disable that first.

Fixes #14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants