Test with pytest from master/features #79

nicoddemus · 2017-08-31T21:40:47Z

The environment for pytest-features is failing because pytest currently pins pluggy to >=4.0,<5, so tox cannot create the environment.

I propose we change pytest's setup.py to optionally take the pluggy specification from an environment variable _PYTEST_SETUP_PLUGGY_SPEC if defined instead of what's encoded in setup.py, roughly:

def main():
    install_requires = ['py>=1.4.33', 'six>=1.10.0','setuptools', 'pluggy>=0.4.0,<0.5']

Into:

def main():
    pluggy_spec = os.environ.get('_PYTEST_SETUP_PLUGGY_SPEC', 'pluggy>=0.4.0,<0.5')
    install_requires = ['py>=1.4.33', 'six>=1.10.0','setuptools', pluggy_spec]

We will probably need to do something like that anyway for pytest-dev/pytest#2737.

RonnyPfannschmidt · 2017-09-01T06:34:17Z

tox.ini

-  pytest28: pytest~=2.8.0
-  pytest29: pytest~=2.9.0
-  pytest30: pytest~=3.0.0
+  pytest


why completely remove those?

I thought we agreed that it didn't make sense to run the tests using old pytest versions given that those versions vendor pluggy anyway.

we should leave one with pytest 3.2 to have one "working" pytest version

That's the pytest version we use for the normal test environments for all python versions, so we're covered. For example: https://travis-ci.org/pytest-dev/pluggy/jobs/270596649

All the "normal" builds (Python 2.6 and TOEXENV=py26) executes with the latest pytest.

RonnyPfannschmidt · 2017-09-01T06:34:45Z

tox.ini

+  git+https://github.com/pytest-dev/pytest.git@master
+
+[testenv:pytest-features]
+commands=py.test {posargs:testing/}


those could just be factors as well

Those are not factors on purpose because I don't think we need to test pytest master and develop on all python versions again, only on py27 and py36 seems good enough to me to save CI time. What do you think?

we dont have to express those factors for all python versions

I agree, and that's what I am doing here... perhaps I'm misunderstanding something, could you clarify.

For reference, take a look at the build matrix that was executed by this change: https://travis-ci.org/pytest-dev/pluggy/builds/270596643

you made own environments, not factors

Sorry @RonnyPfannschmidt, still don't know what you want me to do. 😕

Let me try to illustrate.

Right now we have the following builds:

py26 (pytest: 3.2.1) py27 (pytest: 3.2.1) py34 (pytest: 3.2.1) py34 (pytest: 3.2.1) py35 (pytest: 3.2.1) py36 (pytest: 3.2.1) pypy (pytest: 3.2.1) py37 (pytest: 3.2.1) benchmark (python: 2.7) benchmark (python: 3.6) check (python: 2.7) check (python: 3.6) docs (python: 3.6) pytest-master (python: 2.7, pytest: master) pytest-master (python: 3.6, pytest: master) pytest-features (python: 2.7, pytest: features) pytest-features (python: 3.6, pytest: features)

you made own environments, not factors

From this I understand you want me to change [envlist] to:

[tox] envlist=check,py{26,27,34,35,36,py}-pytest{release,master,features}

But this would explode the builds matrix to:

py26 (pytest: 3.2.1) py27 (pytest: 3.2.1) py34 (pytest: 3.2.1) py34 (pytest: 3.2.1) py35 (pytest: 3.2.1) py36 (pytest: 3.2.1) pypy (pytest: 3.2.1) py37 (pytest: 3.2.1) py26 (pytest: master) py27 (pytest: master) py34 (pytest: master) py34 (pytest: master) py35 (pytest: master) py36 (pytest: master) pypy (pytest: master) py37 (pytest: master) py26 (pytest: features) py27 (pytest: features) py34 (pytest: features) py34 (pytest: features) py35 (pytest: features) py36 (pytest: features) pypy (pytest: features) py37 (pytest: features) benchmark (python: 2.7) benchmark (python: 3.6) check (python: 2.7) check (python: 3.6) docs (python: 3.6)

Is that it? If so I don't think we need it.

If that's not it, could you clarify then? If you know exactly what syntax you want feel free to write it as well.

envlist=check,py{26,27,34,35,36,py}-pytestrelease,py{27,36}-pytest{master,features}

OK, I think I got it now, thanks.

RonnyPfannschmidt

.

Fix pytest-dev#76

goodboy · 2017-09-02T16:36:16Z

tox.ini

-  pytest28: pytest~=2.8.0
-  pytest29: pytest~=2.9.0
-  pytest30: pytest~=3.0.0
+  pytestrelease: pytest


👍 perfect

goodboy · 2017-09-02T16:39:17Z

.travis.yml

    - python: '3.6'
      env: TOXENV=check
+    - python: '3.6'


I have to say I've always found this a bit funny/redundant that we're routing through tox when Travis is really enabling the same thing via it's own job generation system. This include matrix just ends up being a big combo-list defeating the entire purpose. I think tox makes sense if you don't have a generative system built into your CI server already.

That's another thing tox doesn't support parallel jobs does it?
That's something that'd be super handy when testing locally.
I'd also be interested in using travis' stages to do docs and benchmarking runs in parallel with the regular suite.

his include matrix just ends up being a big combo-list defeating the entire purpose. I think tox makes sense if you don't have a generative system built into your CI server already.

One reason to also use tox in this context is to avoid duplicating logic from it like dependency installation and such.

I'd also be interested in using travis' stages to do docs and benchmarking runs in parallel with the regular suite.

Actually all jobs already run in parallel, give or take Travis current work availability when the build runs.

goodboy · 2017-09-02T16:44:37Z

@nicoddemus for this:

The environment for pytest-features is failing because pytest currently pins pluggy to >=4.0,<5, so tox cannot create the environment.

~~I'm not entirely following why this happens? Don't we already have 0.5.1 up?~~
Wait nm, got it. We're trying to install al local version which sits outside that interval.

goodboy

Even though I'm cranky about the matrix stuff ;) looks good!

RonnyPfannschmidt · 2017-09-02T19:27:07Z

so pytest needs a fix before merging this one

nicoddemus · 2017-09-05T22:12:01Z

We don't need an actual release, we can trigger this build again once we merge pytest-dev/pytest#2744.

nicoddemus · 2017-09-05T22:20:50Z

Tested locally by changing pytestfeatures to use pluggy-master from my fork and it works. 👍

RonnyPfannschmidt · 2017-09-06T16:00:58Z

merging on your word then, thanks !

nicoddemus · 2017-09-06T16:12:24Z

Err, pytest-dev/pytest#2744 should have been merged first.

nicoddemus · 2017-09-06T16:12:33Z

Sorry if that was not clear.

RonnyPfannschmidt · 2017-09-06T16:33:48Z

oh, whops - to me it did read like its tested and ready

nicoddemus · 2017-09-06T16:41:12Z

No worries, let's merge pytest-dev/pytest#2744 asap then. 👍

nicoddemus · 2017-09-06T19:28:37Z

I didn't manage to restart Travis, but AppVeyor restarted and worked so I think we are OK.

goodboy · 2017-09-06T21:01:18Z

@nicoddemus i just restarted it. Badge should show the result.

nicoddemus · 2017-09-06T21:05:53Z

It is not building the latest master for some reason (there's no pytestrelease/pytestfeatures envs): https://travis-ci.org/pytest-dev/pluggy/builds/254165070

No idea why is that.

nicoddemus requested review from RonnyPfannschmidt and goodboy August 31, 2017 21:40

RonnyPfannschmidt reviewed Sep 1, 2017

View reviewed changes

RonnyPfannschmidt requested changes Sep 1, 2017

View reviewed changes

Test with pytest from master/features

8ff2e9d

Fix pytest-dev#76

nicoddemus force-pushed the ci-pytest branch from 6901640 to 8ff2e9d Compare September 1, 2017 21:17

nicoddemus mentioned this pull request Sep 1, 2017

Add test environment using pluggy from master branch pytest-dev/pytest#2744

Merged

goodboy reviewed Sep 2, 2017

View reviewed changes

tox.ini

pytest28: pytest~=2.8.0

pytest29: pytest~=2.9.0

pytest30: pytest~=3.0.0

pytestrelease: pytest

Copy link

Contributor

goodboy Sep 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 perfect

goodboy reviewed Sep 2, 2017

View reviewed changes

goodboy approved these changes Sep 2, 2017

View reviewed changes

RonnyPfannschmidt approved these changes Sep 2, 2017

View reviewed changes

goodboy mentioned this pull request Sep 5, 2017

0.5.2 release #82

Closed

Test using the locally installed pluggy

04a0257

RonnyPfannschmidt merged commit b1b9de2 into pytest-dev:master Sep 6, 2017

nicoddemus deleted the ci-pytest branch September 6, 2017 16:12

This was referenced Nov 24, 2017

Update pluggy to 0.6.0 amureki/whoisp#76

Closed

Update pluggy to 0.6.0 AngellusMortis/django_microsoft_auth#3

Merged

This was referenced Dec 8, 2017

Initial Update anapaulagomes/saul#8

Merged

Update pluggy to 0.6.0 davidemoro/cookiecutter-qa#220

Merged

This was referenced Dec 20, 2017

Initial Update uwcirg/truenth-portal#1721

Closed

Update pluggy to 0.6.0 uwcirg/truenth-portal#1729

Closed

Update pluggy to 0.6.0 uwcirg/truenth-portal#1742

Merged

pyup-bot mentioned this pull request Jan 19, 2018

Update pluggy to 0.6.0 Poorvak/http_pipeline#8

Merged

pyup-bot mentioned this pull request Feb 8, 2018

Initial Update aio-libs/aiocassandra#9

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test with pytest from master/features #79

Test with pytest from master/features #79

nicoddemus commented Aug 31, 2017

RonnyPfannschmidt Sep 1, 2017

nicoddemus Sep 1, 2017

RonnyPfannschmidt Sep 1, 2017

nicoddemus Sep 1, 2017

nicoddemus Sep 1, 2017

RonnyPfannschmidt Sep 1, 2017

nicoddemus Sep 1, 2017

RonnyPfannschmidt Sep 1, 2017

nicoddemus Sep 1, 2017

RonnyPfannschmidt Sep 1, 2017

nicoddemus Sep 1, 2017

RonnyPfannschmidt Sep 1, 2017

nicoddemus Sep 1, 2017

RonnyPfannschmidt left a comment

goodboy Sep 2, 2017

goodboy Sep 2, 2017

nicoddemus Sep 5, 2017

goodboy commented Sep 2, 2017 •

edited

Loading

goodboy left a comment

RonnyPfannschmidt commented Sep 2, 2017

nicoddemus commented Sep 5, 2017

nicoddemus commented Sep 5, 2017

RonnyPfannschmidt commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

RonnyPfannschmidt commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

goodboy commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

Test with pytest from master/features #79

Test with pytest from master/features #79

Conversation

nicoddemus commented Aug 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RonnyPfannschmidt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goodboy commented Sep 2, 2017 • edited Loading

goodboy left a comment

Choose a reason for hiding this comment

RonnyPfannschmidt commented Sep 2, 2017

nicoddemus commented Sep 5, 2017

nicoddemus commented Sep 5, 2017

RonnyPfannschmidt commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

RonnyPfannschmidt commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

goodboy commented Sep 6, 2017

nicoddemus commented Sep 6, 2017

goodboy commented Sep 2, 2017 •

edited

Loading