Surprising test run order when test function reused

[samueljsb]

issue

When a class is composed form another class, the tests do not run in the order they are defined.

e.g:

import pytest


class TestFoo:
    def test_foo(self, request):
        pass

    def test_bar(self, request):
        pass


class TestBar:
    def test_foo(self, request):
        pass

    test_bar = TestFoo.test_bar

$ pytest --verbose
…
test_test.py::TestFoo::test_foo PASSED
test_test.py::TestFoo::test_bar PASSED
test_test.py::TestBar::test_bar PASSED
test_test.py::TestBar::test_foo PASSED
…

I would expect the last 2 tests to be executed in the opposite order (i.e. TestBar::test_foo then TestBar::test_bar) since this is the order they appear in the file. I'm not sure this is a bug per se, but it's something I'd like to see documented if it's decided it should stay this way.

venv

$ pip list
Package        Version
-------------- -------
appdirs        1.4.3
atomicwrites   1.3.0
attrs          19.1.0
black          19.3b0
Click          7.0
more-itertools 7.0.0
pip            19.1
pluggy         0.9.0
py             1.8.0
pytest         4.4.1
setuptools     40.6.2
six            1.12.0
toml           0.10.0

platform darwin -- Python 3.7.2, pytest-4.4.1, py-1.8.0, pluggy-0.9.0

MacOS 10.14.4 (Mojave)

[asottile]

I believe tests are attempted to be executed in definition order. The TestBar.test_bar is identical to the Testfoo.test_bar, so the line number / positioning appears earlier in the file than TestBar.test_foo

[samueljsb]

I figured that might be the case. Would we want to change this to run in the expected order*, or is the current behaviour desirable?

*This is definitely not going to be a trivial change, and may well be more effort than it’s worth, but I’m willing to have a poke around if it’s worth it

[asottile]

I'm not sure it's possible to support this for python<3.6 since it's not known where the assignment is from just the imported module (though maybe we could mark up the ast with an import hook? hmmmm)

but once we're py36+ I think we can stop sorting and let the definition order handle it (due to ordered mapping types by default)

Just curious -- is this breaking something for you?

[asottile]

didn't mean to close, oops

[samueljsb]

I have some incremental tests (end-to-end UI tests) where I wanted to reuse a class and just change one of the methods, so yes, but it isn’t a big deal.

Might be something to revisit if/when we drop 2.7 and 3.5 support, but sounds like a no-go for now. I’m happy to close this until then.

[nicoddemus]

Agreed. Closing in favor of #5196 then.