Split the test suite for spec versions

[asmeurer]

The 2021 version of the spec is going to be finalized soon. The test suite isn't finished yet. However, we want to have a version of the test suite that runs only against 2021, so that people can run the suite without having to worry about features that are in the 2022 draft.

We should figure out some way to enable or disable test suite features. It would also be nice to be able to do this for the extensions, so that people can easily run the test suite without the linear algebra extension, for instance. Perhaps the best way to do this is using pytest marks? Another idea would be to split the parts of the suite into submodules so that people can run pytest array_api_tests/2021 or pytest array_api_tests/2021/extensions/linear_algebra. For vendoring people could remove any submodules that aren't relevant.

I'm guessing pytest marks will be better, but I need to investigate them to see how well suited they are for this.

[honno]

Something useful I learnt recently—we can specify markers for all test methods in a module via assigning the pytestmark variable once per-file.

pytestmark = [...]

[honno]

Okay, a proposal. By default, test methods are assumed for all versions. If test methods do not cover every version, we pin the version via a mark.

# No marker
test_method_in_all_versions(...):
    ...

@pytest.mark.xp_version("2021.12")
test_method_only_in_one_version(...):
    ...

@pytest.mark.xp_version(">2021.12")
test_method_from_version(...):
    ...

@pytest.mark.xp_version("<=2022.09")
test_method_to_version(...):
    ...

@pytest.mark.xp_version(">2022.03,<=2022.09")
test_method_in_version_range(...):
    ...

In the pytest collection hook, we can get the xp version from a custom option, fallbacking to something reasonable (e.g. only testing with the unmarked methods? assuming the most recent version?). Then we can just dynamically add the skip mark pytest.mark.skip to any test case that doesn't fit the (parsed) acceptable version range.

[asmeurer]

That sounds reasonable, but we're also going to want to have a way to do skipping inside of the test function, like

if version == '2021.1':
    assert ...
elif version == '2022.1':
    assert ...

since I expect most changes will be minor things. Some changes will require separate functions, especially if they involve changing a function's signature.

Relatedly, we should probably just split the stubs like function_stubs/2021.1/ and so on (I think it's better to just duplicate the stubs rather than trying to reuse code as that makes it easier for library implementers to copy-paste them).

[honno]

I'll formalise my thoughts later, just to say we added versioning support to Hypothesis in HypothesisWorks/hypothesis#3456, which I'll probably take inspiration from when working on this issue.