Unable to disable / deselect currently unsupported dtypes

[pmeier]

I'm currently running this test suite against PyTorch. For now uint16, uint32, and uint64 are not supported. Unfortunately, this marks a lot of tests as failing completely instead of only failing for these dtypes. For example in

array-api-tests/array_api_tests/test_broadcasting.py

Lines 113 to 117 in b47e667

	@pytest.mark.parametrize('func_name', [i for i in
	elementwise_functions.__all__ if
	nargs(i) > 1])
	@given(shape1=shapes, shape2=shapes, dtype=data())
	def test_broadcasting_hypothesis(func_name, shape1, shape2, dtype):

only the operator name is regarded as parameter, whereas the shapes and the dtype in handled within the test.

To make this easier it should be possible to

• disable unsupported dtypes in one central place, or
• make the dtype a parameter so it can be deselected with pytest's -k flag.

[asmeurer]

I guess we should parameterize over the dtypes in this test. The test shouldn't actually need to worry about dtypes (it's unlikely that a function would not function correctly with broadcasting only with a specific dtype). The only reason we have it like this is because different functions accept different dtypes, so we have to pick one that is compatible.

[asmeurer]

Then again, this will also be a problem elsewhere. Right now I am only testing floating point in some places where I should be testing all dtypes, like the special cases tests. Making it easier to just disable some dtypes might be a better solution.

[asmeurer]

This will take some refactoring. I pushed a simple fix for the broadcasting tests for now. Let me know if you have this issue with any of the other tests. As far as I know, all the other tests that use arbitrary dtypes parameterize against them, so you should be able to just ignore the failures for the dtypes you don't care about (specifically the type promotion tests). That is except for the float dtypes, which are both assumed to exist.

[asmeurer]

Oh nevermind the elementwise tests are also done this way. Those probably actually should just parameterize over the dtypes, though it will take some refactoring.

[asmeurer]

And it looks like the special case tests actually do use integer dtypes. So those will need to be fixed as well.

[asmeurer]

I ended up going with a flag to allow undefined dtypes for tests that pick a dtype with hypothesis, as this was much simpler than converting tests to generate dtypes using pytest parameterization (honestly, if I could exclusively use hypothesis and never use pytest features, I would).

The flag is FILTER_UNDEFINED_DTYPES in array_api_tests/hypothesis_helpers.py and it is currently set to True by default. Let me know if you encounter any more issues with tests that can't be used because a dtype is undefined.

The type promotion tests still use normal pytest parameterization over dtypes, so for those, you will need to either skip or ignore those tests.

[djl11]

I'm still having issues with this. As an example, when running test_full() with PyTorch, and FILTER_UNDEFINED_DTYPES set to True and also command line flags -k 'not uint16 and not uint32 and not uint64' I get the following stack trace:

ivy_tests/test_array_api/array_api_tests/test_creation_functions.py:355 (test_full)
@given(
>      shape=hh.shapes(),
        fill_value=full_fill_values(),
        kw=st.shared(hh.kwargs(dtype=st.none() | xps.scalar_dtypes()), key="full_kw"),
    )

ivy_tests/test_array_api/array_api_tests/test_creation_functions.py:357: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ivy_tests/test_array_api/array_api_tests/test_creation_functions.py:353: in full_fill_values
    return draw(xps.from_dtype(dtype))
/usr/local/lib/python3.8/dist-packages/hypothesis/internal/conjecture/data.py:861: in draw
    strategy.validate()
/usr/local/lib/python3.8/dist-packages/hypothesis/strategies/_internal/strategies.py:406: in validate
    self.do_validate()
/usr/local/lib/python3.8/dist-packages/hypothesis/strategies/_internal/lazy.py:129: in do_validate
    w = self.wrapped_strategy
/usr/local/lib/python3.8/dist-packages/hypothesis/strategies/_internal/lazy.py:108: in wrapped_strategy
    base = self.function(*self.__args, **self.__kwargs)
/usr/local/lib/python3.8/dist-packages/hypothesis/extra/array_api.py:795: in from_dtype
    return _from_dtype(
/usr/local/lib/python3.8/dist-packages/hypothesis/extra/array_api.py:208: in _from_dtype
    iinfo = xp.iinfo(dtype)

Maybe this issue should be re-opened? Or am I making a mistake in my approach to running the test @asmeurer?

[honno]

@djl11 Did the exception message also print somewhere? If so, please share it. Not aware of any existing issues with using PyTorch in the test suite, as we have made efforts to make sure it's lack of uints (save uint8) doesn't hinder anything.

(Generally you might be interested in this probably-outdated summary of the outcomes of testing PyTorch.)

[djl11]

Yes my bad, there was an issue with my own code. Will take a bit more time checking through my own code in future. Appreciate the prompt response!

[djl11]

The prod test is still failing for me on this line with a PyTorch backend, even with FILTER_UNDEFINED_DTYPES = True and flag -k "not (uint16 or uint32 or uint64)"

The error is: AssertionError: out.dtype=uint8, but should be uint32 [prod(uint8)]

The PR in question with failing CI is here.

[honno]

The prod test is still failing for me

Ah this is interesting @djl11. The spec introduces a concept of "default unsigned integer" just for xp.prod()
that mandates a uint32 or uint64 (respective to the default integer) when x=<uint array>, dtype=None.

My initial impression is the test should remain as-is, because strictly PyTorch is non-compliant here. Of course we do support a lack of dtypes generally in the test suite, so maybe we could just skip this assertion if the relevant dtypes are not found. In any case, best to xfail/skip test_pow for now.

[djl11]

Thanks @honno, yes I will skip this for now then. As for possible updates to the test suite, I agree that skipping this assertion if the relevant dtypes are not found seems to be a good approach. Happy to try a PR for this myself or leave it to you guys.