Use the pytest-randomly plugin to run tests in random order

[seisman]

Description of proposed changes

Fixes #

Reminders

• Run make format and make check to make sure the code follows the style guide.
• Add tests for new features or tests that would have caught the bug that you're fixing.
• Add new public functions/methods/classes to doc/api/index.rst.
• Write detailed docstrings for all functions/methods.
• If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
• If adding new functionality, add an example to docstrings or tutorials.
• Use underscores (not hyphens) in names of Python files and directories.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

• /format: automatically format and lint the code
• /test-gmt-dev: run full tests on the latest GMT development version

[seisman]

@weiji14 Randomly running the tests finds a minor bug in PR #2774. Some tests like below fail:

______________________test_virtualfile_from_matrix_slice______________________dtypes= ['int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', ...]
deftest_virtualfile_from_matrix_slice(dtypes):
""" Test transforming a slice of a larger array to virtual file dataset. """shape= (10, 6)
fordtypeindtypes:
>full_data=np.arange(shape[0] *shape[1], dtype=dtype).reshape(shape)
ETypeError: data type 'int8[pyarrow]'notunderstood

The failures are because dtypes is a list and is mutable, so changes in dtypes affect all later tests:

pygmt/pygmt/tests/test_clib_virtualfiles.py

Lines 328 to 329 in 925e90e

	iffind_spec("pyarrow") isnotNone:
	dtypes.extend([f"{dtype}[pyarrow]"fordtypeindtypes])

[weiji14]

The failures are because dtypes is a list and is mutable, so changes in dtypes affect all later tests:

Good spotting, I opened a PR at #2941 to fix this.

[seisman]

Note to myself:

This is the command to reproduce the errors stably

$ pytest --randomly-seed=2418297293 pygmt/tests/test_grdimage_image.py pygmt/tests/test_grdclip.py
pygmt/tests/test_grdclip.py::test_grdclip_no_outgrid PASSED [ 9%]
pygmt/tests/test_grdclip.py::test_grdclip_outgrid PASSED [ 18%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[uint32] PASSED [ 27%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[int16] PASSED [ 36%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[float64] PASSED [ 45%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[uint16] PASSED [ 54%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image FAILED [ 63%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[float32] PASSED [ 72%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[int8] PASSED [ 81%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[int32] PASSED [ 90%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray FAILED

If adding verbose="d" to the test_grdimage_image test, we will see debugging messages like:

...
pygmt-session [DEBUG]: Input remote grid modified to have registration: @earth_day_01d_p
pygmt-session [DEBUG]: Revised remote file name to @earth_day_01d_p
pygmt-session [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter.
pygmt-session [DEBUG]: Use PS filename /Users/seisman/.gmt/sessions/gmt_session.6186/gmt_5.ps-
pygmt-session [DEBUG]: gmtinit_get_current_panel: No current panel selected so not in subplot mode
pygmt-session [DEBUG]: Modern mode: Added -R to options since history is available [-53/-49/-19/-16].
pygmt-session [DEBUG]: Modern: Adding -JQ15c+du to options since there is no history available.
pygmt-session [DEBUG]: Revised options: @earth_day_01d_p -Vd -R -JQ15c+du
grdimage [DEBUG]: History: Process -R
grdimage [DEBUG]: History: Process -JQ15c+du
grdimage [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter.
grdimage [DEBUG]: Got regular w/e/s/n for region (-53/-49/-19/-16)
...

The region (-53/-49/-19/-16) comes from history and is leaked from test_grdclip_no_outgrid.

[seisman]

import pytest
from pygmt import Figure, grdclip
from pygmt.helpers.testing import load_static_earth_relief
def test_grdclip_no_outgrid():
"""
Test the below and above parameters for grdclip with no set outgrid.
"""
grid = load_static_earth_relief()
grdclip(
grid=grid, below=[550, -1000], above=[700, 1000], region=[-53, -49, -19, -16]
)
@pytest.mark.mpl_image_compare
def test_grdimage_image():
"""
Plot a 3-band RGB image using file input.
"""
fig = Figure()
fig.grdimage(grid="@earth_day_01d", verbose="d")
return fig

Save the content above into a test file pygmt/test/test_random.py and run pytest pygmt/test/test_random.py to reproduce the failures. The reason for the failures is because we're using a single GMT CLI session, so the region setting from test_grdclip_no_outgrid is saved in GMT session history and is further used in test_grdimage_image. If we swap the order of the two tests, then they pass.

Here are two normal scripts to understand the differences.

from pygmt import Figure, grdclip
from pygmt.helpers.testing import load_static_earth_relief
grid = load_static_earth_relief()
grdclip(
grid=grid, below=[550, -1000], above=[700, 1000], region=[-53, -49, -19, -16]
)
fig = Figure()
fig.grdimage(grid="@earth_day_01d", verbose="d")
fig.show()

from pygmt import Figure, grdclip
from pygmt.helpers.testing import load_static_earth_relief
fig = Figure()
fig.grdimage(grid="@earth_day_01d", verbose="d")
fig.show()
grid = load_static_earth_relief()
grdclip(
grid=grid, below=[550, -1000], above=[700, 1000], region=[-53, -49, -19, -16]
)

Related to #1242 but is more complicated than that one.