Use pytest-doctestplus to skip some inline doctests

[seisman]

Description of proposed changes

We're adding more and more inline code examples in function docstrings as tracked in issue #1686. We don't want these new doctests slow down our CI, so we skip these doctests using the # doctest: +SKIP directive. Here is an example doctest in grdlandmask:

 >>> import pygmt # doctest: +SKIP
>>> # Create a landmask grid with an x-range of 125 to 130,
>>> # and a y-range of 30 to 35
>>> landmask = pygmt.grdlandmask(
... spacing=1, region=[125, 130, 30, 35]
... ) # doctest: +SKIP

As you can see, there are many downsides for this kind of doctests:

1. # doctest: +SKIP only works for one single line, so we have to add # doctest: +SKIP many times
2. # doctest: +SKIP has 18 characters. So even a short code like landmask = pygmt.grdlandmask(spacing=1, region=[125, 130, 30, 35]) must be split into multiple lines
3. there is no simple way to run these tests

In this PR, the pytest-doctestplus plugin is used to solve the problems above. For each module, we just need to define an internal variable __doctest_skip__, which contains a list of functions/classes whose tests should be skipped.
See the changes in pygmt/src/grdlandmask.py as an example and also the official documentation of pytest-doctestplus.

As you can see, it solves the problems 1 and 2 above.

Before	After

For problem 3, there is still no easy way, but I think it's possible to comment the __doctest_skip__ line, run the tests, and then revert the changes.

[seisman]

Ping @meghanrjones @weiji14@willschlitzer@michaelgrund for comments on the new doctest method.

[weiji14]

Great find! Should have found this earlier 😆 I have one suggestion in regard to Problem 3 below.

[weiji14]

For problem 3, there is still no easy way, but I think it's possible to comment the doctest_skip line, run the tests, and then revert the changes.

Actually, from reading the doctest-plus docs (https://github.com/astropy/pytest-doctestplus/tree/v0.12.0#setup-and-configuration), it says:

Using pytest's built-in --doctest-modules option will override the behavior of this plugin, even if doctest_plus = enabled in setup.cfg, and will cause the default doctest plugin to be used. However, if for some reason both --doctest-modules and --doctest-plus are given, the pytest-doctestplus plugin will be used, regardless of the contents of setup.cfg.

So maybe we just need to edit the Makefile, and have have a make doctest command that uses --doctest-modules instead of doctest-plus to run all of these doctests? Could also setup a CI to have it run weekly if needed.

[seisman]

Good suggestion. I removed the --doctest-plus option from pyproject.toml, and added it to make test command. I also the make fulltest target with the --doctest-modules option (I think the make doctest command is misleading).

Now, running make test collects 541 tests, and running make fulltest collects 542 tests. The extra one in make fulltest comes is the grdlandmask doctest.

[maxrjones]

This is great!

I think the contributors guide should provide information about adding inline examples/doctests including how to skip tests using __doctest_skip__ and run the full tests using make fulltest or pytest pygmt/src/<module>.py --doctest-modules, but this could be handled in a separate PR.

[seisman]

pytest pygmt/src/.py --doctest-modules

I added the --doctest-modules option back to pytest's default options so that people don't have to give the --doctest-modules option when running tests in a single module.

Now here are three different ways to run tests:

• make test: use the default pytest options (i.e., --doctest-modules --doctest-plus), so that all tests and some doctests will run
• make fulltest: use --doctest-modules, all tests and all doctests will run
• pytest pygmt/src/<module>.py: any doctests in the module will run