Adding ability to run doctests with datastore system tests. by dhermes · Pull Request #2738 · googleapis/google-cloud-python

dhermes · 2016-11-15T04:40:17Z

Since Travis / CircleCI won't run this by default, here is some sample output:

$ tox -e system-tests -- datastore
GLOB sdist-make: .../setup.py
system-tests inst-nodeps: .../.tox/dist/google-cloud-0.21.0.zip
...
system-tests runtests: commands[1] | python .../system_tests/attempt_system_tests.py datastore
test_it (datastore.TestDoctest) ... Running Sphinx v1.4.8
making output directory...
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [doctest]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] contents
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
running tests...
Document: contents
------------------
1 items passed all tests:
9 tests in constructors
9 tests in 1 items.
9 passed and 0 failed.
Test passed.
Doctest summary
===============
9 tests
0 failures in tests
0 failures in setup code
0 failures in cleanup code
build succeeded.
ok
----------------------------------------------------------------------
Ran 1 test in 1.088s
OK
____________________________________ summary ____________________________________
system-tests: commands succeeded
congratulations :)

Sign in to view

+ def test_it(self):
+ from sphinx import application
+
+ docs_dir = self._make_temp_docs()


Sign in to view

+ return docs_dir
+
+ def test_it(self):
+ from sphinx import application


dhermes · 2016-11-16T21:42:22Z

@tseaver PTAL

dhermes · 2016-11-18T21:26:30Z

@tseaver PTAL

Sign in to view

 def __repr__(self):
- return '<Key%s, project=%s>' % (self.path, self.project)
+ return '<Key%s, project=%s>' % (self._flat_path, self.project)



Sign in to view

+ >>> entity['answer'] = 42
+ >>> entity
+ <Entity('EntityKind', 1234) {'answer': 42}>
+ >>> query = client.query(kind='EntityKind')


Sign in to view

 super(Entity, self).__repr__())
 else:
- return '<Entity %s>' % (super(Entity, self).__repr__())
+ return '<Entity %s>' % (super(Entity, self).__repr__(),)


Sign in to view

- self.assertEqual(repr(entity), "<Entity/bar/baz {'foo': 'Foo'}>")
+ entity_vals = {'foo': 'Foo'}
+ entity.update(entity_vals)
+ expected = '<Entity%s %s>' % (flat_path, entity_vals)


dhermes · 2016-11-28T19:35:35Z

@tseaver Only comments on the doctest itself / associated repr() changes.

Any comments on the approach of making a fake docs directory and then running doctest there?

tseaver · 2016-11-28T19:56:47Z

Any comments on the approach of making a fake docs directory and then running doctest there?

We're going to need a way to test all the examples, not just those embedded in docstrings: do you have a plan for that?

tseaver · 2016-11-29T04:29:55Z

@dhermes Memory eludes me: what stops us from just running sphinx-build -b doctest in place in the normal docs directory, rather than synthesizing one?

dhermes · 2016-11-29T04:39:13Z

We're going to need a way to test all the examples, not just those embedded in docstrings: do you have a plan for that?

Yes, though that is "beyond" the scope of this PR. There are two possible approaches that I can see. The first is to "guess" which hand-written RST files apply to a given package (I am not "pro" on this approach). The second is as follows:

Discontinue handwritten RST files and use sphinx-apidoc to generate our docs (I wrote the docs gen for oauth2client using this and @jonparrott "copied" that approach in https://github.com/GoogleCloudPlatform/google-auth-library-python/)
- In this world, our "package usage" docs will go in google/cloud/foo/__init__.py
Add checks that literal blocks are as we expect, e.g. no python literal blocks that aren't tested

Memory eludes me: what stops us from just running sphinx-build -b doctest in place in the normal docs directory, rather than synthesizing one?

The synthesized one is an attempt to isolate on a per-package basis (just as is done with the system tests). It's not strictly necessary.

The "real" reason I did it was because our docs build takes quite a long time (since the project is so large), and I wanted to have a snappier doctest build.

dhermes · 2016-11-29T22:53:32Z

@tseaver Can we keep moving on this?

tseaver · 2016-11-30T21:37:59Z

@dhermes I'd still say that running sphinx-build -b doctest (maybe in a separate [testenv:doctest] environment w/ appropriate system-tests deps) would be the right way to move forward: it will pick up any snippets in non-autodoc files, for instance.

Sign in to view


- from google.cloud import datastore
+ import os
+ os.environ['GOOGLE_CLOUD_PROJECT'] = u'my-project'


dhermes · 2016-11-30T21:44:47Z

I'd still say that running sphinx-build -b doctest (maybe in a separate [testenv:doctest] environment w/ appropriate system-tests deps) would be the right way to move forward: it will pick up any snippets in non-autodoc files, for instance.

This is fine, though I see it having two issues:

We can't cherry pick which packages we run it for (this is actually really nasty because the docs build is very slow)
We have to set up and teardown system test-like behavior yet again in CI (as opposed to the scenario where doctests run with the system tests)

dhermes · 2016-12-02T22:16:04Z

Plan moving forward:

use sphinx-apidoc for most RST files
leave usage docs in handwritten RST files
make a (potentially "bit-rot"-able) script to pick up usage docs along with other package files when running isolated doctest
write TOC by hand

tseaver · 2016-12-02T22:22:34Z

LGTM.

Adding ability to run doctests with datastore system tests.

dhermes added api: datastore Issues related to the Datastore API. testing labels Nov 15, 2016

dhermes assigned daspecster and tseaver Nov 15, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 15, 2016

dhermes commented Nov 15, 2016
View reviewed changes

Comment threadsystem_tests/datastore.py
def test_it(self):
from sphinx import application

docs_dir = self._make_temp_docs()
This comment was marked as spam.
Sign in to view

dhermes commented Nov 15, 2016
View reviewed changes

Comment threadsystem_tests/datastore.py
return docs_dir

def test_it(self):
from sphinx import application
This comment was marked as spam.
Sign in to view

dhermes added the docs label Nov 15, 2016

dhermes force-pushed the sphinx-doctest-datastore branch from be9f0dc to 5a5b307Compare November 15, 2016 17:06

Adding ability to run doctests with datastore system tests.
4e7d49f

dhermes force-pushed the sphinx-doctest-datastore branch from 5a5b307 to 4e7d49fCompare November 15, 2016 17:31

dhermes mentioned this pull request Nov 18, 2016
Add Flask Helper to Error Reporting #2751
Merged

tseaver reviewed Nov 28, 2016
View reviewed changes

tseaver reviewed Nov 30, 2016
View reviewed changes

Comment threaddatastore/google/cloud/datastore/__init__.py

from google.cloud import datastore
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = u'my-project'
This comment was marked as spam.
Sign in to view
This comment was marked as spam.
Sign in to view

dhermes merged commit e4ce734 into googleapis:masterDec 2, 2016

dhermes deleted the sphinx-doctest-datastore branch December 2, 2016 22:23

dhermes mentioned this pull request Dec 9, 2016
Update versions for mega-release. #2846
Merged

dhermes mentioned this pull request Mar 20, 2017
Test all links and code examples in docs for GA/Beta. #3172
Closed
24 tasks

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017
Merge pull requestgoogleapis#2738from dhermes/sphinx-doctest-datastore
6224c08
Adding ability to run doctests with datastore system tests.

theacodes unassigned daspecster Sep 28, 2018

parthea pushed a commit that referenced this pull request Nov 24, 2025
Merge pull request#2738from dhermes/sphinx-doctest-datastore
9720552
Adding ability to run doctests with datastore system tests.

Uh oh!

Conversation

dhermes commented Nov 15, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Nov 16, 2016

Uh oh!

dhermes commented Nov 18, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Nov 28, 2016

Uh oh!

tseaver commented Nov 28, 2016

Uh oh!

tseaver commented Nov 29, 2016

Uh oh!

dhermes commented Nov 29, 2016

Uh oh!

dhermes commented Nov 29, 2016

Uh oh!

tseaver commented Nov 30, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Nov 30, 2016

Uh oh!

dhermes commented Dec 2, 2016

Uh oh!

tseaver commented Dec 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants