feat: [WIP] The first stage of `nox` implementation by mf2199 · Pull Request #468 · googleapis/python-spanner-django

mf2199 · 2020-08-25T21:50:53Z

As suggested in #466 , this represents the first item of the recommended list aimed to implement nox testing automation in multiple steps.

Change list:

Added new noxfile.py with the same targets as the existing tox.ini [the commented areas reflect future targets];
Updated .gitignore.

From the results of the initial testing, the code has numerous assertion errors, see the test log below, which appear to be "normal" errors related to failing tests rather the test setup as such. Similar errors observed when using the tox framework.

Towards #474

mf2199 · 2020-08-25T21:58:58Z

Below is the initial test log. Most deprecation warnings were omitted for better readability, leaving a few as an example.

(python-spanner-django) C:\git\python-spanner-django>nox
nox > Running session unit-2.7
nox > Session unit-2.7 skipped: Python interpreter 2.7 not found.
nox > Running session unit-3.5
nox > Session unit-3.5 skipped: Python interpreter 3.5 not found.
nox > Running session unit-3.6
nox > Creating virtual environment (virtualenv) using python.exe in .nox\unit-3-6
nox > pip install mock pytest pytest-cov
nox > pip install -e .
nox > py.test tests\spanner_dbapi
========================================================================================================= test session starts =========================================================================================================
platform win32 -- Python 3.6.8, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: C:\git\python-spanner-django
plugins: cov-2.10.1
collected 31 items tests\spanner_dbapi\test_globals.py . [ 3%]
tests\spanner_dbapi\test_parse_utils.py ........... [ 38%]
tests\spanner_dbapi\test_parser.py ........ [ 64%]
tests\spanner_dbapi\test_types.py ....F.F [ 87%]
tests\spanner_dbapi\test_utils.py .. [ 93%]
tests\spanner_dbapi\test_version.py .. [100%]
============================================================================================================== FAILURES ===============================================================================================================
____________________________________________________________________________________________________ TypesTests.test_TimeFromTicks ____________________________________________________________________________________________________
self = <tests.spanner_dbapi.test_types.TypesTests testMethod=test_TimeFromTicks>
def test_TimeFromTicks(self):
epochTicks = 1572851662.9782631 # Sun Nov 03 23:14:22 2019
got = TimeFromTicks(epochTicks)
# Since continuous integration infrastructure such as Travis CI
# uses clocks on UTC, it is useful to be able to compare against
# either of UTC or the known standard time.
want = (
datetime.time(23, 14, 22),
datetime.datetime(2019, 11, 4, 7, 14, 22, tzUTC).time(),
)
matches = got in want
> self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
E AssertionError: False is not true : `02:14:22` not present in any of
E `(datetime.time(23, 14, 22), datetime.time(7, 14, 22))`
tests\spanner_dbapi\test_types.py:58: AssertionError
_________________________________________________________________________________________________ TypesTests.test_TimestampFromTicks __________________________________________________________________________________________________
self = <tests.spanner_dbapi.test_types.TypesTests testMethod=test_TimestampFromTicks>
def test_TimestampFromTicks(self):
epochTicks = 1572851662.9782631 # Sun Nov 03 23:14:22 2019
got = TimestampFromTicks(epochTicks)
# Since continuous integration infrastructure such as Travis CI
# uses clocks on UTC, it is useful to be able to compare against
# either of UTC or the known standard time.
want = (
datetime.datetime(2019, 11, 3, 23, 14, 22),
datetime.datetime(2019, 11, 4, 7, 14, 22, tzUTC),
)
matches = got in want
> self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
E AssertionError: False is not true : `2019-11-04 02:14:22` not present in any of
E `(datetime.datetime(2019, 11, 3, 23, 14, 22), datetime.datetime(2019, 11, 4, 7, 14, 22))`
tests\spanner_dbapi\test_types.py:71: AssertionError
======================================================================================================= short test summary info =======================================================================================================
FAILED tests/spanner_dbapi/test_types.py::TypesTests::test_TimeFromTicks - AssertionError: False is not true : `02:14:22` not present in any of
FAILED tests/spanner_dbapi/test_types.py::TypesTests::test_TimestampFromTicks - AssertionError: False is not true : `2019-11-04 02:14:22` not present in any of
==================================================================================================== 2 failed, 29 passed in 0.44s =====================================================================================================
nox > Command py.test tests\spanner_dbapi failed with exit code 1
nox > Session unit-3.6 failed.
nox > Running session unit-3.7
nox > Creating virtual environment (virtualenv) using python.exe in .nox\unit-3-7
nox > pip install mock pytest pytest-cov
nox > pip install -e .
nox > py.test tests\spanner_dbapi
========================================================================================================= test session starts =========================================================================================================
platform win32 -- Python 3.7.8, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: C:\git\python-spanner-django
plugins: cov-2.10.1
collected 31 items tests\spanner_dbapi\test_globals.py . [ 3%]
tests\spanner_dbapi\test_parse_utils.py ........... [ 38%]
tests\spanner_dbapi\test_parser.py ........ [ 64%]
tests\spanner_dbapi\test_types.py ....F.F [ 87%]
tests\spanner_dbapi\test_utils.py .. [ 93%]
tests\spanner_dbapi\test_version.py .. [100%]
============================================================================================================== FAILURES ===============================================================================================================
____________________________________________________________________________________________________ TypesTests.test_TimeFromTicks ____________________________________________________________________________________________________
self = <tests.spanner_dbapi.test_types.TypesTests testMethod=test_TimeFromTicks>
def test_TimeFromTicks(self):
epochTicks = 1572851662.9782631 # Sun Nov 03 23:14:22 2019
got = TimeFromTicks(epochTicks)
# Since continuous integration infrastructure such as Travis CI
# uses clocks on UTC, it is useful to be able to compare against
# either of UTC or the known standard time.
want = (
datetime.time(23, 14, 22),
datetime.datetime(2019, 11, 4, 7, 14, 22, tzUTC).time(),
)
matches = got in want
> self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
E AssertionError: False is not true : `02:14:22` not present in any of
E `(datetime.time(23, 14, 22), datetime.time(7, 14, 22))`
tests\spanner_dbapi\test_types.py:58: AssertionError
_________________________________________________________________________________________________ TypesTests.test_TimestampFromTicks __________________________________________________________________________________________________
self = <tests.spanner_dbapi.test_types.TypesTests testMethod=test_TimestampFromTicks>
def test_TimestampFromTicks(self):
epochTicks = 1572851662.9782631 # Sun Nov 03 23:14:22 2019
got = TimestampFromTicks(epochTicks)
# Since continuous integration infrastructure such as Travis CI
# uses clocks on UTC, it is useful to be able to compare against
# either of UTC or the known standard time.
want = (
datetime.datetime(2019, 11, 3, 23, 14, 22),
datetime.datetime(2019, 11, 4, 7, 14, 22, tzUTC),
)
matches = got in want
> self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
E AssertionError: False is not true : `2019-11-04 02:14:22` not present in any of
E `(datetime.datetime(2019, 11, 3, 23, 14, 22), datetime.datetime(2019, 11, 4, 7, 14, 22))`
tests\spanner_dbapi\test_types.py:71: AssertionError
======================================================================================================= short test summary info =======================================================================================================
FAILED tests/spanner_dbapi/test_types.py::TypesTests::test_TimeFromTicks - AssertionError: False is not true : `02:14:22` not present in any of
FAILED tests/spanner_dbapi/test_types.py::TypesTests::test_TimestampFromTicks - AssertionError: False is not true : `2019-11-04 02:14:22` not present in any of
==================================================================================================== 2 failed, 29 passed in 0.40s =====================================================================================================
nox > Command py.test tests\spanner_dbapi failed with exit code 1
nox > Session unit-3.7 failed.
nox > Running session unit-3.8
nox > Creating virtual environment (virtualenv) using python.exe in .nox\unit-3-8
nox > pip install mock pytest pytest-cov
nox > pip install -e .
nox > py.test tests\spanner_dbapi
========================================================================================================= test session starts =========================================================================================================
platform win32 -- Python 3.8.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: C:\git\python-spanner-django
plugins: cov-2.10.1
collected 31 items tests\spanner_dbapi\test_globals.py . [ 3%]
tests\spanner_dbapi\test_parse_utils.py ........... [ 38%]
tests\spanner_dbapi\test_parser.py ........ [ 64%]
tests\spanner_dbapi\test_types.py ....F.F [ 87%]
tests\spanner_dbapi\test_utils.py .. [ 93%]
tests\spanner_dbapi\test_version.py .. [100%]
============================================================================================================== FAILURES ===============================================================================================================
____________________________________________________________________________________________________ TypesTests.test_TimeFromTicks ____________________________________________________________________________________________________
self = <tests.spanner_dbapi.test_types.TypesTests testMethod=test_TimeFromTicks>
def test_TimeFromTicks(self):
epochTicks = 1572851662.9782631 # Sun Nov 03 23:14:22 2019
got = TimeFromTicks(epochTicks)
# Since continuous integration infrastructure such as Travis CI
# uses clocks on UTC, it is useful to be able to compare against
# either of UTC or the known standard time.
want = (
datetime.time(23, 14, 22),
datetime.datetime(2019, 11, 4, 7, 14, 22, tzUTC).time(),
)
matches = got in want
> self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
E AssertionError: False is not true : `02:14:22` not present in any of
E `(datetime.time(23, 14, 22), datetime.time(7, 14, 22))`
tests\spanner_dbapi\test_types.py:58: AssertionError
_________________________________________________________________________________________________ TypesTests.test_TimestampFromTicks __________________________________________________________________________________________________
self = <tests.spanner_dbapi.test_types.TypesTests testMethod=test_TimestampFromTicks>
def test_TimestampFromTicks(self):
epochTicks = 1572851662.9782631 # Sun Nov 03 23:14:22 2019
got = TimestampFromTicks(epochTicks)
# Since continuous integration infrastructure such as Travis CI
# uses clocks on UTC, it is useful to be able to compare against
# either of UTC or the known standard time.
want = (
datetime.datetime(2019, 11, 3, 23, 14, 22),
datetime.datetime(2019, 11, 4, 7, 14, 22, tzUTC),
)
matches = got in want
> self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
E AssertionError: False is not true : `2019-11-04 02:14:22` not present in any of
E `(datetime.datetime(2019, 11, 3, 23, 14, 22), datetime.datetime(2019, 11, 4, 7, 14, 22))`
tests\spanner_dbapi\test_types.py:71: AssertionError
========================================================================================================== warnings summary ===========================================================================================================
.nox\unit-3-8\lib\site-packages\google\api\http_pb2.py:30
c:\git\python-spanner-django\.nox\unit-3-8\lib\site-packages\google\api\http_pb2.py:30: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get
/find descriptors from generated code or query the descriptor_pool.
DESCRIPTOR = _descriptor.FileDescriptor(
.nox\unit-3-8\lib\site-packages\google\api\http_pb2.py:46
c:\git\python-spanner-django\.nox\unit-3-8\lib\site-packages\google\api\http_pb2.py:46: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use ge
t/find descriptors from generated code or query the descriptor_pool.
_descriptor.FieldDescriptor(
[ ... 97 similar deprecation warnings ... ]
.nox\unit-3-8\lib\site-packages\google\iam\v1\policy_pb2.py:150
c:\git\python-spanner-django\.nox\unit-3-8\lib\site-packages\google\iam\v1\policy_pb2.py:150: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please
use get/find descriptors from generated code or query the descriptor_pool.
_descriptor.FieldDescriptor(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================================================================================= short test summary info =======================================================================================================
FAILED tests/spanner_dbapi/test_types.py::TypesTests::test_TimeFromTicks - AssertionError: False is not true : `02:14:22` not present in any of
FAILED tests/spanner_dbapi/test_types.py::TypesTests::test_TimestampFromTicks - AssertionError: False is not true : `2019-11-04 02:14:22` not present in any of
============================================================================================= 2 failed, 29 passed, 100 warnings in 0.72s ==============================================================================================
nox > Command py.test tests\spanner_dbapi failed with exit code 1
nox > Session unit-3.8 failed.
nox > Ran multiple sessions:
nox > * unit-2.7: skipped
nox > * unit-3.5: skipped
nox > * unit-3.6: failed
nox > * unit-3.7: failed
nox > * unit-3.8: failed
(python-spanner-django) C:\git\python-spanner-django>

c24t

Thanks for the quick response on #466.

Before merging this nox should run without errors. We can also lose the commented lines.

mf2199 · 2020-08-25T22:34:52Z

@c24t For the reference, here's the tox log:

GLOB sdist-make: C:\git\python-spanner-django\setup.py
py37-django22 inst-nodeps: C:\git\python-spanner-django\.tox\.tmp\package\1\django-google-spanner-2.2a0.zip
py37-django22 installed: cachetools==4.1.1,certifi==2020.6.20,chardet==3.0.4,Django==2.2.15,django-google-spanner @ file:///C:/git/python-spanner-django/.tox/.tmp/package/1/django-google-spanner-2.2a0.zip,google-api-core==1.22.1,goo
gle-auth==1.20.1,google-cloud-core==1.4.1,google-cloud-spanner==1.17.1,googleapis-common-protos==1.52.0,grpc-google-iam-v1==0.12.3,grpcio==1.31.0,grpcio-gcp==0.2.2,idna==2.10,protobuf==3.13.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pytz
==2020.1,requests==2.24.0,rsa==4.6,six==1.15.0,sqlparse==0.3.1,urllib3==1.25.10
py37-django22 run-test-pre: PYTHONHASHSEED='984'
py37-django22 run-test: commands[0] | python runtests.py
........................F.F....
======================================================================
FAIL: test_TimeFromTicks (tests.spanner_dbapi.test_types.TypesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\git\python-spanner-django\tests\spanner_dbapi\test_types.py", line 58, in test_TimeFromTicks
self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
AssertionError: False is not true : `02:14:22` not present in any of
`(datetime.time(23, 14, 22), datetime.time(7, 14, 22))`
======================================================================
FAIL: test_TimestampFromTicks (tests.spanner_dbapi.test_types.TypesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\git\python-spanner-django\tests\spanner_dbapi\test_types.py", line 71, in test_TimestampFromTicks
self.assertTrue(matches, '`%s` not present in any of\n`%s`' % (got, want))
AssertionError: False is not true : `2019-11-04 02:14:22` not present in any of
`(datetime.datetime(2019, 11, 3, 23, 14, 22), datetime.datetime(2019, 11, 4, 7, 14, 22))`
----------------------------------------------------------------------
Ran 31 tests in 0.015s
FAILED (failures=2)
ERROR: InvocationError for command 'C:\git\python-spanner-django\.tox\py37-django22\Scripts\python.EXE' runtests.py (exited with code 1)
flake8 installed: flake8==3.8.3,importlib-metadata==1.7.0,isort==5.4.2,mccabe==0.6.1,pycodestyle==2.6.0,pyflakes==2.2.0,zipp==3.1.0
flake8 run-test-pre: PYTHONHASHSEED='984'
flake8 run-test: commands[0] | flake8
flake8 run-test: commands[1] | isort --recursive --check-only --diff
_ _
(_) ___ ___ _ __| |_
| |/ _/ / _ \/ '__ _/
| |\__ \/\_\/| | | |_
|_|\___/\___/\_/ \_/
isort your imports, so you don't have to.
VERSION 5.4.2
Nothing to do: no files or paths have have been passed in!
Try one of the following:
`isort .` - sort all Python files, starting from the current directory, recursively.
`isort . --interactive` - Do the same, but ask before making any changes.
`isort . --check --diff` - Check to see if imports are correctly sorted within this project.
`isort --help` - In-depth information about isort's available command-line options.
Visit https://timothycrosley.github.io/isort/ for complete information about how to use isort.
Error: arguments passed in without any paths or content.
ERROR: InvocationError for command 'C:\git\python-spanner-django\.tox\flake8\Scripts\isort.EXE' --recursive --check-only --diff (exited with code 1)
_______________________________________________________________________________________________________________ summary _______________________________________________________________________________________________________________
ERROR: py37-django22: commands failed
ERROR: flake8: commands failed

As you might see, the assertion errors are the same as with nox, meaning they are not attributed to this PR.

c24t · 2020-08-25T22:39:35Z

As you can see, the assertion errors are the same as with nox, meaning they are not attributed to this PR.

SGTM as long as we've got the same failing tests in each version. Looks like 2.7 and 3.5 aren't installed on your machine? Hopefully those have the same failures.

c24t · 2020-08-25T22:42:09Z

If the goal is to replace tox with nox, can we remove tox.ini in this PR too? What about things like runtests.py that are only called from tox now?

… by `noxfile.py`

mf2199 · 2020-08-25T22:49:17Z

If the goal is to replace tox with nox, can we remove tox.ini in this PR too? What about things like runtests.py that are only called from tox now?

Sure thing, both tox.ini and runtests.py can be removed if we decide to use nox only from now on. Just thought of a least destructive way of introducing the changes, i.e. adding the new stuff first, making sure it works, then removing the deprecated items.

Done!

c24t

LGTM to go ahead, next step is to get these (previously failing) tests working under nox and add support for multiple django versions.

mf2199 · 2020-08-25T23:05:41Z

LGTM to go ahead, next step is to get these (previously failing) tests working under nox and add support for multiple django versions.

Sounds like plan, thanks! Shall it be a separate PR or an amendment to this one?

c24t · 2020-08-25T23:08:34Z

Sounds like plan, thanks! Shall it be a separate PR or an amendment to this one?

A separate PR SGTM.

One more wrinkle though, tox is still used in CI:

python-spanner-django/.kokoro/build.sh

Lines 32 to 38 in 11222db

# Install tox
 python3.6 -m pip install --upgrade --quiet tox flake8 isort 
 python3.6 -m tox --version 
 python3.6 -m tox 
 python3.6 -m isort --recursive --check-only --diff 
 python3.6 -m flake8 

. To merge this PR we need to replace all uses of tox with nox.

mf2199 · 2020-08-25T23:30:04Z

One more wrinkle though, tox is still used in CI:
python-spanner-django/.kokoro/build.sh
Lines 32 to 38 in 11222db
# Install tox
python3.6 -m pip install --upgrade --quiet tox flake8 isort
python3.6 -m tox --version
python3.6 -m tox
python3.6 -m isort --recursive --check-only --diff
python3.6 -m flake8
. To merge this PR we need to replace all uses of tox with nox.

👍 Good catch! Done.

Just for kicks, installed Py27, Py35, and re-ran the tests - no additional errors.

c24t · 2020-08-25T23:48:52Z

I see 43b186e removes flake8 too. I think that's fine for now, but something we need to add again later.

mf2199 · 2020-08-25T23:51:16Z

I see 43b186e removes flake8 too. I think that's fine for now, but something we need to add again later.

Totally, as we add lint to the session list.

mf2199 · 2020-08-25T23:53:55Z

I see 43b186e removes flake8 too. I think that's fine for now, but something we need to add again later.

Actually, this may not even be necessary, as flake8 installation is contained in other APIs noxfiles, meaning same thing can be done here as well.

feat: 1st stage of nox implementation
b7707c5

google-claBot added the cla: yes This human has signed the Contributor License Agreement. label Aug 25, 2020

mf2199 requested a review from c24t August 25, 2020 21:59

mf2199 added api: spanner Issues related to the googleapis/python-spanner-django API. type: process A process-related concern. May include testing, release, or the like. labels Aug 25, 2020

c24t reviewed Aug 25, 2020
View reviewed changes

Comment threadnoxfile.py Outdated
Comment threadnoxfile.py Outdated
Comment threadnoxfile.py Outdated
Comment threadnoxfile.py Outdated

cleanup: Commented lines removed
1857f0f

cleanup: Removed deprecated tox.ini and runtests.py as superseded…
397283c
… by `noxfile.py`

c24t approved these changes Aug 25, 2020
View reviewed changes

fix: updated .kokoro/build.sh for nox usage
43b186e

cleanup: removed "2.7" from nox session targets.
c59213c

c24t merged commit 5551b58 into googleapis:masterAug 26, 2020

c24t mentioned this pull request Aug 28, 2020
Add nox target for django tests #474
Closed

c24t pushed a commit that referenced this pull request Sep 15, 2020
feat: [WIP] The first stage ofnoximplementation (#468)
96f2223

IlyaFaer deleted the nox-1 branch February 1, 2021 08:10

This was referenced Apr 17, 2024
chore(main): release 3.0.0 #869
Closed
chore(main): release 3.0.0 #875
Closed
chore(main): release 3.0.0 #877
Closed
chore(main): release 3.1.0 #878
Closed

release-pleaseBot mentioned this pull request Apr 23, 2024
chore(main): release 3.1.0 #881
Merged

Uh oh!

Conversation

mf2199 commented Aug 25, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mf2199 commented Aug 25, 2020

Uh oh!

c24t left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mf2199 commented Aug 25, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

c24t commented Aug 25, 2020

Uh oh!

c24t commented Aug 25, 2020

Uh oh!

mf2199 commented Aug 25, 2020

Uh oh!

c24t left a comment

Choose a reason for hiding this comment

Uh oh!

mf2199 commented Aug 25, 2020

Uh oh!

c24t commented Aug 25, 2020

Uh oh!

mf2199 commented Aug 25, 2020

Uh oh!

c24t commented Aug 25, 2020

Uh oh!

mf2199 commented Aug 25, 2020

Uh oh!

mf2199 commented Aug 25, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mf2199 commented Aug 25, 2020•
edited
Loading

mf2199 commented Aug 25, 2020•
edited
Loading

mf2199 commented Aug 25, 2020•
edited
Loading