test: ensure prerelease versions of pandas and arrow are tested nightly

[tswast]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes#882 🦕

[tswast]

Local output, verifying expected package versions are actually used now:

nox > python -m pip install --no-deps -e .[all]
nox > python -c import grpc; print(grpc.__version__)
1.40.0
nox > python -c import pandas; print(pandas.__version__)
1.4.0.dev0+639.g00e10a5c48
nox > python -c import pyarrow; print(pyarrow.__version__)
6.0.0.dev226

[tswast]

Internal change 396006532 pending review to actually enable this continuous job.

[plamut]

Looks good, just had a comment about possibly improving/simplifying pattern matching.

[plamut]

What if we somehow end up with a comment line that starts with whitespace? Unlikely, indeed, but it would feel clumsy if we accidentally break the pipeline with redundant whitespace in a text file. 😆

BTW, should we instead match the entire lines containing version pins and directly extract the dependency name with grouping in the regex pattern itself? Feels more straightforward, to me at least.

Edit: Something like this maybe (untested):

\s*(\S+)(?===\S+) # .group(1) is the dependency name

[tswast]

I like it! I made one modification, which was to add ^ to only match from the start of a line. Otherwise, it picked up "foo" from the comments.

[plamut]

Oh, BTW, we might not want to hardcode the 3.6 Python version, but instead use UNIT_TEST_PYTHON_VERSIONS[0], i.e. the oldest one supported?

[tswast]

It's a bit odd to use UNIT_TEST_PYTHON_VERSIONS in something that's a system test, but I agree it's worth having. I added a comment explaining why it's that version.

[tswast]

Expected pre-release dependencies are being used:

nox > python -c import grpc; print(grpc.__version__)
1.40.0
nox > python -c import pandas; print(pandas.__version__)
1.4.0.dev0+678.ga54aa399dd
nox > python -c import pyarrow; print(pyarrow.__version__)
6.0.0.dev233

https://source.cloud.google.com/results/invocations/c54bbd58-2026-421d-a174-153f13f8f86b/targets/cloud-devrel%2Fclient-libraries%2Fpython%2Fgoogleapis%2Fpython-bigquery%2Fpresubmit%2Fprerelease-deps-3.8/log

[plamut]

@tswast Do we first need to wait for the internal change to be applied, or can we already go ahead and merge this?

[tswast]

We need to merge this first, actually.