test: reduce testing to 2 python versions per sample

[parthea]

Please review #8872 first

Once #8872 is merged, the versions of python tested will be 2.7, 3.7, 3.8, 3.9, 3.10 and 3.11

To save on resources, let's restrict testing to the end points. So, 3.7 and 3.11 (plus 2.7 when needed).

[leahecole]

removed earlier comment, realized we were editing the noxfile configs per product not the base test image. sorry!

[ace-n]

FYI: App Engine does not fully support 3.11 yet.

[leahecole]

Composer LGTM
data-science-onramp - I am seeing the protocol buffers descriptors error (but don't with Python 3.10 or below) - I am wondering if it's really just that python-aiplatform isn't ready yet?

[leahecole]

I'm going to merge with main to kick off another build but I'm thinking that...

3.7

• retail failure looks like it might be resolved by chore(samples): remove flaky test conditions #8968

3.11

• dataproc should be fixed now that I gave the default service account permissions
• talent/jobs should be resolved from enabling the api and approving the log thing
• healthcare should be resolved since we enabled the api
• secretmanager should be resolved since we enabled the API

Update
3.7 timed out
3.11

• dataproc confirmed resolved
• jobs/talent confirmed resolved
• healthcare resolved many, but I just gave it an IAM role to hopefully resolve the rest
• secretmanager resolved
• I will create a datastore instance to fix the blog one

[dandhlee]

I believe some products only serve 3.8 or 3.9 (Run and Data* products).

Fundamentally, I'd strongly push back against making this change. If users run into problems before we do, simply because we reduced the number of versions being tested, that'd be a really bad experience for everyone.

If we want to save resources, can we run the nightly / continuous less frequently instead?

[kweinmeister]

@dandhlee I believe this change does not affect samples that have a dependency on a specific version (e.g. 3.8 or 3.9). It should only address projects with open-ended 3.x support. If there are spots where the PR is not meeting this criteria, we should definitely update it.

This PR is aligned with a risk-based testing strategy that is followed in other languages (e.g. Java 8, 11, 17 and not 8..17). If the endpoints are both passing, the risk of a deterministic language-specific issue in 3.8..3.10 is very low and acceptable for sample code given tradeoffs IMO. We can incrementally add versions back to the matrix if such issues arise -- e.g. a midpoint version (3.9) next. Also, in addition to this change, as you suggested we can review the frequency of periodic builds.

FYI @nicain

[leahecole]

FYI: App Engine does not fully support 3.11 yet.

@ace-n the doc you linked is for app engine standard, but iiuc these tests are for app engine flexible - does flexible also not support 3.11?

[ace-n]

@leahecole here's my understanding:

GAE Flex

Flexible supports Python 3.4 - 3.7.

GAE Standard

The App Engine standard template (appengine/standard/noxfile-template.py)'s TEST_CONFIG["ignored_versions"]does not match the IGNORED_VERSIONS variable lower in the file, and I'm not sure which one takes priority.

GAE standard_python3

For appengine/standard_python3 - this PR would test 3.7 and 3.11. I assume we should change 3.11 to 3.10? (cc @engelke as FYI)

[leahecole]

Suggested change

	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	# Test min and max supported versions which are python 3.7 and 3.11
	# Test min and max supported versions which are python 3.7 and 3.10

[leahecole]

Suggested change

	# Test min and max supported versions which are python 3.7 and 3.11
	# Test min and max supported versions which are python 3.7 and 3.10

[leahecole]

Suggested change

	# Test min and max supported versions which are python 3.7 and 3.11
	# Test min and max supported versions which are python 3.7 and 3.10

[leahecole]

Suggested change

	# Test min and max supported versions which are python 3.7 and 3.11
	# Test min and max supported versions which are python 3.7 and 3.10

[leahecole]

Suggested change

	# Test min and max supported versions which are python 3.7 and 3.11
	# Ignoring python 2.7 which is no longer supported.
	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	# Test min and max supported versions which are python 3.7 and 3.10
	# Ignoring python 2.7 which is no longer supported.
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	# Test min and max supported versions which are python 3.7 and 3.11
	# Ignoring python 2.7 which is no longer supported.
	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	# Test min and max supported versions which are python 3.7 and 3.10
	# Ignoring python 2.7 which is no longer supported.
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

Suggested change

	# Test min and max supported versions which are python 3.7 and 3.11
	# Ignoring python 2.7 which is no longer supported.
	"ignored_versions": ["2.7", "3.8", "3.9", "3.10"],
	# Test min and max supported versions which are python 3.7 and 3.10
	# Ignoring python 2.7 which is no longer supported.
	"ignored_versions": ["2.7", "3.8", "3.9", "3.11"],

[leahecole]

requesting changes to appengine per @ace-n 's comment

[engelke]

A lot has changed since this PR was proposed, and there are a lot of items for discussion. Could we close this for now and revisit the whole thing following some planned fixes?