Download spark_python_task workspace files behind --download-spark-python-files flag

[janniklasrose]

Changes

Re-applies #5799 (reverted in #5837) and puts the behaviour behind a new opt-in flag.

bundle generate job only downloaded notebook tasks; files referenced by spark_python_task were left as absolute /Workspace/... paths, so the source file was never downloaded and the config wasn't portable. This PR restores the download+rewrite path (reusing the same markFileForDownload helper as pipeline libraries) but gates it behind a new --download-spark-python-files flag on bundle generate job, defaulting to off. Git-sourced files (source: GIT) and cloud URIs (dbfs:/, s3:/, adls:/, gcs:/) are left untouched.

The flag is threaded through the Downloader as a functional option (WithSparkPythonFiles), so the pipeline, app, and import callers are unaffected.

Why

#5799 was reverted in #5837 because a Python file often imports sibling files that the downloader does not capture, so downloading only the entry point can produce a job that fails at runtime with missing imports — whereas leaving the absolute workspace path alone "just works". Making the download opt-in keeps the default safe while letting users who know their spark_python_task is self-contained pull the file into their bundle.

Commits

1. Re-apply Download spark_python_task workspace files in bundle generate job #5799 — a single re-revert of Revert #5799 #5837, restoring the original change verbatim (only the NEXT_CHANGELOG.md conflict resolved).
2. Gate behind --download-spark-python-files — the new flag, functional-option plumbing, tests, and regenerated help/changelog.

Tests

• Unit tests in bundle/generate/downloader_test.go: the download+rewrite path (with the option), the skipped cases (cloud URI, source: GIT), and a new test asserting the default-off behaviour makes no requests.
• Acceptance test acceptance/bundle/generate/spark_python_task_job exercising the full CLI with the flag: a workspace-file task is downloaded and rewritten, a dbfs:/ cloud-URI task is preserved. Identical output on both terraform and direct engines.
• Regenerated the bundle generate job --help golden file.

This pull request and its description were written by Isaac, an AI coding agent.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 06965a8

Run: 29580602178

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	4	4	227	1121	4:52
💚​	aws windows	4	4	229	1119	6:44
💚​	aws-ucws linux	4	4	316	1037	6:15
💚​	aws-ucws windows	4	4	318	1035	7:40
💚​	azure linux	4	4	227	1120	4:55
💚​	azure windows	4	4	229	1118	6:45
💚​	azure-ucws linux	4	4	318	1034	6:46
💚​	azure-ucws windows	4	4	320	1032	7:48
💚​	gcp linux	4	4	226	1122	4:49
💚​	gcp windows	4	4	228	1120	6:40

8 interesting tests: 4 RECOVERED, 4 SKIP

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R

Top 10 slowest tests (at least 2 minutes):

duration	env	testname
6:25	aws-ucws windows	TestAccept
6:23	azure-ucws windows	TestAccept
6:19	azure windows	TestAccept
6:17	aws windows	TestAccept
6:15	gcp windows	TestAccept
3:02	gcp linux	TestAccept
2:56	aws linux	TestAccept
2:56	azure linux	TestAccept
2:51	azure-ucws linux	TestAccept
2:50	aws-ucws linux	TestAccept