Update the default-python template according to Lakeflow conventions

[lennartkats-db]

Changes

This updates the default-python template according to the latest Lakeflow conventions as established in #3671. Notably, the new template moves away from the use of notebooks for pipeline source code.

The new layout looks as follows when the user selects they want both the sample job and the sample pipeline:

📁 resources
├── sample_job.job.yml
└── sample_etl.pipeline.yml
📁 src
├── 📁 my_project — shared source code for use in jobs and/or pipelines
│ ├── __init__.py
│ └── main.py
└── 📁 my_project_etl — source code for the sample_etl pipeline
├── __init__.py
├── 📁 transformations
│ ├── __init__.py
│ ├── sample_zones_my_project.py
│ └── sample_trips_my_project.py
├── 📁 explorations — exploratory notebooks
│ ├── __init__.py
│ └── sample_exploration.ipynb
└── README.md
📁 tests — unit tests
📁 fixtures — fixtures (these can now be used with load_fixture)
databricks.yml
pyproject.toml
README.md

The template prompts have been updated to cater to this structure. Notably, they include a new prompt to manage the catalog and schema used by the template. These settings are propagated to both the job and the pipeline:

Welcome to the default Python template for Databricks Asset Bundles!
Answer the following questions to customize your project.
You can always change your configuration in the databricks.yml file later.
Note that https://e2-dogfood.staging.cloud.databricks.com is used for initialization.
(For information on how to change your profile, see https://docs.databricks.com/dev-tools/cli/profiles.html.)
Unique name for this project [my_project]: my_project
Include a Lakeflow job that runs a notebook: yes
Include an ETL pipeline: yes
Include a sample Python package that builds into a wheel file: yes
Use serverless compute: yes
Default catalog for any tables created by this project [main]: main
Use a personal schema for each user working on this project.
(This is recommended. Your personal schema will be 'main.lennart_kats'.): yes
✨ Your new project has been created in the 'my_project' directory!
To get started, refer to the project README.md file and the documentation at https://docs.databricks.com/dev-tools/bundles/index.html.

Testing

• Standard unit testing, acceptance testing
• AI excercised templates and with all permutations of options, deploying/testing/running/inspecting the result
• Bug bash of the original lakeflow-pipelines template from Prepare a new "generic" template, use "src layout" for Lakeflow template #3671

[eng-dev-ecosystem-bot]

Run: 19064653887

	Env	🔄​flaky	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip
💚​	aws linux		1	1	326	591
💚​	aws windows		1	1	327	590
💚​	aws-ucws linux		1	1	449	486
💚​	aws-ucws windows		1	1	450	485
💚​	azure linux		1	1	326	590
💚​	azure windows		1	1	327	589
💚​	azure-ucws linux		1	1	445	485
💚​	azure-ucws windows		1	1	446	484
🔄​	gcp linux	8	1	1	317	592
🔄​	gcp windows	2	1	1	324	591

12 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
TestAccept/bundle/resources/jobs/fail-on-active-runs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/resources/jobs/fail-on-active-runs/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/run/app-with-job	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.10	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f

[fjakobs]

This looks good to me