Uh oh!
There was an error while loading. Please reload this page.
[EXPERIMENT -- not for review] - #263
Draft
reuben olinsky (reubeno) wants to merge 1 commit into
Draft
Conversation
… refs Rework the test-suites configuration into a richer test model while keeping the on-disk `test-suites` / `testSuites` wire names unchanged, so existing config files and `config dump` consumers are not broken (the config-key rename is intentionally deferred to a separate change). - Test groups: named bundles of tests referenceable from images/components. - Per-component test references, in addition to per-image references. - A closed `kind` enum (functional/performance) and a `long-running` hint. - Project-level validation that image/component/group references resolve, plus tightened JSON schema constraints (exactly-one ref shape; per-type required subtable) for editor-time feedback. - `azldev image test` drives the new model. - Reference docs and regenerated JSON schema / CLI docs / snapshots. In-code identifiers use the new "test" naming; only the TOML key (`test-suites`) and JSON dump key (`testSuites`) retain the legacy names to avoid a breaking change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: bhagyapathak <bhagyapathak@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the project’s legacy test-suites configuration into a richer test model (tests + test-groups + structured refs) while intentionally keeping the on-disk wire keys (test-suites / testSuites) stable to avoid breaking existing configs and config dump consumers.
Changes:
- Introduces
TestConfig,TestGroupConfig, andTestRef(name vs group) plus new metadata (kind,long-running) and updates the JSON schema accordingly. - Adds project-level validation for image/component/group references and group membership resolution.
- Updates
azldev image testto run the new model (including expanding test-groups) and renames the selection flag to--test.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| schemas/azldev.schema.json | Updates schema to the new test model (tests, test-groups, structured refs, constraints). |
| scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap | Regenerated snapshot reflecting updated JSON schema output. |
| scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap | Regenerated snapshot reflecting updated JSON schema output. |
| internal/projectconfig/testsuite.go | Removes legacy test-suite config implementation. |
| internal/projectconfig/testgroup.go | Adds TestGroupConfig type with validation/merge helpers. |
| internal/projectconfig/test.go | Adds TestConfig, TestKind, TestRef, validation, and schema extensions. |
| internal/projectconfig/test_test.go | Updates/extends unit tests for new test model behavior and validation. |
| internal/projectconfig/project.go | Updates ProjectConfig to Tests/TestGroups and adds reference/membership validation. |
| internal/projectconfig/loader.go | Updates loader merge logic to load tests and test-groups. |
| internal/projectconfig/loader_test.go | Updates loader tests for renamed types/fields and new errors. |
| internal/projectconfig/image.go | Updates image test associations to []TestRef and adds accessors for names/groups. |
| internal/projectconfig/fingerprint_test.go | Registers ComponentConfig.Tests as excluded from fingerprints. |
| internal/projectconfig/configfile.go | Validates tests/test-groups and validates test ref shapes at file load time. |
| internal/projectconfig/configfile_test.go | Adds validation tests for mismatched subtables and invalid ref shapes. |
| internal/projectconfig/component.go | Adds component-level test associations and ensures WithAbsolutePaths preserves them. |
| internal/app/azldev/cmds/image/test.go | Reworks image test command to expand refs/groups and run tests; renames flag to --test. |
| internal/app/azldev/cmds/image/test_test.go | Updates command flag wiring test for --test. |
| internal/app/azldev/cmds/image/pytestrunner.go | Renames runner entrypoint to RunPytest and switches to TestConfig. |
| internal/app/azldev/cmds/image/pytestrunner_test.go | Updates runner test for renamed entrypoint/types. |
| internal/app/azldev/cmds/image/list.go | Adjusts image list output summary formatting for structured test refs/groups. |
| internal/app/azldev/cmds/image/list_test.go | Updates list command tests for TestRef-based configuration. |
| go.mod | Promotes ordered-map dependency to direct (used by JSON schema extensions). |
| docs/user/reference/config/tests.md | Adds new documentation for tests/test-groups and references. |
| docs/user/reference/config/test-suites.md | Removes legacy “test suites” documentation page. |
| docs/user/reference/config/images.md | Updates image test reference docs to new TestRef model and links. |
| docs/user/reference/config/config-file.md | Updates top-level config structure docs to include test-groups and new meaning of test-suites. |
| docs/user/reference/config/components.md | Documents component-level test references. |
| docs/user/reference/cli/azldev_image_test.md | Regenerated CLI docs for updated image test command/flags. |
Comment on lines
+55
to
+58
| Tests are defined in the [tests] section of azldev.toml and referenced by images | ||
| via the [images.NAME.tests] subtable, either directly by name or through a | ||
| test-group. Each test specifies a type and framework-specific configuration in | ||
| a matching subtable. |
Comment on lines
+306
to
+309
| errs = append(errs, fmt.Errorf( | ||
| "%w: image %#q references test %#q, which is not defined in [tests]", | ||
| ErrUndefinedTest, imageName, testName, | ||
| )) |
Comment on lines
+346
to
+349
| errs = append(errs, fmt.Errorf( | ||
| "%w: component %#q references test %#q, which is not defined in [tests]", | ||
| ErrUndefinedTest, componentName, testName, | ||
| )) |
Comment on lines
+386
to
+389
| errs = append(errs, fmt.Errorf( | ||
| "%w: test-group %#q references test %#q, which is not defined in [tests]", | ||
| ErrUndefinedTest, groupName, member, | ||
| )) |
Comment on lines
+114
to
+116
| // Tests is the list of test references that apply to this image. Each entry | ||
| // must reference either a single test (by name, key in [tests]) or a | ||
| // test-group (by group, key in [test-groups]). |
Comment on lines
+316
to
+318
| // Tests is the list of test references that apply to this component. Each entry | ||
| // must reference either a single test (by name, key in [tests]) or a | ||
| // test-group (by group, key in [test-groups]). |
Comment on lines
+300
to
+304
| // Name references a single [TestConfig] by key. | ||
| Name string `toml:"name,omitempty" json:"name,omitempty" jsonschema:"title=Name,description=Name of a test (key in [tests]); mutually exclusive with 'group'"` | ||
| // Group references a [TestGroupConfig] by key. | ||
| Group string `toml:"group,omitempty" json:"group,omitempty" jsonschema:"title=Group,description=Name of a test-group (key in [test-groups]); mutually exclusive with 'name'"` |
Comment on lines
+34
to
+36
| // Tests lists the names of [TestConfig]s that belong to this group. | ||
| Tests []string `toml:"tests,omitempty" json:"tests,omitempty" jsonschema:"title=Tests,description=Names of tests (keys in [tests]) that belong to this group"` | ||
Contributor
Hey there and thank you for opening this pull request! 👋🏼 We require pull request titles to follow the Conventional Commits specification and it looks like your proposed title needs to be adjusted. Details: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rework the test-suites configuration into a richer test model while keeping the on-disk
test-suites/testSuiteswire names unchanged, so existing config files andconfig dumpconsumers are not broken (the config-key rename is intentionally deferred to a separate change).kindenum (functional/performance) and along-runninghint.azldev image testdrives the new model.In-code identifiers use the new "test" naming; only the TOML key (
test-suites) and JSON dump key (testSuites) retain the legacy names to avoid a breaking change.