Uh oh!
There was an error while loading. Please reload this page.
Add invariant tests - #4326
Merged
Merged
Conversation
denik
requested review from
andrewnester, anton-107, pietern and shreyas-goenka
as code ownersJanuary 19, 2026 16:36
Collaborator
Commit: 73a1501 |
denikforce-pushed
the
denik/invariant-tests
branch
from
January 20, 2026 12:02
63ee262 to
a195594Comparepietern
approved these changes
Jan 21, 2026
| no_drift test checks that there are no actions planned after successful deploy. If that's not the case, the | ||
| test will dump full JSON plan to the output. | ||
| In order to add a new test, add a config to configs/ and include it in test.toml. |
Contributor
There was a problem hiding this comment.
The ones in this PR are specific to bundles. Will they generalize? If not, they can be moved the bundles dir.
| @@ -0,0 +1,39 @@ | |||
| #!/usr/bin/env python3 | |||
| """ | |||
| Check that all actions in plan are "skip". | |||
| foo: | ||
| warehouse_id: $TEST_DEFAULT_WAREHOUSE_ID | ||
| display_name: test-alert-$UNIQUE_NAME | ||
| file_path: ./alert.dbalert.json |
Contributor
There was a problem hiding this comment.
@shreyas-goenka Do we need one with the alert inlined?
| cat LOG.deploy | contains.py '!panic' '!internal error' > /dev/null | ||
| # Special message to fuzzer that generated config was fine. | ||
| # Any failures after this point will be considered as "bug detected" by fuzzer. |
Contributor
There was a problem hiding this comment.
Fuzzer == vary over input configs?
denikforce-pushed
the
denik/invariant-tests
branch
from
January 21, 2026 13:25
39b9307 to
3de7df5CompareAdd 9 minimal resource configuration templates to test the no-drift invariant across different resource types (jobs, pipelines, experiments, models, volumes, schemas, clusters, model serving endpoints). Changes: - Add configs/ directory with minimal .yml.tmpl files for each resource type - Add data/ directory for supporting files (pipeline.py) - Update no_drift/script to copy data files at test start - Update test.toml to include new configs and ignore *.py files - Update out.test.toml with expanded config matrix All configs tested and verified on both testserver and AWS cloud with direct and terraform deployment engines. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add 4 more resource configurations to the no_drift invariant test, bringing the total to 13 working configs. New configs: - alerts (with .dbalert.json file) - dashboards (with .lvdash.json file) - apps (with source_code_path directory) - secret_scopes Also created sql_warehouse.yml.tmpl but excluded from test due to drift issues. Changes: - Add 5 new config templates in configs/ - Add 3 new data files in data/ (alert.dbalert.json, dashboard.lvdash.json, app/) - Update test.toml to include new configs and ignore *.json and app directory - Update out.test.toml with expanded config matrix All 13 configs tested on testserver with both direct and terraform engines. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Complete the no_drift invariant test coverage by adding the remaining resource types, bringing the total to 17 working configs. New configs: - quality_monitor (with dependent schema) - database_instance - database_catalog (with dependent database_instance) - synced_database_table (with dependent instance and catalog) Changes: - Add 3 new config templates in configs/ - Fix quality_monitor to use $CURRENT_USER_NAME - Update test.toml with 4 additional configs - Update out.test.toml with full config matrix All 17 configs tested successfully on testserver with both direct and terraform deployment engines, showing zero drift after deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
App names must be between 2-30 characters. Changed from 'test-app-' to 'app-' to fit within the limit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
denikforce-pushed
the
denik/invariant-tests
branch
from
January 22, 2026 11:26
3de7df5 to
10b8a2fComparedenik
enabled auto-merge
January 22, 2026 11:51
denik
disabled auto-merge
January 22, 2026 14:56
Uh oh!
There was an error while loading. Please reload this page.
eng-dev-ecosystem-bot
commented
Jan 22, 2026
Collaborator
Commit: 93b59f2
33 interesting tests: 12 flaky, 8 RECOVERED, 5 SKIP, 5 FAIL, 3 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
denik added a commit
that referenced
this pull request
May 20, 2026
## Changes Add a new kind of test - invariant test for DABs (a subtype of acceptance test). Invariant tests are acceptance tests that can be run against many configs to check for certain properties. Unlike regular acceptance tests full output is not recorded, unless the condition is not met. In this PR there is one invariant test added - no_drift, that checks that there are no actions planned after successful deploy. If that's not the case, the test will dump full JSON plan to the output. ## Why Simplify resource implementation testing - implementors only need to provide a config to get all invariants check. With no_drift in particular, we've had several cases where this condition is not met and it's still the case for some configurations of permissions and grants. It's also going to become more important as we replace "server_side_default" with something more precise. The other use of invariant tests is fuzzing - since they work on any config, they can be used on randomly generated configs to find bugs. --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add a new kind of test - invariant test for DABs (a subtype of acceptance test).
Invariant tests are acceptance tests that can be run against many configs to check for certain properties.
Unlike regular acceptance tests full output is not recorded, unless the condition is not met.
In this PR there is one invariant test added - no_drift, that checks that there are no actions planned after successful deploy. If that's not the case, the test will dump full JSON plan to the output.
Why
Simplify resource implementation testing - implementors only need to provide a config to get all invariants check.
With no_drift in particular, we've had several cases where this condition is not met and it's still the case for some configurations of permissions and grants. It's also going to become more important as we replace "server_side_default" with something more precise.
The other use of invariant tests is fuzzing - since they work on any config, they can be used on randomly generated configs to find bugs.