Skip to content

direct: parse state via json.RawMessage into appropriate type - #4138

Merged
denik merged 4 commits into
mainfrom
denik/big-int
Dec 12, 2025
Merged

direct: parse state via json.RawMessage into appropriate type#4138
denik merged 4 commits into
mainfrom
denik/big-int

Conversation

@denik

Copy link
Copy Markdown
Contributor

Changes

Instead of parsing into any and then converting into the target type, we now store json.RawMessage and then parse into target type when needed.

Why

One issue it fixes is that JSON parser by default parses integers as float64, losing precision. This was found by @varundeepsaini and a fix is proposed in #4136

This PR is more general though, fully relying on the type to parse itself and it should more efficient as well as we're not walking the type twice & we're not having 2 copies of it.

Tests

New test where int64 field is initialized with large numbers. It fails on main.

@denikdenik changed the title direct: parse state directly into appropriate typedirect: parse state via json.RawMessage into appropriate typeDec 12, 2025
@denik
denik enabled auto-merge December 12, 2025 14:46
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Dec 12, 2025

Copy link
Copy Markdown
Collaborator

Commit: 9312b3f

Run: 20170175513

Env🟨​KNOWN💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux7138064220:54
🟨​aws windows7138264018:56
💚​aws-ucws linux7152352722:01
💚​aws-ucws windows7152552520:34
💚​azure linux1338164017:11
💚​azure windows1338363817:52
💚​azure-ucws linux1352052524:27
💚​azure-ucws windows1352252321:43
💚​gcp linux1337064616:52
💚​gcp windows1337264414:42
8 interesting tests: 7 KNOWN, 1 SKIP
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
Top 24 slowest tests (at least 2 minutes):
durationenvtestname
6:50aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:50aws-ucws linuxTestAccept/bundle/resources/synced_database_tables/basic
6:44aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:43aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:36aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:36aws-ucws windowsTestAccept/bundle/resources/synced_database_tables/basic
6:15aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:06aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:05aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:50gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:46gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:27gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:15gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:45azure windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:31azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:25azure windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:24azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:22azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:15azure linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:49azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:40azure-ucws windowsTestAccept/bundle/resources/synced_database_tables/basic
3:36azure linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:30aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:06azure-ucws linuxTestAccept/bundle/resources/synced_database_tables/basic

@denik
denik added this pull request to the merge queueDec 12, 2025
Merged via the queue into main with commit f915956Dec 12, 2025
19 checks passed
@denik
denik deleted the denik/big-int branch December 12, 2025 15:11
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Commit: f915956

Run: 20171081761

Env🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux7141662848:39
🟨​aws windows7141862647:19
💚​aws-ucws linux7157750762:54
🔄​aws-ucws windows45157750578:29
💚​azure linux1341762653:24
🔄​azure windows51341462472:00
🔄​azure-ucws linux41357050589:52
🔄​azure-ucws windows61357050391:54
🔄​gcp linux3339563542:36
💚​gcp windows1339963344:04
21 interesting tests: 13 flaky, 7 KNOWN, 1 SKIP
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K💚​R💚​R💚​R💚​R💚​R💚​R🔄​f💚​R
🔄​TestAccept/bundle/integration_whl/base✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION✅​p✅​p✅​p✅​p✅​p🔄​f✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task✅​p✅​p✅​p✅​p✅​p🔄​f🔄​f✅​p✅​p✅​p
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p🔄​f✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p🔄​f🔄​f✅​p✅​p
🔄​TestAccept/bundle/resources/dashboards/simple✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p
🔄​TestAccept/bundle/resources/dashboards/simple/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestAccept/bundle/resources/permissions/factcheck✅​p✅​p✅​p✅​p✅​p✅​p🔄​f🔄​f🙈​s🙈​s
🔄​TestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p🔄​f🔄​f
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K💚​R🔄​f🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R🔄​f
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
Top 50 slowest tests (at least 2 minutes):
durationenvtestname
23:33azure-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
22:59azure-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
22:51azure-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
16:07azure windowsTestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
14:55azure windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=direct
14:03gcp windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
13:57azure linuxTestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform
12:52azure-ucws linuxTestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=direct
12:42azure-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
12:06azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
11:37aws-ucws windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
11:36azure linuxTestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=direct
11:25azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
11:19aws-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
11:06gcp windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
11:00aws linuxTestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS
10:39azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
10:37gcp linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
10:17aws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
10:03gcp linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
10:02aws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
9:57aws linuxTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform
9:54aws-ucws windowsTestAccept/bundle/resources/database_instances/single-instance/DATABRICKS_BUNDLE_ENGINE=direct
9:53azure linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
9:50azure windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
9:46aws linuxTestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_14.3_LTS
9:41aws-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
9:40aws-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
9:39azure-ucws windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
9:23aws-ucws windowsTestAccept/bundle/integration_whl/custom_params/DATABRICKS_BUNDLE_ENGINE=terraform
9:13aws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
9:13azure-ucws linuxTestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
9:11azure windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
9:05azure-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
9:04aws windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
8:52aws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:51azure windowsTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform
8:48aws-ucws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:47aws-ucws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:45aws-ucws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:42azure-ucws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
8:38azure-ucws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:34aws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:30azure linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:29azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
8:23gcp linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:18aws linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=direct
8:13aws linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
8:13aws-ucws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:11aws-ucws windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=direct

denik added a commit that referenced this pull request May 20, 2026
## Changes
Instead of parsing into any and then converting into the target type, we
now store json.RawMessage and then parse into target type when needed.
## Why
One issue it fixes is that JSON parser by default parses integers as
float64, losing precision. This was found by @varundeepsaini and a fix
is proposed in #4136
This PR is more general though, fully relying on the type to parse
itself and it should more efficient as well as we're not walking the
type twice & we're not having 2 copies of it.
## Tests
New test where int64 field is initialized with large numbers. It fails
on main.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@denik@eng-dev-ecosystem-bot@andrewnester