feat: add JSONL draft diff logic and update command support

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM

[nborges-aws]

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

• add examples missing from the remote draft
• update examples where exampleId exists remotely but has content changes
• delete remote examples no longer present in local draft
• ensure untouched examples remain intact

Summary of changes:

• Adds eval dataset update --id <dataset-id> --file-path <path>.
• Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
• Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
• Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
• Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
• Validate remote exampleId's are unique and responses have one ID per example
• Adds shared IO helpers for JSONL parsing and generic text file reads
• Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
• Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

Manually tested commands using bun run ... against personal account with following procedure:

• Created dataset from local JSONL (and waited until ACTIVE)
• downloaded remote draft
• edited local JSONL with one add, one update, one delete
• ran eval dataset udpate
• confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
• confirmed local JSONL rewritten with new exampleId
• downloaded remote draft, confirmed it matched the updated local file
• published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

• bun test (851 pass, 0 fail)
• bun run typecheck clean
• bun run lint:check clean
• bun run format:check clean
• bun run build clean

How have you tested the change?

• I ran npm run test:unit and npm run test:integ
• I ran npm run typecheck
• I ran npm run lint
• If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1926 +/- ##
============================================
+ Coverage 96.88% 96.95% +0.06% 
============================================
Files 342 346 +4 Lines 19263 19737 +474 ============================================
+ Hits 18663 19136 +473 - Misses 600 601 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

LGTM, but I would add onProgress bar

[jariy17]

I don't know if lineNumber would be useful metadata for telemetry.

[jariy17]

runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

[nborges-aws]

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

[jariy17]

We need golden tests for update flow.

[jariy17]

Do we need this tests? Will the handler tests cover this for us?

[nborges-aws]

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

[jariy17]

Do we need these unit tests? Won't the handler tests cover this?

[nborges-aws]

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

[jariy17]

Nice job. The update diff is pretty good!

[aidandaly24]

Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

[aidandaly24]

I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

[nborges-aws]

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

[aidandaly24]

Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

[nborges-aws]

Great catch; absolutely should factor datasetId into the calc. Updated

[aidandaly24]

Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

[nborges-aws]

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

[jariy17]

LGTM