fix(awareness-service): dedupe deliveries by content hash so envelope updates re-deliver

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coodos]

Description of change

Fixes the delivery dedup collision in awareness-service that silently dropped every MetaEnvelope update after the first ingest.

Delivery was uniquely keyed on (subscriptionId, packetId), and since Packet is upserted on the MetaEnvelope id, every re-ingest of an updated envelope reused the same packetId. IngestService.ingest inserts with .orIgnore(), so the second delivery hit the unique key and was silently discarded — a subscriber received each envelope exactly once, ever.

The MetaEnvelope id (packetId) is intentionally not part of the dedup decision anymore; content is. The fix adds a contentHash column to Delivery and changes the unique key to (subscriptionId, packetId, contentHash):

• Re-ingest of the same payload → same hash → .orIgnore() still dedupes. ✅
• Re-ingest of an updated payload → new hash → no collision → a new delivery is queued. ✅
• Retry / dead-letter flow is untouched.

contentHash = sha256(stableStringify(payload.data)), where stableStringify sorts object keys recursively so key ordering does not affect the hash.

Includes a migration that adds the column (nullable → backfilled with a constant for legacy rows → NOT NULL) and swaps the unique constraint.

Issue Number

closes#984

Type of change

• Fix (a change which fixes an issue)

How the change has been tested

• npm run build (tsc) in services/awareness-service/api passes.
• Reviewed migration SQL against the Init schema; up applies cleanly and down reverses it.
• Traced the issue's repro: first ingest queues 1 delivery; unchanged re-ingest is deduped by .orIgnore(); changed re-ingest produces a new contentHash and queues a new delivery, so the delivery count increments.

Change checklist

• I have ensured that the CI Checks pass locally
• I have removed any unnecessary logic
• My code is well documented
• I have signed my commits
• My code follows the pattern of the application
• I have self reviewed my code

Summary by CodeRabbit

• Bug Fixes
  • Improved webhook delivery deduplication to account for payload content, preventing duplicate deliveries of identical requests and reducing redundant webhook notifications.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR changes webhook delivery deduplication from packet-based to content-based. It adds a new contentHash utility module that performs deterministic JSON serialization and SHA-256 hashing, updates the Delivery entity schema to include a contentHash column and modifies the unique constraint, applies a database migration, and integrates content hashing into IngestService delivery queueing.

Changes

Webhook Delivery Content-Based Deduplication

Layer / File(s)	Summary
Content hash utility services/awareness-service/api/src/utils/contentHash.ts	Exports stableStringify for deterministic JSON serialization (recursively sorts object keys, preserves array order) and contentHash for SHA-256 hashing of normalized payloads.
Delivery entity schema update services/awareness-service/api/src/database/entities/Delivery.ts	Entity adds contentHash varchar column (SHA-256 of packet payload) and updates unique constraint from (subscriptionId, packetId) to (subscriptionId, packetId, contentHash).
Database migration services/awareness-service/api/src/database/migrations/1780000000000-AddDeliveryContentHash.ts	Migration adds contentHash column to deliveries table, backfills existing rows with empty-string default, enforces NOT NULL, and replaces prior unique constraint with new three-column constraint. Down migration reverses.
IngestService integration services/awareness-service/api/src/services/IngestService.ts	Service imports contentHash utility, computes payload hash, and assigns it to each queued Delivery. Updated comments describe idempotency: unchanged payloads skip redelivery; updated envelopes queue new deliveries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• #984: Implements the same code-level changes—adds contentHash column, updates the Delivery unique constraint, modifies IngestService to set content hashes, and includes the database migration.

Poem

🐰 Hash the packets, sort the keys with care,
Content-wise we deduplicate with flair!
No more phantom doubles on the wire—
Same payload, same hash, one truth to admire.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding content-hash-based deduplication to delivery queuing so that envelope updates trigger new deliveries instead of being silently dropped.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: issue number, type of change, testing details, and completed checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/delivery-dedup-content-hash

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}