docs: prepare Outly shadow audit deliverables - #16
Conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Warning Review limit reached
Next review available in:31 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds three non-canonical Outly shadow-mode JSON Schemas and documentation covering schema boundaries, readiness evidence, implementation gaps, and unresolved pilot questions. ChangesOutly shadow-mode preparation
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/outly/OUTLY-OPEN-QUESTIONS.md`:
- Around line 45-55: Update the evidence questions in the “Evidence and audit”
section to explicitly address the required evidence-reference pair of
evidence_id and entry_hash, including whether Outly can provide and consume both
values and how cAPI verifies their binding to the associated action, decision,
and outcome.
- Around line 32-36: Update the open questions around DecisionV1 to explicitly
define consistency invariants between decision, modifications, and
human_review_required: specify when modifications must be empty or allowed, how
the HUMAN_REVIEW decision maps to human_review_required, and whether expired
decisions may be executed or must be rejected.
- Around line 26-28: Update the retry and replay questions around items 14–16 to
define stability and change semantics for action_id, execution_id, nonce, and
idempotency_key across retries. Explicitly specify the deduplication rule and
how already-seen correlation values are classified, covering both repeated
retries and duplicate outcomes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 07e40fea-7544-4a4b-bad7-af3d20039686
📒 Files selected for processing (6)
contracts/drafts/outly/ActionOutcomeV1.schema.jsoncontracts/drafts/outly/DecisionV1.schema.jsoncontracts/drafts/outly/ProposedActionV1.schema.jsoncontracts/drafts/outly/README.mddocs/outly/OUTLY-OPEN-QUESTIONS.mddocs/outly/OUTLY-SHADOW-MODE-READINESS-AUDIT.md
| 14. How does Outly retry a proposal, and which idempotency key remains stable across retries? | ||
| 15. What replay behavior should Outly expect for an already-seen nonce or idempotency key? | ||
| 16. What should happen if Veklom is unavailable: fail open, fail closed, or continue in an explicitly marked audit-only mode? |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Define retry identity across all correlation fields.
The draft schemas require action_id, execution_id, nonce, and idempotency_key, but these questions only cover action ID and idempotency-key stability. Specify which fields remain stable versus change per retry and the deduplication rule; otherwise retries may be misclassified as new executions or duplicate outcomes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/outly/OUTLY-OPEN-QUESTIONS.md` around lines 26 - 28, Update the retry
and replay questions around items 14–16 to define stability and change semantics
for action_id, execution_id, nonce, and idempotency_key across retries.
Explicitly specify the deduplication rule and how already-seen correlation
values are classified, covering both repeated retries and duplicate outcomes.
| 17. What exactly should `MODIFY` mean to Outly: modify parameters, route to another capability, reduce scope, cap amount, change timing, or something else? | ||
| 18. Can Outly consume a structured list of modifications, and which fields are safe to modify? | ||
| 19. What does `HUMAN_REVIEW` mean operationally, and who is the reviewer? | ||
| 20. Does `DENY` prevent Outly execution, or is the first pilot strictly observational? | ||
| 21. For a lane-3 action, what CAPPO authorization artifact can Outly correlate to the decision? |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Capture invariants between decision fields.
DecisionV1 requires decision, modifications, and human_review_required, but the questions do not define their consistency rules. Clarify whether modifications must be empty unless the decision is MODIFY, whether human_review_required must correspond to HUMAN_REVIEW, and how expired decisions affect execution.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/outly/OUTLY-OPEN-QUESTIONS.md` around lines 32 - 36, Update the open
questions around DecisionV1 to explicitly define consistency invariants between
decision, modifications, and human_review_required: specify when modifications
must be empty or allowed, how the HUMAN_REVIEW decision maps to
human_review_required, and whether expired decisions may be executed or must be
rejected.
| 27. What evidence reference does Outly already create, if any, and can it be linked to the Veklom decision? | ||
| ## Evidence and audit | ||
| 28. Which decision, proposal, and outcome fields must be retained? | ||
| 29. What audit retention, deletion, and data-residency requirements apply? | ||
| 30. Should evidence be anchored to gnomledger synchronously, asynchronously, or both? | ||
| 31. What should the system report when gnomledger is unavailable? | ||
| 32. Which fields are confidential and must not be treated as merely base64-encoded? | ||
| 33. Which parties may read decision and outcome evidence? | ||
| 34. What evidence receipt or ledger hash format can Outly consume? |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Make the evidence-reference contract explicit.
DecisionV1 and ActionOutcomeV1 require a non-null evidence reference to contain both evidence_id and entry_hash, while src/app/api/capi/v1/evidence/[id]/route.ts currently returns mocked evidence without entry_hash. Add an explicit question covering whether Outly can provide and consume this required pair, and how cAPI verifies its binding to the action, decision, and outcome.
🧰 Tools
🪛 LanguageTool
[grammar] ~51-~51: Ensure spelling is correct
Context: ...ply? 30. Should evidence be anchored to gnomledger synchronously, asynchronously, or both?...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~52-~52: Ensure spelling is correct
Context: ... 31. What should the system report when gnomledger is unavailable? 32. Which fields are co...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/outly/OUTLY-OPEN-QUESTIONS.md` around lines 45 - 55, Update the evidence
questions in the “Evidence and audit” section to explicitly address the required
evidence-reference pair of evidence_id and entry_hash, including whether Outly
can provide and consume both values and how cAPI verifies their binding to the
associated action, decision, and outcome.
Uh oh!
There was an error while loading. Please reload this page.
Summary by CodeRabbit
New Features
Documentation