When a journaled model step fails, Restate correctly uses SdkInternalException for replay control flow. The wrapper then does:
exceptSdkInternalBaseExceptionase:
raiseException("Internal error during model call") fromeThis conflicts with Restate’s Python error-handling guidance (re-raise internal exceptions; don’t catch them as generic Exception). Integrators must walk __cause__ chains to recover semantics.
For LLM provider non-retryable 4xx (e.g. invalid tool schema → HTTP 400), the failure should be TerminalError inside the journaled step, not a retryable transient. Otherwise Restate retries permanently invalid requests—contrary to the AI integration guide (“handle terminal errors”, bound LLM retries).
Proposed behavior:
exceptSdkInternalBaseExceptionase:
ifisinstance(e, SuspendedException):
raise# Optional: inspect e.__cause__ for provider HTTP errorsifis_non_retryable_provider_4xx(e.__cause__):
raiseTerminalError("Model request rejected by provider", status_code=400) frome.__cause__raise# re-raise SdkInternalBaseException unchanged — do NOT wrap in Exception(...)Policy suggestion for upstream defaults:
| Provider status | Treatment |
|---|
| 400, 401, 403, 404, 422, … | TerminalError (non-retryable) |
| 408, 429 | Retryable (existing transient path) |
| 5xx | Retryable per RunOptions |
Proposals:
- Stop wrapping SdkInternalBaseException in generic Exception.
- Either adopt a default provider-status policy or expose a hook on RestateAgent / RestateModelWrapper for classifying model failures before journaling.
- Document recommended RunOptions(max_attempts=…) for LLM journal steps.
When a journaled model step fails, Restate correctly uses SdkInternalException for replay control flow. The wrapper then does:
This conflicts with Restate’s Python error-handling guidance (re-raise internal exceptions; don’t catch them as generic Exception). Integrators must walk __cause__ chains to recover semantics.
For LLM provider non-retryable 4xx (e.g. invalid tool schema → HTTP 400), the failure should be TerminalError inside the journaled step, not a retryable transient. Otherwise Restate retries permanently invalid requests—contrary to the AI integration guide (“handle terminal errors”, bound LLM retries).
Proposed behavior:
Policy suggestion for upstream defaults:
Proposals: