Uh oh!
There was an error while loading. Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork 45
Documentation update#279
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Uh oh!
There was an error while loading. Please reload this page.
Documentation update #279
Changes from all commits
7bcec34b90716573896e4699adf8011d49df77baaf8aa599105d4d6c748a643377c72ede0ac2c669d4391fafc4cd729df78f824b1e21e85a192f75fFile filter
Filter by extension
Conversations
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Uh oh!
There was an error while loading. Please reload this page.
This file was deleted.
Uh oh!
There was an error while loading. Please reload this page.
This file was deleted.
Uh oh!
There was an error while loading. Please reload this page.
This file was deleted.
Uh oh!
There was an error while loading. Please reload this page.
This file was deleted.
Uh oh!
There was an error while loading. Please reload this page.
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,149 @@ | ||||||||||||||||||||||
| # Exosphere Architecture | ||||||||||||||||||||||
| This document provides a comprehensive overview of Exosphere's top-level architecture, focusing on state execution, fanout mechanisms, and the unites keyword for stage unification. | ||||||||||||||||||||||
| ## Overview | ||||||||||||||||||||||
| Exosphere is built around a **state-based execution model** where workflows are composed of discrete states that can be executed independently. The architecture consists of several key components: | ||||||||||||||||||||||
| - **State Manager**: Central orchestrator that manages state lifecycle and transitions | ||||||||||||||||||||||
| - **Runtimes**: Distributed execution engines that process individual states | ||||||||||||||||||||||
| - **Graph Templates**: Declarative workflow definitions | ||||||||||||||||||||||
| - **States**: Individual execution units with inputs, outputs, and metadata | ||||||||||||||||||||||
| ## State Execution Model | ||||||||||||||||||||||
| ### State Lifecycle | ||||||||||||||||||||||
| Each state in Exosphere follows a well-defined lifecycle: | ||||||||||||||||||||||
| ``` | ||||||||||||||||||||||
| CREATED → QUEUED → EXECUTED/ERRORED → SUCCESS | ||||||||||||||||||||||
| ``` | ||||||||||||||||||||||
| 1. **CREATED**: State is initialized with inputs and dependencies | ||||||||||||||||||||||
| 2. **QUEUED**: State is ready for execution and waiting for a runtime | ||||||||||||||||||||||
| 3. **EXECUTED**: State finished (with outputs) — terminal for that state | ||||||||||||||||||||||
| 4. **ERRORED**: State failed during executionState and all its dependencies are complete | ||||||||||||||||||||||
| 5. **SUCCESS**: Workflow/branch-level success once all dependent states complete | ||||||||||||||||||||||
Comment on lines
+24
to
+29
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Clarify lifecycle text; fix concatenated sentence. Tighten definitions and remove the stray “executionState …” fragment. Apply: 1. **CREATED**: State is initialized with inputs and dependencies
2. **QUEUED**: State is ready for execution and waiting for a runtime
-3. **EXECUTED**: State finished (with outputs) — terminal for that state-4. **ERRORED**: State failed during executionState and all its dependencies are complete-5. **SUCCESS**: Workflow/branch-level success once all dependent states complete+3. **EXECUTED**: State finished (with outputs) — terminal for that state+4. **ERRORED**: State failed during execution+5. **SUCCESS**: Workflow/branch-level success once all dependent states complete📝 Committable suggestion
Suggested change
🧰 Tools🪛 LanguageTool[grammar] ~24-~24: There might be a mistake here. (QB_NEW_EN_OTHER) [grammar] ~26-~26: There might be a mistake here. (QB_NEW_EN_OTHER) [grammar] ~26-~26: There might be a mistake here. (QB_NEW_EN_OTHER) [grammar] ~27-~27: There might be a mistake here. (QB_NEW_EN_OTHER) [grammar] ~27-~27: There might be a mistake here. (QB_NEW_EN_OTHER) [grammar] ~28-~28: There might be a mistake here. (QB_NEW_EN_OTHER) 🤖 Prompt for AI Agents | ||||||||||||||||||||||
| ## Fanout Mechanism | ||||||||||||||||||||||
| ### Single vs Multiple Outputs | ||||||||||||||||||||||
| Exosphere supports two execution patterns: | ||||||||||||||||||||||
| 1. **Single Output**: A state produces one output and continues to the next stage | ||||||||||||||||||||||
| 2. **Multiple Outputs (Fanout)**: A state produces multiple outputs, creating parallel execution paths | ||||||||||||||||||||||
| ### Fanout Example | ||||||||||||||||||||||
| Consider a data processing workflow: | ||||||||||||||||||||||
| ```python hl_lines="9-11" | ||||||||||||||||||||||
| class DataSplitterNode(BaseNode): | ||||||||||||||||||||||
| async def execute(self) -> list[Outputs]: | ||||||||||||||||||||||
| data = json.loads(self.inputs.data) | ||||||||||||||||||||||
| chunk_size = 100 | ||||||||||||||||||||||
| outputs = [] | ||||||||||||||||||||||
| for i in range(0, len(data), chunk_size): | ||||||||||||||||||||||
| chunk = data[i:i + chunk_size] | ||||||||||||||||||||||
| outputs.append(self.Outputs( | ||||||||||||||||||||||
| chunk=json.dumps(chunk) | ||||||||||||||||||||||
| )) | ||||||||||||||||||||||
| return outputs # This creates fanout on each output | ||||||||||||||||||||||
| ``` | ||||||||||||||||||||||
| When this node executes: | ||||||||||||||||||||||
| 1. **Original state** gets the first chunk as output | ||||||||||||||||||||||
| 2. **Additional states** are created for each remaining chunk | ||||||||||||||||||||||
| 3. **All states** are marked as EXECUTED | ||||||||||||||||||||||
| 4. **Next stages** are created for each state independently | ||||||||||||||||||||||
| **This enables parallel processing of data chunks across multiple runtime instances.** | ||||||||||||||||||||||
| ## Unites Keyword | ||||||||||||||||||||||
| ### Purpose | ||||||||||||||||||||||
| The `unites` keyword is a powerful mechanism for **synchronizing parallel execution paths**. It allows a node to wait for multiple parallel states to complete before executing. | ||||||||||||||||||||||
| ### Unites Logic | ||||||||||||||||||||||
| When a node has a `unites` configuration: | ||||||||||||||||||||||
| 1. **Execution is deferred** until all states with the specified identifier are complete | ||||||||||||||||||||||
| 2. **State fingerprinting** ensures only one unites state is created per unique combination | ||||||||||||||||||||||
| 3. **Dependency validation** ensures the unites node depends on the specified identifier | ||||||||||||||||||||||
| ### Unites Example | ||||||||||||||||||||||
| ```json hl_lines="22-24" | ||||||||||||||||||||||
| { | ||||||||||||||||||||||
| "nodes": [ | ||||||||||||||||||||||
| { | ||||||||||||||||||||||
| "node_name": "DataSplitterNode", | ||||||||||||||||||||||
| "identifier": "data_splitter", | ||||||||||||||||||||||
| "next_nodes": ["processor_1"] | ||||||||||||||||||||||
| }, | ||||||||||||||||||||||
| { | ||||||||||||||||||||||
| "node_name": "DataProcessorNode", | ||||||||||||||||||||||
| "identifier": "processor_1", | ||||||||||||||||||||||
| "inputs":{ | ||||||||||||||||||||||
| "x":"${{data_splitter.outputs.data_chunk}}" | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
| "next_nodes": ["result_merger"] | ||||||||||||||||||||||
| }, | ||||||||||||||||||||||
Comment on lines
+96
to
+100
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fix invalid JSON: add missing commas after inputs objects. This currently fails JSON parsing and will trip CI linters. Apply: "inputs":{
"x":"${{data_splitter.outputs.data_chunk}}"
- }+ },
"next_nodes": ["result_merger"]
@@
"inputs":{
"x_processed":"${{processor_1.outputs.processed_data}}"
- }+ },
"unites": {
"identifier": "data_splitter"
},Also applies to: 104-107 🤖 Prompt for AI Agents | ||||||||||||||||||||||
| { | ||||||||||||||||||||||
| "node_name": "ResultMergerNode", | ||||||||||||||||||||||
| "identifier": "result_merger", | ||||||||||||||||||||||
| "inputs":{ | ||||||||||||||||||||||
| "x_processed":"${{processor_1.outputs.processed_data}}" | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
| "unites": { | ||||||||||||||||||||||
| "identifier": "data_splitter" | ||||||||||||||||||||||
| }, | ||||||||||||||||||||||
| "next_nodes": [] | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
| ] | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
| ``` | ||||||||||||||||||||||
NiveditJain marked this conversation as resolved.
Uh oh!There was an error while loading. Please reload this page. | ||||||||||||||||||||||
| In this example: | ||||||||||||||||||||||
| 1. `data_splitter` creates fanout with 3 outputs | ||||||||||||||||||||||
| 2. `processor_1` executes in parallel for all three data chunks | ||||||||||||||||||||||
| 3. `result_merger` waits for all processors to complete (unites with `data_splitter`) | ||||||||||||||||||||||
| 4. Only one `result_merger` state is created due to fingerprinting | ||||||||||||||||||||||
| ## Architecture Benefits | ||||||||||||||||||||||
| ### Scalability | ||||||||||||||||||||||
| - **Horizontal scaling**: Add more runtime instances to handle increased load | ||||||||||||||||||||||
| - **Parallel processing**: Fanout enables concurrent execution | ||||||||||||||||||||||
| - **Load distribution**: State manager distributes work across available runtimes | ||||||||||||||||||||||
| ### Fault Tolerance | ||||||||||||||||||||||
| - **State persistence**: All states are stored in the database | ||||||||||||||||||||||
| - **Retry mechanisms**: Failed states can be retried automatically | ||||||||||||||||||||||
| - **Recovery**: Workflows can resume from where they left off | ||||||||||||||||||||||
| ### Flexibility | ||||||||||||||||||||||
| - **Dynamic fanout**: Nodes can produce variable numbers of outputs | ||||||||||||||||||||||
| - **Synchronization**: Unites keyword provides precise control over parallel execution | ||||||||||||||||||||||
| - **Dependency management**: Automatic resolution of complex dependencies | ||||||||||||||||||||||
| ### Observability | ||||||||||||||||||||||
| - **State tracking**: Complete visibility into execution progress | ||||||||||||||||||||||
| - **Error handling**: Detailed error information and retry logic | ||||||||||||||||||||||
| - **Performance monitoring**: Track execution times and resource usage | ||||||||||||||||||||||
| Exosphere's architecture provides a robust foundation for building distributed, scalable workflows. The combination of state-based execution, fanout mechanisms, and the unites keyword enables complex parallel processing patterns while maintaining simplicity and reliability. | ||||||||||||||||||||||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧹 Nitpick (assertive)
Add a language tag to the lifecycle fence (MD040).
Apply:
In docs/docs/exosphere/architecture.md around lines 20 to 22, the lifecycle
fence is missing a language tag (MD040); update the code block opening from
totext and ensure the block content reads " CREATED → QUEUED →EXECUTED/ERRORED → SUCCESS" (preserve the leading space as shown), then close
the block with ``` so the fence is explicitly tagged as text.