Skip to content

[Feature] collectors internals docs - #3796

Merged
vmoens merged 3 commits into
pytorch:mainfrom
theap06:feat/collectors-internals-docs
Jun 1, 2026
Merged

[Feature] collectors internals docs#3796
vmoens merged 3 commits into
pytorch:mainfrom
theap06:feat/collectors-internals-docs

Conversation

@theap06

@theap06 theap06 commented May 24, 2026

Copy link
Copy Markdown
Collaborator

Fixes #3746

Summary

Adds the architecture / internals layer between the API reference and tutorials that contributors and debuggers have been missing for the collector subsystem and TorchRL-specific vocabulary. Docs-only. No runtime changes.

Targets the gap where SyncDataCollector's per-step mechanics (_carrier, the three sync points, the device-cast flags) are load-bearing for anyone debugging RNN, replay, or device-placement issues but have until now only lived in inline comments inside torchrl/collectors/_single.py.

theap06 and others added 2 commits May 23, 2026 00:26
@theap06 @claude
[Doc] Collector internals page + glossary + rollout docstring expansion
ff17645 
Adds the architecture/internals layer between API reference and tutorials
that contributors and debuggers have been missing for the collector and
key TorchRL-specific vocabulary.

- New collectors_internals.rst: per-timestep rollout flow diagram,
  carrier (formerly "shuttle") lifecycle, the three sync points
  (_sync_policy / _sync_env / _sync_storage), and the device-cast flags
  (_cast_to_policy_device / _cast_to_env_device / _shuttle_has_no_device).
- New glossary.rst with entries for carrier/shuttle, in_keys/out_keys,
  _AcceptedKeys, set_keys, recurrent mode, TensorDictPrimer, is_init,
  trajectory ID, storing/policy/env_device, no_cuda_sync, compact_obs,
  functional loss, tensor_keys.
- Expanded SyncDataCollector.rollout docstring with a 5-bullet
  high-level outline of the per-timestep flow, cross-linking to the new
  internals page.
- collectors_basics.rst gains a track_traj_ids paragraph and a short
  Trajectory IDs section explaining the downstream consumers
  (SliceSampler, split_trajectories).
- Both new pages wired into the corresponding toctrees.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@theap06
[Docs] Fix recurrent-mode glossary entry
918f4ec 
Three factual corrections in the glossary's `recurrent mode` entry:

- Removed reference to ``LSTMModule.set_recurrent_mode``, which was
  removed in v0.8 (the method now raises and redirects users to the
  standalone context manager).
- Clarified that ``recurrent_mode_state_manager`` is the backing
  singleton, not a context manager — the context manager is
  ``set_recurrent_mode``. Noted the thread / asyncio-task locality
  via ``contextvars.ContextVar``.
- Fixed the sequential/recurrent label swap: collectors run in
  sequential mode (one step per call), losses in recurrent mode
  (full sequence per call).
@pytorch-bot

pytorch-bot Bot commented May 24, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3796

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures

As of commit 2e2cb9a with merge base 634979d (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 24, 2026
@github-actions

github-actions Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR Title Label Error

PR title must start with a label prefix in brackets (e.g., [BugFix]).

Current title: Feat/collectors internals docs

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix Label Applied Example
[BugFix] BugFix [BugFix] Fix memory leak in collector
[Feature] Feature [Feature] Add new optimizer
[Doc] or [Docs] Documentation [Doc] Update installation guide
[Refactor] Refactoring [Refactor] Clean up module imports
[CI] CI [CI] Fix workflow permissions
[Test] or [Tests] Tests [Tests] Add unit tests for buffer
[Environment] or [Environments] Environments [Environments] Add Gymnasium support
[Data] Data [Data] Fix replay buffer sampling
[Performance] or [Perf] Performance [Performance] Optimize tensor ops
[BC-Breaking] bc breaking [BC-Breaking] Remove deprecated API
[Deprecation] Deprecation [Deprecation] Mark old function
[Quality] Quality [Quality] Fix typos and add codespell

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

@theap06 theap06 changed the title Feat/collectors internals docs [Feat] collectors internals docs May 24, 2026
@github-actions

github-actions Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR Title Label Error

Unknown or invalid prefix [Feat].

Current title: [Feat] collectors internals docs

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix Label Applied Example
[BugFix] BugFix [BugFix] Fix memory leak in collector
[Feature] Feature [Feature] Add new optimizer
[Doc] or [Docs] Documentation [Doc] Update installation guide
[Refactor] Refactoring [Refactor] Clean up module imports
[CI] CI [CI] Fix workflow permissions
[Test] or [Tests] Tests [Tests] Add unit tests for buffer
[Environment] or [Environments] Environments [Environments] Add Gymnasium support
[Data] Data [Data] Fix replay buffer sampling
[Performance] or [Perf] Performance [Performance] Optimize tensor ops
[BC-Breaking] bc breaking [BC-Breaking] Remove deprecated API
[Deprecation] Deprecation [Deprecation] Mark old function
[Quality] Quality [Quality] Fix typos and add codespell

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

@theap06 theap06 changed the title [Feat] collectors internals docs [Feature] collectors internals docs May 24, 2026
@github-actions github-actions Bot added the Feature New feature label May 24, 2026
vmoens
vmoens reviewed May 31, 2026
View reviewed changes
Comment thread docs/source/reference/collectors_internals.rst Outdated
Comment thread docs/source/reference/collectors_internals.rst Outdated
Comment thread docs/source/reference/collectors_internals.rst Outdated
Comment thread docs/source/reference/glossary.rst Outdated
Comment thread docs/source/reference/glossary.rst
@vmoens

vmoens commented May 31, 2026

Copy link
Copy Markdown
Collaborator

@theap06 I edited a bunch of things in response to my own comments hope you don't mind

@theap06

theap06 commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator Author

@theap06 I edited a bunch of things in response to my own comments hope you don't mind

Thanks for the cleanup! the persistence-across-next() is better than what I had and I'll keep it in mind for the recurrent lifecycle docs in #3792 too. Happy with all the changes;

@theap06
theap06 requested a review from vmoens June 1, 2026 02:11
@vmoens
vmoens merged commit 551bcd3 into pytorch:main Jun 1, 2026
104 of 109 checks passed
theap06 added a commit to theap06/rl that referenced this pull request Jun 12, 2026
@theap06
[Doc] Tutorial: recurrent training on sequence batches
69543ac 
Adds tutorials/sphinx-tutorials/recurrent_sequence_training.py — the
multi-step / sequence-training complement to dqn_with_rnn.py (which
covers single-step recurrent DQN at collection time).

Walks through the post-pytorch#3695 recurrent contract end-to-end:

- Collector auto-wiring of InitTracker + the recurrent-state primer
  via auto_register_policy_transforms=True
- Trajectory-aware sampling with SliceSampler
- Multi-step LSTM forward under set_recurrent_mode(True)
- Boundary safety: hand-built two-trajectory packed batch + isolation
  check that proves hidden state does not leak across is_init markers
- A tiny end-to-end training loop closing the BC-style sequence path

Runs in ~3s on CPU. Cross-references the recurrent state lifecycle
guide (pytorch#3792), collector internals page (pytorch#3796), and the glossary.

Toctree entry added to docs/source/index.rst next to dqn_with_rnn.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vmoens vmoens vmoens approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Collectors Documentation Improvements or additions to documentation Feature New feature Integrations/torch_geometric Integrations

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Doc] Improve navigability for new contributors

2 participants

@theap06 @vmoens