[BugFix] Build TD3Trainer with multiprocessing collectors

[aswanth-07]

Description

Allow TD3Trainer to use action bounds already configured on its loss partial. The trainer now reads a local collector environment only when the loss does not provide bounds or action_spec.

This lets AsyncCollector, MultiSyncCollector, and MultiAsyncCollector build a TD3 trainer without creating an environment in the main process. The existing single-process fallback still derives the action spec from collector.env.

The regression builds the trainer against collector interfaces that deliberately have no env attribute and verifies that the configured bounds reach TD3Loss unchanged.

Motivation and Context

Closes #4366.

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds core functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Validation

• python -m pytest -q test/test_configs.py -k "td3_trainer_uses_loss_bounds_without_collector_env or td3_trainer_config" (4 passed)
• python -m ruff check torchrl/trainers/algorithms/configs/trainers.py test/test_configs.py
• git diff --check

Checklist

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

AI assistance disclosure: AI-assisted development tools were used during investigation and implementation. The reported tests were run against the final diff.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4368

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

The TD3 fix looks correct and appropriately scoped: explicitly configured bounds or action_spec take precedence, and the existing local-environment fallback remains available. No blocking correctness findings.

Validation at 0a7a9ed: the four targeted pytest cases passed; additional construction checks covered explicit bounds/specs, a Composite spec, a Hydra loss partial, local fallback/overrides, and rejection of missing or conflicting action domains. Construction also passed with real AsyncCollector, MultiSyncCollector, and MultiAsyncCollector workers, including shutdown. The broader CI workflows are still awaiting approval.

Follow-ups to keep separate from this fix:

• Apply the same “only query the environment when necessary” principle to SACTrainer and CQLTrainer. They already respect an explicit loss action spec, but still call getattr_env when it is absent even if numeric target_entropy or the actor's spec makes that query unnecessary. This matters for collectors that do not expose that accessor.
• Remove analogous local-env assumptions from other training recipes, notably GAIL's make_gail_discriminator(cfg, collector.env, ...), whose helper only needs observation/action specs. TD3 could also eventually infer omitted bounds through the existing collector.getattr_env API where supported.
• Support async environments and async policy inference consistently across all recipes, using torchrl.envs.AsyncEnvPool and torchrl.modules.inference_server.InferenceServer / PolicyClientModule, with AsyncBatchedCollector as the integrated path. Expose batching/device placement through InferenceServerConfig / InferenceDeviceConfig and shared Hydra configuration; the current registry only has sync, async, multi_sync, and multi_async. Include recipe smoke coverage for collection, weight synchronization, and shutdown, while preserving each algorithm's rollout and policy-freshness requirements.

[vmoens]

Nonblocking test cleanup: the three class-spec mocks exercise the same missing-env branch without starting any workers. With roughly 50 added test lines for 18 implementation lines (~2.8:1), keep the useful real actor/loss setup but prefer parametrizing meaningful cases such as explicit bounds, explicit action_spec, and the local-env fallback over equivalent mock interfaces. This would cover more of the changed behavior without growing the fixture. Also move the newly added imports to module scope, per the repository import rule.