[Feature Request] Implement TQC for the example algorithms

[maxweissenbacher]

Motivation

I suggest adding an implementation of TQC to the examples. I suggest adding this as a request in the 'call for distributions' stack. I would be happy to take on the implementation.

Solution

Add a performant, clear and minimal implementation of TQC to the examples. I would base the implementation on the already existing implementation of SAC in the examples. This seems reasonable due to the overall similar structure of the algorithms. In the Stable-Baselines3 library implementation of TQC, the implementation is likewise based on their corresponding implementation of SAC.

Checklist

• I have checked that there is no similar issue in the repo (required)
• Add a request to the call for contributions.
• Adapt the implementation of SAC from the examples to TQC
• Benchmark?

### Tasks

[vmoens]

Hi @maxweissenbacher
Adding TQC would be fantastic and greatly appreciated!
Any help we can provide, just let us know. I'm adding this to the call for contributions straight away!

[maxweissenbacher]

Hi @vmoens, that's great, thank you! I'll get started right away and come back here in case I need assistance (more likely than not).

[maxweissenbacher]

I cannot seem to get the logger to work for me. By running the (unchanged) training script for the SAC example, trying to execute theget_logger function gives me the following error:

wandb: WARNING Path sac_logging/wandb/ wasn't writable, using system temp directory.

Any suggestions?

[maxweissenbacher]

Another (very silly) question... this is my first open source contrib - how do I contribute the code to the github? Do I need to fork the repository first, and submit a pull request?

[vmoens]

For the logger it seems you lack write access in the folder where you're executing the code.

For the OSS contribution:

• make a fork of torchrl on your username
• write code in your separate branch locally on your machine
• when you're at a stage where you feel the code is mature and/or you want to show it to us to get feeeback, commit the changes on that branch and push it to your fork. Then create a PR from that branch towards main on the PyTorch org.

You will need to format the code etc for which I invite you to read the CONTRIBUTE.md file in torchrl.

Thanks again for your help!

[maxweissenbacher]

Thank you for all this info. I have submitted a pull request now, hopefully I did it right!

I couldn't get the logging to work, so I implemented a custom logger to use on my local machine instead - however the logging is just as in the other example files in the pull request, so it should hopefully work as expected for you.

[chinu0609]

hi @vmoens I am new here I want to contribute could you point me to any beginner friendly issue, or something in your mind thank you.

[gtnv]

I think this needs to be done and put to rest. I will make the PR today. Claiming ownership.

[vmoens]

You can use [forward slash]assign

[gtnv]

nice. didnt know that was available here.

[gtnv]

/assign

[gtnv]

cc @theap06 @vmoens Im done implementing TQC and have some updates to share.

The PRs are in order of merge-priority.
[BugFix] [TQC] [1/3] Support list-valued critics in SoftUpdate

[Feature] [TQC] [2/3] Add truncated quantile critics

[Test] [TQC] [3/3] Add TQC numerical contracts test

While testing my work (across all 3) on h100s, I found 2 old bugs resurfacing as blockers for a functional implementation. The first was this one: [BUG] SoftUpdate fails for list-valued SAC critics . Its a leftover bug exposed by an old bugfix. The second is [BUG] Numerical Instability issues with torchrl.modules.TanhNormal This second bug is linked to loss behavior. evidence in #4078 . (I didnt observe the exact same bug, but codex told me this is the issue for it. codex is wrong so I will probably make a new issue and pr to fix).

I have a fix up for the first problem. will draft second fix. Thanks as always guys.