Skip to content

Refine RL workflow & tune RL models under GYM - #577

Merged
Jinyu-W merged 49 commits into
v0.3from
rl_workflow_refine
Feb 17, 2023
Merged

Refine RL workflow & tune RL models under GYM#577
Jinyu-W merged 49 commits into
v0.3from
rl_workflow_refine

Conversation

@lihuoran

@lihuoranHuoran Li (lihuoran) commented Feb 6, 2023

Copy link
Copy Markdown
Contributor

Description

  • Refine RL workflow
    • Refine rollout workflow. Now we can:
      • Run specific number of steps in rollout.
      • Use num_eval_episodes to control the number of episodes during evaluation.
    • Add AbsEnvSampler.metrics. Let env samplers use this attribute to manage metrics during rollout.
    • Add Callback, a generate interface to add customized operations in each phase of the workflow. Add two Callback instances: Checkpoint & MetricsRecorder.
    • Other miner code refinements.
  • Tune RL models under GYM
    • Add DDPG. Optimize the performance of PPO, SAC, and DDPG.
    • Fix several RL algorithm bugs.
    • Re-organize RL jobs' output paths.
    • Re-organize RL test's file structure.

Linked issue(s)/Pull request(s)

Type of Change

  • Non-breaking bug fix
  • Breaking bug fix
  • New feature
  • Test
  • Doc update
  • Docker update

Related Component

  • Simulation toolkit
  • RL toolkit
  • Distributed toolkit

Has Been Tested

  • OS:
    • Windows
    • Mac OS
    • Linux
  • Python version:
    • 3.7
    • 3.8
    • 3.9
  • Key information snapshot(s):

Needs Follow Up Actions

  • New release package
  • New docker image

Checklist

  • Add/update the related comments
  • Add/update the related tests
  • Add/update the related documentations
  • Update the dependent downstream modules usage

Huoran Li (lihuoran)and others added 30 commits April 24, 2022 11:58
@lihuoran
PPO, SAC, DDPG passed
6bc7f0b
@lihuoran
Explore in SAC
b3f5aef
@lihuoran
Test GYM on server
5dab711
@lihuoran
Sync server changes
211c06f
@lihuoran
Merge branch 'v0.3' into rl_benchmark_debug
f92f7f1
@lihuoran
pre-commit
514250a
@lihuoran
Ready to try on server
fc0c02d
@lihuoran
.
9fcdf42
@lihuoran
.
01b5a94
@lihuoran
.
dd27eed
@lihuoran
.
1c8f258
@lihuoran
.
1aa1085
@lihuoran
Performance OK
148af38
@lihuoran
Move to tests
99ff7b9
@lihuoran
Remove old versions
65ba1a1
@lihuoran
PPO done
f4a85b8
@lihuoran
Start to test AC
2349191
@lihuoran
Start to test SAC
f6f7dae
@lihuoran
SAC test passed
110fec4
@lihuoran
Multiple round in evaluation
2a1ccd5
@lihuoran
Modify config.yml
c371220
@lihuoran
Add Callbacks
a65d902
@lihuoran
[wip] SAC performance not good
aa484f8
@lihuoran
[wip] still not good
84ec6e6
@Jinyu-WJinyu Wang
update for some PR comments; Add a MARKDOWN file (#576)
0ceaac4
Co-authored-by: Jinyu Wang <wang.jinyu@microsoft.com>
@lihuoran
Use FullyConnected to replace mlp
aad41d9
@lihuoran
Update action bound
8884231
@lihuoran
Merge branch 'rl_benchmark_debug' into rl_workflow_refine
0a01fb1
@lihuoran
???
0bd25ca
@lihuoran
Change gym env wrapper metrics logci
8781dd6
@codecov

codecovBot commented Feb 6, 2023•
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #577 (9371949) into v0.3 (214383f) will not change coverage.
The diff coverage is n/a.

@@ Coverage Diff @@## v0.3 #577 +/- ##
=======================================
Coverage 79.30% 79.30% =======================================
Files 86 86 Lines 5464 5464 =======================================
Hits 4333 4333 Misses 1131 1131
FlagCoverage Δ
unittests79.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@lihuoran
Minor refinement of CIM RL
627b7d1
@Jinyu-W
Jinyu/rl workflow refine (#578)
8386312
* remove useless files; add device mapping; update pdoc
* add default checkpoint path; fix distributed worker log path issue; update example log path
* update performance doc
* remove tests/rl/algorithms folder
Jinyu-W
Jinyu-W reviewed Feb 9, 2023
View reviewed changes
Comment threadexamples/rl/cim_distributed.yml
@@ -10,7 +10,7 @@

job: cim_rl_workflow

@Jinyu-WJinyu-WFeb 8, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: runtime error

Comment threadmaro/rl/workflows/callback.py Outdated
Comment threadmaro/rl/workflows/callback.py Outdated
Comment threadmaro/rl/workflows/callback.py
Comment threadmaro/rl/workflows/callback.py
Comment threadmaro/rl/rollout/env_sampler.py Outdated
Comment threadmaro/rl/rollout/env_sampler.py
@lihuoran
Resolve PR comments
b05c849
@lihuoran
Compare PPO with spinning up (#579)
ab5e675
* [wip] compare PPO
* PPO matching
* Revert unnecessary changes
* Minor
* Minor
@Jinyu-W@lihuoran
SAC Test parameters update (#580)
e180f10
* fix sac to_device issue; update sac gym test parameters
* add rl test performance plot func
* update sac eval interval config
* update sac checkpoint interval config
* fix callback issue
* update plot func
* update plot func
* update plot func
* update performance doc; upload performance images
* Minor fix in callbacks; refine plot.py format.
* Add n_interactions. Use n_interactions to plot curves.
* pre-commit
---------
Co-authored-by: Huoran Li <huo53926@126.com>
Co-authored-by: Huoran Li <huoranli@microsoft.com>
@lihuoran
Episode truncation & early stopping (#581)
9371949
* Add truncated logic
* (To be tested) early stop
* Early stop test passed
* Test passed
* Random action. To be tested.
* Warmup OK
* Pre-commit
* random seed
* Revert pre-commit config
@Jinyu-W
Jinyu-W merged commit b8a955e into v0.3Feb 17, 2023
@lihuoran
Huoran Li (lihuoran) deleted the rl_workflow_refine branch February 22, 2023 04:53
@Jinyu-WJinyu-W mentioned this pull request Mar 20, 2023
Merged
21 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@Jinyu-WJinyu-WJinyu-W approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lihuoran@Jinyu-W