Learning Deep Reinforcement Learning

[xariusdrake (x)]

I am currently learning about deep reinforcement learning, which builds upon the foundations of deep learning. I think that understanding deep reinforcement learning can be especially helpful for those who already have a foundation in deep learning.

I am currently learning about reinforcement learning and implementing MuZero and RLHF (a technique used to train ChatGPT) from scratch. I would like to share my learning progress here.

MuZero’s repo: GitHub - xrsrke/muzero: Implement MuZero from scratch [WORK IN PROGRESS]

InstructGPT’s repo: GitHub - xrsrke/instructGOOSE: Implementation of Reinforcement Learning from Human Feedback (RLHF)

P/S: If posting about reinforcement learning is not allowed in this channel, please let me know and I will move my posts to a different location

EDIT 1: Added github repo

[xariusdrake (x)]

TIL: how to calculate the loss of reinforce algorithm (policy gradient)

[xariusdrake (x)]

TIL: Fixed my reinforce algorithm training loop (previously i got the loss function wrong), understand how advantage function works in actor-critic (will implement from scratch very soon)

Github: reinforcement-learning/05_policy_gradient.ipynb at main · xrsrke/reinforcement-learning · GitHub

[xariusdrake (x)]

TIL: how the soft actor-critic optimizes its policy and warming up my math muscles

[xariusdrake (x)]

TIL: yay, finally understand the update rule for the policy in the soft actor-critic (updated version of previous note)

---

[xariusdrake (x)]

TIL: Understand why the three loss functions in soft actor-critic are designed the way they are

and learning RLHF

---

---

---

---

[xariusdrake (x)]

the last few days i learned: some basics of RLHF, and PPO

[xariusdrake (x)]

the last two day i learned: the idea of MuZero, some basic of PPO, and how MCTS works (I will post my notes on the last two topics once I finish going through them)

[xariusdrake (x)]

the last four days i learned: implemented PPO from scratch

---

---

---

---

---

[xariusdrake (x)]

the last five days i learned: how RLHF works and implemented a vanilla representation network in MuZero (will fix it when all the components are put together)

---

---

[xariusdrake (x)]

the last three day i learned: implemented pairwise dataset, and reward model and reward loss in RLHF, and 1/4 of MuZero’s self-play

---

---

[xariusdrake (x)]

the last four days, I learned: implemented the agent objective, a reward trainer, 1/3 of the agent trainer in RLHF, 3/10 of MCTS in MuZero

[melonkernel (Christoffer Björkskog)]

Nice! Good work

[xariusdrake (x)]

Thanks

[xariusdrake (x)]

the last 6 days i learned (yes, I was stuck that long): figured out how to train use PPO to train RLHF

next i’m working on implement the RLHFTrainer: Actions · xrsrke/instructGOOSE · GitHub

---

[balnazzar (Andrea de Luca)]

@xariusdrake

yes, I was stuck that long

Six days is not a long time to be stuck with something, trust me

[xariusdrake (x)]

Technically speaking, I’ve been stuck on this for two weeks. But I learn different subjects every day, so it balances out. I count it as 6 days

[melonkernel (Christoffer Björkskog)]

Great work, keep it up

[xariusdrake (x)]

The last three days I learned: created a custom gym environment for RLHF

---

[xariusdrake (x)]

the last three days i learned : implemented RLHF Trainer

---

---

---