P4-19.4 Supplementary Learning: A Map of Later Reinforcement-Learning Branches¶

Section ID: P4-19.4 Version: v2026.07.24

Subtitle: What problem concerns split DQN, PPO, and RLHF into later reinforcement-learning branches?

After reading P4-19.1 through P4-19.3, readers quickly encounter many more names when they study reinforcement learning further.

• DQN
• PPO, TRPO, A2C, A3C
• safe reinforcement learning
• offline reinforcement learning
• domain randomization
• RLHF, preference optimization

These names came from different periods and different bottlenecks, but they often rush in all at once. This Section focuses not on learning each implementation, but on organizing the larger flow of why did these names branch out?

This supplementary learning Section does not explain the basic definition of reinforcement learning from the beginning again. The handle for value-based reinforcement learning remains in P4-19.1, the handle for policy-based reinforcement learning remains in P4-19.2, and the handle for application risk remains in P4-19.3 and the concept glossary. Here, only the later names are arranged as a genealogy.

Scope Of This Supplementary Section¶

This Section answers the following questions.

• Why is DQN so often mentioned as the representative follow-up example after value-based reinforcement learning?
• What difficulty in policy-based reinforcement learning are PPO, TRPO, A2C, and A3C trying to reduce?
• Why did safe RL and offline RL become separate topics in real-world application contexts?
• How is domain randomization connected to sim-to-real problems?
• Why does RLHF connect general reinforcement-learning problems and LLM alignment?

The detailed training pipeline of RLHF and alignment practice returns in Part 5.

Goals Of This Supplementary Section¶

• You can organize the later names of reinforcement learning into four branches: value-based expansion, policy-based stabilization, real-world reinforcement, and LLM-alignment connection.
• You can explain on what tradition DQN and PPO each stand.
• You can say that safe RL, offline RL, and domain randomization all connect to the problem of not being able to try freely in reality.
• You can understand RLHF not as directly transplanting ordinary reinforcement learning into LLMs, but as a later branch reshaped for alignment problems.

First Draw The Large Map¶

The following table gives the large map.

So these later algorithm names are better read not as smarter reinforcement learning, but as branches that let reinforcement learning operate under wider real-world constraints.

When first reading this map, it is better to write together not only the names, but also what intuition broke first?

In this supplementary Section too, the names should not be listed alone. Even names inside the same reinforcement-learning family are trying to reduce different failure costs and different real-world constraints, so the problem signal must be written before the name.

Which Follow-Up Branch Should Be Looked Up First?¶

When later reinforcement-learning names look numerous, the branch should be chosen not by the name, but by what bottleneck is visible first right now?

So looking up later names is not the task of choosing which algorithm is more famous, but of classifying first is the bottleneck state representation, policy stabilization, real-world constraint, or human-preference reflection?

Why Is DQN Mentioned So Often Separately?¶

The Q-learning of P4-19.1 has strong intuition, but once states and actions become large, a table can no longer hold all the values. DQN appears at that point.

So DQN is a line that expresses the Q-value through a function approximator instead of a table, in order to handle larger state spaces.

That is why DQN is not a completely new philosophy, but a representative case of extending value-based reinforcement learning into a larger problem.

That flow can be written more tightly like this.

So DQN is better read not as a name that discarded value-based learning, but as a name that moved the value-based intuition into a larger state space.

Why Are PPO And The Actor-Critic Families Used So Widely?¶

The policy-based reinforcement learning of P4-19.2 has the advantage of adjusting the behavior directly, but the learning can easily become unstable. Names such as PPO, TRPO, A2C, and A3C are largely tied to that problem.

• We want the policy not to change too abruptly
• We want to reduce the fluctuation of the learning signal
• We want to use the division of roles between actor and critic more stably

So these names are later answers to the problem that policy-based reinforcement learning often shakes in practice.

This can be read more directly like this.

So although PPO, TRPO, A2C, and A3C differ in name, they share the direction keep the advantage of learning the policy directly, but do not let it collapse too unstably.

Why Did Safe RL And Offline RL Become Their Own Topics?¶

As P4-19.3 showed, exploration itself can be dangerous or expensive in reality. That is why two branches grew separately.

• safe RL: a line that tries to treat risk constraints more strictly even while exploration and policy improvement continue
• offline RL: a line that tries to learn the policy mostly from already collected data instead of making many new exploratory trials

Both came from the real-world problem that we cannot simply try anything we want.

It is useful to bundle them together while also leaving their difference.

So both arise from real-world constraints, but safe RL is closer to still try, but more safely, while offline RL is closer to try less, and learn first from existing data.

Why Is Domain Randomization Connected To Sim-To-Real?¶

One reason a policy learned well in simulation fails in reality is environmental difference. Domain randomization can be understood as a direction that intentionally shakes simulator conditions more widely, so that the policy becomes less brittle against real-world difference.

The core idea is this.

If reality cannot be copied perfectly, then do not train only on one overly clean simulation setting.

Made more concrete:

So domain randomization is better read not as a technique for maximizing simulator score, but as a reinforcement that prevents the learner from seeing only an overly clean world inside the simulator.

Why Is RLHF Important Here Too, And Why Does It Return Again In Part 5?¶

RLHF, reinforcement learning from human feedback, looks by name like one branch of reinforcement learning. But in the LLM context, it is not a direct copy of ordinary game or robot-control problems.

• In LLMs, it is often hard to turn human preference or evaluation criteria directly into correct labels
• So a flow appeared that converts human feedback into something reward-like and adjusts the policy from it
• At that point, reinforcement-learning language and alignment language meet

So RLHF is part of general reinforcement-learning thinking, but because it belongs to the separate context of LLM alignment, it must be read again in more detail in Part 5.

Placed side by side with general reinforcement-learning problems:

So RLHF should be read not as RL is also used for LLMs, but as a branch where the question of how human feedback should be interpreted like reward comes to the front.

Cases And Examples¶

Case 1. If DQN, PPO, And RLHF Appear Together, What Should Be Separated First?¶

When beginners look for reinforcement-learning material, they often encounter DQN in game-performance examples, PPO in policy-stabilization explanations, and RLHF in the context of LLM alignment almost side by side. A person first tends to group them with criteria such as reinforcement-learning names that appear often these days or stronger recent algorithms.

But that criterion quickly shows its limit. If the names are grouped only by recency, why DQN appeared, why PPO became widely used, and why RLHF must be read differently from ordinary control problems all become blurred. In reality, the three names do not appear together because they are all just improved reinforcement learning. They branched out as answers to different bottlenecks.

flowchart TD
  A["meet DQN, PPO, RLHF together"]
  B["ask what bottleneck appears first"]
  C["state space is too large"]
  D["policy updates are unstable"]
  E["human preference must guide outputs"]
  F["read as DQN branch"]
  G["read as PPO or actor-critic branch"]
  H["read as RLHF or preference branch"]

  A --> B
  B --> C --> F
  B --> D --> G
  B --> E --> H

If this case is compressed like a handoff memo, it can be written as follows.

The more names appear in the same place, the more important it becomes to record first not what does it compute?, but what problem came first? The visible result in this case is not I know DQN, PPO, and RLHF, but did the bottleneck question that should come first change for each name? That is exactly the purpose of this supplementary Section.

Practice And Example¶

This practice focuses on rereading later reinforcement-learning names not as memorizing algorithm names, but as bottlenecks and branches.

Problem situation:

• when names such as DQN, PPO, offline RL, and RLHF appear together, they can easily look like one bundle called better reinforcement learning

Input:

• four bottlenecks that appeared first
• candidate follow-up branches that respond to each bottleneck

Expected output:

• a matching table for which branch should be looked up first

Concept to check:

• later algorithm names should be read first by what bottleneck produced them, not by fame
• value-based expansion, policy stabilization, real-world constraint reinforcement, and LLM-alignment connection are different branches

The same judgment can also be checked with a tiny vector comparison. The code below gives each follow-up branch a score vector for the problem axes it is closest to, then uses cosine_similarity to find the branch closest to the observed bottleneck. This example is not trying to prove the research genealogy with these numbers. It is a small computational check for the habit of reading the bottleneck before the name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

axes = ["state_space", "policy_stability", "real_world_constraint", "human_preference"]
branches = pd.DataFrame(
    [
        [0.95, 0.10, 0.05, 0.05],
        [0.20, 0.95, 0.10, 0.05],
        [0.10, 0.25, 0.95, 0.05],
        [0.05, 0.25, 0.20, 0.95],
    ],
    index=["DQN", "PPO", "offline_RL", "RLHF"],
    columns=axes,
)

bottlenecks = pd.DataFrame(
    [
        [1.00, 0.10, 0.10, 0.00],
        [0.10, 1.00, 0.20, 0.10],
        [0.10, 0.20, 1.00, 0.10],
        [0.00, 0.20, 0.10, 1.00],
    ],
    index=[
        "huge_state_space",
        "unstable_policy_update",
        "cannot_explore_online",
        "human_preference_feedback",
    ],
    columns=axes,
)

scores = cosine_similarity(bottlenecks, branches)
score_table = pd.DataFrame(scores, index=bottlenecks.index, columns=branches.index)

for bottleneck, row in score_table.iterrows():
    best_branch = row.idxmax()
    print(bottleneck, "->", best_branch, "score=", round(row[best_branch], 3))

The output is as follows.

1
2
3
4

huge_state_space -> DQN score= 0.998
unstable_policy_update -> PPO score= 0.989
cannot_explore_online -> offline_RL score= 0.997
human_preference_feedback -> RLHF score= 0.992

This output does not mean we are building an automatic classifier where DQN is always the answer. It shows that if the bottleneck is written first along axes such as state space, policy stability, real-world constraint, and human preference, it becomes clearer which follow-up branch should be checked first.

Try Judging Directly¶

Look at the observations below and choose first which interpretation is safer.

The point of this table is not to memorize names. It is to build the habit of classifying immediately what bottleneck appeared first and which branch should be checked first because of it.

Checklist¶

• Can you explain that DQN is a line that extends value-based reinforcement learning into large state spaces, while PPO, TRPO, A2C, and A3C are lines that try to reduce instability in policy-based reinforcement learning?
• Can you explain that safe RL, offline RL, and domain randomization are branches that appeared because of real-world constraints and deployment risk?
• Can you explain that RLHF is a point where reinforcement learning meets LLM alignment?
• When several names appear at once, have you formed the habit of asking first what bottleneck produced this name??

Sources And References¶

• Richard S. Sutton, Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed., checked on 2026-07-01.
• Volodymyr Mnih et al., Human-level control through deep reinforcement learning, checked on 2026-07-01.
• John Schulman et al., Proximal Policy Optimization Algorithms, checked on 2026-07-01.
• Long Ouyang et al., Training language models to follow instructions with human feedback, checked on 2026-07-01.