P4-19.1 Value-Based Reinforcement Learning¶

Section ID: P4-19.1 Version: v2026.07.20

In P4-2.3, reinforcement learning was framed as learning that adjusts a policy through actions and rewards. If we go one step deeper, the following question appears.

When readers first meet reinforcement-learning algorithms, questions immediately follow.

• On what basis does the model learn which action is good in which state?
• Is good written like a rule, or like a number?
• If both Q-learning and SARSA are reinforcement learning, what is different between them?

Value-based reinforcement learning is an approach that learns by attaching values to how good each action is in the long run from each state.

This Section explains the basic meaning of value-based reinforcement learning, state value, action value, and Q-value. Later Sections continue the judgment in the current context from this handle, and the basic sense of reading the long-term goodness of actions as a score reconnects through this Section and the concept glossary.

Scope Of This Section¶

This Section answers the following questions.

• What does it mean to learn value?
• How are state value and action value different?
• Why is the Q-value important in reinforcement learning?
• In what way are Q-learning and SARSA similar, and in what way are they different?
• What kinds of problems fit value-based reinforcement learning well, and where do its limits begin to show?

This Section first closes the question of what structure reinforcement learning has when it attaches value to actions and learns their long-term goodness. Policy-based reinforcement learning continues in P4-19.2, reward design and cautions in real-world application continue in P4-19.3, the larger flow of DQN and policy-family methods continues in supplementary learning P4-19.4, and the minimum bridge to the Bellman equation and function approximation continues in supplementary learning P4-19.5.

Goals Of This Section¶

• You can explain value-based reinforcement learning as an approach that learns the long-term goodness of actions as numbers.
• You can distinguish state value from action value.
• You can say that Q-learning updates by using the action that looks best in the next state.
• You can say that SARSA updates by using the action that was actually chosen next.
• You can understand that the difference between the two algorithms connects to a difference in learning attitude.

Why Learn Value?¶

In reinforcement-learning problems, a correct label is not given every time. Instead, an agent tries an action, receives a reward, and experiences the next state.

At that point, instead of writing the policy first, there are several advantages in first writing down a number for how acceptable was this action?

• It becomes easier to compare actions.
• Even an unfinished policy can be improved gradually.
• Different action candidates in the same state can be read relative to one another.

In other words, value-based reinforcement learning is closer to scoring what is better first than to memorizing directly what to do.

If we borrow a restaurant recommendation analogy:

• Policy: a way of deciding immediately where to go in this neighborhood
• Value: an expected score for how satisfying that choice will be in the long run

That difference matters. It becomes easier to read value-based reinforcement learning if it is understood as making an expected scoreboard for actions before memorizing the action rule itself.

State Value And Action Value Are Different¶

In reinforcement-learning books and papers, value is usually split into two types rather than being mentioned only as one word.

Imagine a maze game.

• A cell right in front of the exit may have a high state value.
• But even in that cell, the action of moving toward a wall may have a low action value.

So the action value becomes especially important because even inside a good state, there can still be bad actions.

That difference can be read more concretely as follows.

So state value reads where am I, while action value reads what should I do from here. In reinforcement learning, action value is the more direct standard when actual choices are compared.

What Does The Q-Value Write Down?¶

Action value is usually written as the Q-value. Q(s, a) means the expected long-term return when taking action a in state s.

What matters here is the interpretation more than the formula.

The Q-value is an expected score for how acceptable the future path will be if this action is chosen in this state now.

So a Q-table or Q-function is an attempt to answer questions like these.

• Is going upward from here a good move?
• Is going downward better?
• Even if something looks costly now, does it lead to a bigger reward later?

In a small table, it can be drawn like this.

The key is that the Q-value is not just a number for one pleasant choice right now. Even if the immediate reward from moving one step is small, the Q-value can become high if that move leads to the exit later. On the other hand, even if there seems to be a small immediate gain, the Q-value can become low if it leads to a bigger penalty afterward.

The Basic Loop Of Value-Based Reinforcement Learning¶

The core of value-based reinforcement learning is acting, looking at the result, and slightly revising the value table in repetition.

flowchart TB
  A["1. observe state<br/>current state s"]
  B["2. choose action<br/>try action a"]
  C["3. receive reward<br/>reward r and next state s'"]
  D["4. estimate future value<br/>what looks good next?"]
  E["5. update Q-value<br/>revise Q(s, a)"]

  A --> B --> C --> D --> E --> A

This diagram lets value-based reinforcement learning be read as a loop that gradually corrects the scoreboard after seeing action outcomes. The key is not finishing the whole policy at once, but gradually adjusting state-action values while the loop runs.

This loop is one step more concrete than the general reinforcement-learning loop seen in P4-2.3. Here, the focus is on repeatedly refining value estimates such as Q(s, a) instead of changing the entire policy at once.

What Does Q-Learning Learn?¶

Q-learning is the best-known value-based reinforcement-learning algorithm. Its core idea is simple.

When arriving at the next state, update the value of the current action by using the action that looks best there.

So Q-learning updates not from what was actually done next, but from what looks like the best choice in the next state.

Because of this point, Q-learning is usually introduced as an off-policy algorithm.

A way of learning by looking slightly away from the actual behavior flow and using the best-looking choice in the next state

In a small maze example:

• Even if the agent moved downward because of exploration
• The update can still revise the current value by using the truly best action in the next state was right

So Q-learning reflects a somewhat optimistic path that assumes how well the agent could do in the future.

What Does SARSA Learn?¶

SARSA is also a value-based reinforcement-learning algorithm. Its name comes from the initials of state, action, reward, next state, and next action.

The main idea of SARSA is similar to Q-learning, but its standard is different.

Update the current action value by using the action that was actually chosen in the next state.

So SARSA reflects not the best-looking next action, but the action that I actually took next.

Because of that, SARSA is usually explained as an on-policy algorithm.

A way of learning inside the action pattern that I am actually following

For example, if exploration is still mixing in somewhat risky moves, SARSA learns from the actual path that includes that exploratory tendency.

The Difference Between Q-Learning And SARSA¶

Both algorithms update Q-values, but they differ in where the next value is taken from.

flowchart TB
  subgraph QL["Q-learning"]
    Q1["current state-action<br/>Q(s, a)"]
    Q2["next state s'"]
    Q3["use the best-looking next action<br/>max_a' Q(s', a')"]
    Q1 --> Q2 --> Q3
  end

  subgraph SA["SARSA"]
    S1["current state-action<br/>Q(s, a)"]
    S2["next state s'"]
    S3["use the actual next action<br/>Q(s', a_next)"]
    S1 --> S2 --> S3
  end

This diagram visually separates the difference between Q-learning and SARSA. Both look at the next state, but one corrects the value from the next action that looks best, while the other corrects it from the next action that was actually taken.

Compressed into a table:

Here, the feeling matters more than the terminology.

• Q-learning: reflects the theoretically best-looking next choice
• SARSA: reflects the next choice that was actually taken

If the same scene is compressed once more, the two algorithms read it differently like this.

Why Does This Difference Matter?¶

This difference creates a difference in interpretation especially in environments that include risky actions.

Imagine a cliff next to the maze with a large penalty.

• Q-learning tends to increase values in the direction of it is fine if the agent always moves optimally.
• SARSA can reflect more of the fact that during exploration, the agent may actually make mistakes.

So if the current action policy is not careful, SARSA learns that lack of care together with it. For that reason, introductory texts often say that SARSA can look more conservative.

If that difference is rewritten as an operational judgment sentence, it can be read like this.

When Should The Value-Based View Come First?¶

When reinforcement learning is first attached to a problem, the first question should be not the algorithm name, but should this problem be read as a state-action scoreboard?

Where Is It Used?¶

Value-based reinforcement learning gives especially clear intuition in problems where the number of states and actions is relatively clear and the result of each action can be tried repeatedly.

• Grid mazes and game movement
• Simple robot path finding
• Toy simulations of resource allocation
• Introductory examples of sequential choice problems

In practice, once the problem scale becomes large, a simple Q-table alone cannot cover the number of states. Then the flow moves toward function approximation, neural networks, and more complex policy-based methods. That bridge becomes important again in Part 4 and Part 5.

Its limits can be written more directly like this.

So value-based reinforcement learning is very good for building introductory intuition, but as real problems grow, it quickly runs into the limit of how should the value table be maintained?

Cases And Examples¶

Case 1. A Warehouse Robot Learns Through A Scoreboard Which Direction To Take At An Intersection¶

When a warehouse robot goes to pick up an item, the standard that is easiest for a person to write first is usually a simple rule such as the closest path now or the direction that moves one cell faster right away. But some routes may look short now while causing later congestion or larger turning cost, making them worse in the long run. Value-based reinforcement learning attaches a score to each location-action pair for how favorable is it from now on, so the robot gradually learns a better route. That is why, even at the same intersection, the choice that reflects later traffic flow rather than only immediate distance can gradually receive a higher Q-value.

For example, imagine that right before an intersection the robot accumulates experiences like the following.

The key shown by this table is that the action that looks shortest now and the action that is most favorable in the long run can be different. Value-based reinforcement learning is precisely an approach that tries to read that difference by accumulating numbers.

flowchart TD
  A["intersection state"]
  B["left / right / forward"]
  C["take one action"]
  D["observe delay or reward"]
  E["update Q-value for that action"]
  F["compare actions again next visit"]
  G["higher long-run route preference"]

  A --> B --> C --> D --> E --> F --> G

If this case is compressed like a review memo, it can be written as follows.

The observable result of this case appears when comparing whether straight looked faster at first but, after repeated experience, the detour route gets a higher Q-value, and whether the delay and collision pattern left by choosing straight keeps repeating. In other words, the value-based interpretation is not about which direction looked fast, but about which direction's long-term score rose more through actual experience.

Case 2. When A Fast Path And A Safe Path Split Beside A Cliff¶

Suppose an agent has two paths toward the goal. The standard that is easiest for a person to use first is usually the shortest path to arrival or the path where reward rises quickly right now. So a cliff-side corridor with a fast route and a large nearby penalty can look better at first. But in reinforcement learning, long-term reward including the chance of mistakes must be read together. Here, Q-learning can keep the value of the fast path relatively high because it reflects more strongly if the agent can move optimally in the next state. By contrast, SARSA reflects more directly the route where cliff-side mistakes are mixed in during exploration, so it may read the safer detour route as better.

The observable result of this case appears when comparing how often the cliff penalty is actually accumulated on the fast shortcut, whether the safe detour ends with a higher Q-value after repeated learning, and whether the preference gap between the two algorithms grows larger as the exploration rate stays high. So this case helps the reader interpret the algorithmic difference not through formulas, but through how directly is the possibility of mistakes reflected in the value estimate?

Case 3. When A Customer-Support Bot Learns Different Values Between Fast Closure And Safe Resolution¶

When a customer-support bot receives an inquiry, the easiest standard for a person to use first is usually an immediate efficiency metric such as average handling time or does it close quickly in one shot. So among close the response immediately, ask an additional question, and handoff to a human agent, the action that uses the least time right away can look best. But in reality, unresolved inquiries may come back again or customer dissatisfaction may grow. By contrast, handing off to a human agent may look costly immediately, yet help reduce repeat contacts and preserve satisfaction in the long run.

This scene makes clearer why the value-based view is necessary. Instead of writing the policy in sentences ahead of time, the system must learn as values how acceptable is each state-action pair in the long run.

This case shows that the Q-value is not just a speed score, but a device for reading how does the current action change the later state and reward flow?

The observable result of this case appears when comparing whether follow-up costs such as repeat contacts, dissatisfaction, and repeated human intervention keep growing in states where immediate closure is common, and whether actions such as asking more questions or handing off to a human actually create higher long-term reward. So the value-based view records as a score not does it close quickly now, but how does this action change the next state and later cost?

Practice And Example¶

This example focuses on confirming numerically that Q-learning and SARSA read the same experience a little differently.

Problem situation:

• Even from the same reward experience, Q-learning and SARSA can produce different update results because their rule for taking the next value is different

Input:

• Current state S0
• Current action right
• Immediate reward +1
• Next state S1
• Current values in the Q-table

Expected output:

• The update result computed by Q-learning
• The update result computed by SARSA
• The before-and-after values that make the difference between the two methods visible

Concepts to confirm:

• Q-learning corrects the current value by using the action value that looks best in the next state
• SARSA corrects the current value by using the action value of the action actually taken next
• Even the same experience can lead to different learning attitudes when the update rule is different

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# This example compares how Q-learning and SARSA update differently because they read the next value differently.
alpha = 0.5
gamma = 0.9

q_table = {
    ("S0", "right"): 0.40,
    ("S1", "up"): 0.80,
    ("S1", "down"): 0.30,
}

state = "S0"
action = "right"
reward = 1.0
next_state = "S1"
actual_next_action = "down"

old_value = q_table[(state, action)]

# Q-learning: use the largest value in the next state
best_next_value = max(
    q_table[(next_state, "up")],
    q_table[(next_state, "down")],
)
q_learning_target = reward + gamma * best_next_value
q_learning_updated = old_value + alpha * (q_learning_target - old_value)

# SARSA: use the value of the action actually taken next
actual_next_value = q_table[(next_state, actual_next_action)]
sarsa_target = reward + gamma * actual_next_value
sarsa_updated = old_value + alpha * (sarsa_target - old_value)

print("old Q(S0, right) =", round(old_value, 3))
print("Q-learning target =", round(q_learning_target, 3))
print("Q-learning updated =", round(q_learning_updated, 3))
print("SARSA target =", round(sarsa_target, 3))
print("SARSA updated =", round(sarsa_updated, 3))

The execution result can be read like this.

1
2
3
4
5

old Q(S0, right) = 0.4
Q-learning target = 1.72
Q-learning updated = 1.06
SARSA target = 1.27
SARSA updated = 0.835

The important point is that even though both algorithms started from the same current experience, the result changed because the rule for reading the next value was different.

• Q-learning used the value 0.8 of up, which looked best in S1.
• SARSA used the value 0.3 of down, which was the action actually chosen in S1.

That is why the Q-learning update rose more.

Try Changing One Value: If The Actual Next Action Changes, How Much Does The Update Attitude Change?¶

Now keep the current experience the same, but change only the actual next action in the next state to up.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# This example changes the actual next action in SARSA to see how the update gap between Q-learning and SARSA narrows.
alpha = 0.5
gamma = 0.9

q_table = {
    ("S0", "right"): 0.40,
    ("S1", "up"): 0.80,
    ("S1", "down"): 0.30,
}

state = "S0"
action = "right"
reward = 1.0
next_state = "S1"
actual_next_action = "up"

old_value = q_table[(state, action)]
best_next_value = max(
    q_table[(next_state, "up")],
    q_table[(next_state, "down")],
)
q_learning_target = reward + gamma * best_next_value
q_learning_updated = old_value + alpha * (q_learning_target - old_value)

actual_next_value = q_table[(next_state, actual_next_action)]
sarsa_target = reward + gamma * actual_next_value
sarsa_updated = old_value + alpha * (sarsa_target - old_value)

print("Q-learning updated =", round(q_learning_updated, 3))
print("SARSA updated =", round(sarsa_updated, 3))

1
2

Q-learning updated = 1.06
SARSA updated = 1.06

In the first run, the actual next action was down, so SARSA updated more conservatively. But once the actual next action is changed to up, SARSA produces the same value as Q-learning. In other words, the difference between the two methods is less about memorizing names than about how directly the update reflects what action the policy actually continues with.

This comparison exercise shows that even under the same reward, the learning result can change when the actual next action changes. So the evaluation criterion should include not only average reward, but also exploration cost, failure risk, and the policy's real action flow. The key is not the name of the Q-learning formula, but reading the reason the value changed as a difference in the update standard.

Judge For Yourself¶

Choose first which interpretation is safer for the observations below.

The purpose of this table is not to choose which algorithm is superior. It is to build the habit of reading first what update standard was operating even when the same numbers are shown.

This Section also should not leave value explanation alone. It should record together which state-action candidates and which failure costs were placed side by side. Even if two cases look like the same high Q-value, one state may repeat safely while another state may leave larger exploration cost and failure loss, so the review memo must remain together.

The core of this Section is not memorizing two algorithm names, but reading with what attitude the value table is updated.

Checklist¶

• Can you explain value-based reinforcement learning as an approach that learns the long-term goodness of states and actions as values?
• Can you distinguish state value and action value, and explain why action value is more direct for choosing an action?
• Can you explain the Q-value as an expected score for how acceptable the future will be if this action is taken in this state?
• Can you explain why Q-learning and SARSA can produce different values even from the same experience?
• Can you explain why the value-based view comes to mind first in problems where action candidates are few and comparable?
• Do you understand that a high Q-value does not immediately mean safety in real-world deployment?

Sources And References¶

• Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed., The MIT Press, 2018, checked on 2026-06-27. https://mitpress.mit.edu/9780262039246/reinforcement-learning/
• Christopher J. C. H. Watkins, Peter Dayan, Q-learning, Machine Learning, 1992, checked on 2026-06-27. https://link.springer.com/article/10.1007/BF00992698
• Satinder Singh, Tommi Jaakkola, Michael L. Littman, Csaba Szepesvari, Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms, Machine Learning, 2000, checked on 2026-06-27. https://link.springer.com/article/10.1023/A:1022689125041