P4-19.6 Supplementary Learning: First Reading of Policy Gradient¶

Section ID: P4-19.6 Version: v2026.07.24

Subtitle: How does the likelihood ratio trick connect changes in policy probability to expected reward?

Once readers study policy-based reinforcement learning in P4-19.2, the following names quickly appear.

• policy gradient theorem
• likelihood ratio trick

Rather than following the full rigorous proof all the way to the end, this supplementary Section first reads why a change in policy parameters connects to a change in expected reward and why the form of log-probability appears so often.

Scope Of This Section¶

This Section answers the following questions.

• Why is policy gradient read as directly adjusting policy probabilities?
• Why does the likelihood ratio trick connect log-probability and expected-value calculation?
• How does this sense of formulas continue into the interpretation of REINFORCE and actor-critic?

This Section focuses on building an introductory feeling for policy-based formulas through three handles: policy probability, expected reward, and log-probability gradient.

Goals Of This Section¶

• You can explain policy gradient as a gradient that adjusts policy probabilities in the direction that increases expected reward.
• You can explain the likelihood ratio trick as a device that turns differentiation inside a probability distribution into a log-probability gradient so the calculation becomes easier to read.
• You can describe why forms such as log pi(a|s) appear in REINFORCE and actor-critic.

Why This Section Is Needed¶

Policy-based reinforcement learning is easy to follow as intuitive sentences, but once the formulas appear, it suddenly feels unfamiliar.

• Why does a log appear when differentiating expected reward?
• Why is the gradient of an action probability multiplied by reward?

That is exactly where the likelihood ratio trick appears.

So the core of this Section is to connect for the first time how the sentence directly adjust the policy becomes the formula reading gradient of the log-probability.

What Is Policy Gradient Differentiating?¶

Policy-based reinforcement learning ultimately carries the following question.

If the policy parameters change slightly, in what direction does long-term expected reward become larger?

So, in a very short sentence, policy gradient is:

the gradient for moving policy parameters in the direction that increases expected reward

At an introductory level, it is enough to hold on to the following.

So policy gradient is best read as an update that directly fine-tunes the policy distribution.

Why Does The Likelihood Ratio Trick Appear?¶

When a probability distribution sits inside an expectation, differentiation becomes awkward. The likelihood ratio trick is a standard transformation used at that point.

The core feeling is captured by one sentence.

Instead of differentiating the probability directly, change it into the gradient of the log-probability, and the expression becomes easier to read inside the expectation.

If only the form is written very briefly, it looks like this.

1

grad p(x) = p(x) * grad log p(x)

The role of this expression is to make a derivative of a probability distribution inside an expectation easier to connect to sample-based updates.

So the reason a log appears is not decoration. It is to change the computational structure itself.

Then How Should The REINFORCE Formula Be Read?¶

The REINFORCE intuition is usually read in a form like this.

Strengthen the log-probability gradient of actions that gave good reward, and weaken the gradient of actions that gave bad reward.

At an introductory level, the following comparison is the most important.

At this point, the likelihood ratio trick is the bridge that explains why such an update is written in a log-probability form.

How Does This Continue Into Actor-Critic?¶

In P4-19.2, actor-critic was introduced as a structure that directly adjusts the policy while the critic provides a more stable evaluation signal. If this is read through formula intuition, it looks like this.

• actor: still adjusts by following the gradient of the policy's log-probability
• critic: provides a less noisy evaluation signal for how good an action was

So actor-critic does not abandon policy gradient. It can be read as a direction that makes the evaluation signal multiplied onto that gradient more stable.

Cases And Examples¶

Case 1. When You Want Good Ad-Display Ratios To Appear More Often And Bad Ones Less Often¶

Suppose an ad-display policy chooses ratios probabilistically, such as discount banner 70% / recommendation banner 30%. The first human-friendly rule is usually something like keep using the banner mix that worked well this time more often next time.

That intuition points in the right direction, but it quickly becomes blurry once formulas appear. Questions like why are we changing probabilities directly?, why does log-probability appear?, and why can the same action move in opposite directions depending on the reward sign? appear immediately. Policy gradient and the likelihood ratio trick connect that point by turning the intuition raise good probabilities and lower bad ones into a readable formula.

flowchart TD
  A["Observe a good banner mix"]
  B["Want to use it more often"]
  C["Meet log pi(a|s) in the formula"]
  D["Read the reward sign together"]
  E["Positive reward strengthens the probability"]
  F["Negative reward weakens the probability"]
  G["Read policy intuition through the formula"]

  A --> B --> C --> D
  D --> E
  D --> F
  E --> G
  F --> G

The checkable result of this case is whether the sentence good ratios should appear more often, and bad ratios less often can now be reread through three handles: expected reward, log-probability gradient, and reward sign. In other words, the formulas should not erase the policy-based intuition. They should make that intuition more precise.

Practice And Example¶

This exercise focuses on reading directly, with small numbers, good reward -> strengthen the probability of the chosen action, bad reward -> weaken it, and why log-probability appears together.

Problem situation:

• policy gradient and the likelihood ratio trick may sound abstract by name, but in practice they are devices for calculating which direction the probability of the chosen action should be pushed

Input:

• policy logits for two actions
• the selected action
• a positive reward and a negative reward for the same action

Expected output:

• action probabilities before the update
• grad log pi(a|s) for the selected action
• action probabilities after the update, changed by the reward sign

Concepts to check:

• log-probability is a reading device that connects the chosen-action probability to the update calculation
• if the reward sign changes, the adjustment direction for the same action changes too

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import numpy as np


def softmax(logits):
    shifted = logits - np.max(logits)
    exp_values = np.exp(shifted)
    return exp_values / exp_values.sum()


# Two actions: show_discount_banner, show_recommendation_banner
logits = np.array([0.2, -0.2])
chosen_action = 0
learning_rate = 0.4

for reward in [2.0, -2.0]:
    probs_before = softmax(logits)
    grad_log_prob = -probs_before
    grad_log_prob[chosen_action] += 1.0
    logits_after = logits + learning_rate * reward * grad_log_prob
    probs_after = softmax(logits_after)

    print("reward=", reward)
    print("prob_before=", [round(float(v), 3) for v in probs_before])
    print("grad_log_prob=", [round(float(v), 3) for v in grad_log_prob])
    print("prob_after=", [round(float(v), 3) for v in probs_after])

An example result can be read like this.

1
2
3
4
5
6
7
8

reward= 2.0
prob_before= [0.599, 0.401]
grad_log_prob= [0.401, -0.401]
prob_after= [0.739, 0.261]
reward= -2.0
prob_before= [0.599, 0.401]
grad_log_prob= [0.401, -0.401]
prob_after= [0.44, 0.56]

The key point of this example is not to memorize the signs themselves.

1. grad log pi(a|s) is the reading handle that connects the chosen-action probability to the update calculation.
2. With a positive reward, the probability of the selected action rises from 0.599 -> 0.739.
3. With a negative reward, the same gradient makes the probability of the selected action fall from 0.599 -> 0.44.
4. The likelihood ratio trick changes probability differentiation into log-probability gradient so that this connection becomes easier to read.

Judge It Directly¶

Read the observations below and choose which interpretation is safer first.

The purpose of this table is not to prove the formula. It is to hold on to interpretation: how policy probability is pushed and pulled, and why log-probability is needed for that connection.

Checklist¶

• Can you explain that policy gradient is the gradient that moves policy parameters in the direction of increasing expected reward?
• Can you explain that the likelihood ratio trick makes probability differentiation easier to read by turning it into a log-probability gradient?
• Can you explain that both REINFORCE and actor-critic can be read on top of this same log-probability gradient intuition?
• Can you explain that positive reward and negative reward make the probability of the chosen action move in different directions?

Sources And References¶

• Ronald J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning, 1992. Consulted for the REINFORCE family of algorithms and the gradient-following view of expected reinforcement. Accessed 2026-07-19. https://doi.org/10.1007/BF00992696
• Richard S. Sutton, David McAllester, Satinder Singh, Yishay Mansour, Policy Gradient Methods for Reinforcement Learning with Function Approximation, NeurIPS 1999. Consulted for the policy gradient theorem and the connection between approximate value/advantage functions and experience-based estimation. Accessed 2026-07-19. https://papers.nips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html
• Vijay R. Konda, John N. Tsitsiklis, On Actor-Critic Algorithms, SIAM Journal on Control and Optimization, 2003. Consulted for connecting actor-critic to stabilization of evaluation signals in policy-gradient methods. Accessed 2026-07-19. https://doi.org/10.1137/S0363012901385691