P5-14.2 What Does Each Of The Four Transformer Block Components Do?¶

Section ID: P5-14.2 Version: v2026.07.20

In P5-14.1, we saw that explaining the Transformer only with self-attention is not enough. Now we need to separate the roles inside the block more directly.

Inside a Transformer block, what does each of self-attention, feed-forward network, residual connection, and layer normalization do?

The core is not memorizing component names, but dividing roles. To read the path that the same token representation follows inside the block, we need to distinguish relationship reading, position-wise processing, preserving original information, and stabilizing the value range.

Questions Handled By Block Role Division¶

• What does self-attention handle?
• How is the feed-forward network different from attention?
• Why do residual connection and layer normalization appear together?

Self-Attention Reads Relationships¶

As we saw in Chapter 13, self-attention is a method in which each token refers to other tokens in the same sequence and recalculates its own representation.

Self-attention is the device that decides where inside the sentence this token should look more strongly in order to be understood now.

The core is relationship reading. It makes the current token bring in the context it needs from other positions in the sentence.

Feed-Forward Network Processes The Current Position Representation¶

Self-attention mixes token relationships, but that result is not automatically a sufficiently good representation. The feed-forward network reprocesses the current position representation, after context has been mixed in, in a nonlinear way.

If attention mixes context by reflecting relationships with other tokens, feed-forward reprocesses each position representation into something richer.

Here the phrase each position is important. Self-attention is closer to deciding what information the current position should bring from other positions. The feed-forward network changes that mixed representation inside the same position. The same feed-forward network is applied to each token position, but because the input representation is different, the meaning refined at each position is also different.

flowchart LR
  A1["position 1<br/>context-mixed representation"] --> B1["same FFN<br/>within-position transform"] --> C1["position 1<br/>updated representation"]
  A2["position 2<br/>context-mixed representation"] --> B2["same FFN<br/>within-position transform"] --> C2["position 2<br/>updated representation"]
  A3["position 3<br/>context-mixed representation"] --> B3["same FFN<br/>within-position transform"] --> C3["position 3<br/>updated representation"]

  B1 -. "shared weights" .- B2
  B2 -. "shared weights" .- B3

In this diagram, each row means a different token position. The dotted lines mean that the same feed-forward network weights are shared across positions, and the solid lines mean that each position representation is processed separately inside its own position. So the feed-forward network is not a device that chooses a new token to refer to, but a device that changes an already context-mixed representation into the next representation for that position.

In a work-permit sentence, for example, attention makes the restart position look together at pressure unreleased and hold. The feed-forward network then processes that result inside the current position, so that the restart representation is read not as a simple action name, but as an action that must be blocked because a condition is attached.

The difference is clearer if we focus on one token.

So the feed-forward network should not be read as simple post-processing. If attention opens what should be seen together, feed-forward handles how the mixed representation should become the next representation of the current position. This distinction is needed to read the Transformer block not as attention alone, but as a repeating unit with divided roles.

Why a feed-forward network can apply the same weights to several positions while still producing different representations at each position is separated into P5-14.6 Supplementary Reading: Why Does The Feed-Forward Network Handle Position-Wise Representation Processing?.

Residual Connection Leaves The Original Information Flow¶

In a deep neural network, repeated new computations can overwrite the original information too strongly or make learning unstable. A residual connection passes the previous representation together with the new computation result, preserving the original information flow.

A safety device that does not trust only the completely new computation, but also leaves the original input representation and sends it to the next stage together.

The important point is that residual connection does not remove the new computation. The new representations made by self-attention or feed-forward are still needed. But if only that new representation is passed to the next stage, the basic meaning that the original token had can be covered too easily. So residual connection should be read as a path that leaves new computation result + original input representation together.

flowchart TD
  A["input representation<br/>original token meaning"]
  B["new calculation<br/>attention or feed-forward output"]
  C["add<br/>original representation + new calculation"]
  D["next-stage representation<br/>new meaning and original axis remain together"]

  A --> B
  B --> C
  A -. "skip path" .-> C
  C --> D

In this diagram, the solid path is the representation made by the new computation, and the dotted path is the bypass route where the original input representation is added. The core of residual connection is not blocking the new computation, but letting the new computation and the original representation move together to the next stage.

If feed-forward handles how should the current representation change, residual connection handles how can the changed representation avoid completely losing its original starting point. If layer normalization organizes the value range, residual connection is closer to leaving a bypass path through which information can travel.

This distinction prevents residual connection from being reduced to mere addition. A more accurate intuition is a device that leaves a path for original information even when new computation enters, so deep block repetition can endure.

Why residual connection is not just a skip, but a path that passes original representation and new computation together, is separated into P5-14.7 Supplementary Reading: Why Does Residual Connection Leave A Path For The Original Representation?.

Layer Normalization Organizes The Value Range¶

When many layers and large matrix operations repeat, the size and distribution of representation values can shake. Layer normalization organizes each position representation into a range that is easier to handle, helping the next computation shake less.

Layer normalization is a device that organizes the size and distribution of representation values so the next computation can continue stably.

Here, organizing does not mean judging meaning anew. Representations made by self-attention and feed-forward consist of several numeric axes, and once residual connection is added, some axes can become too large while others become relatively small. If the value scale keeps spreading, the next attention or feed-forward step has to compute from a difficult range even when it receives a similar kind of input.

Layer normalization resets the mean and spread of values inside one position representation, so the next component can start from a similar baseline. At the introductory level, it is enough to read it as follows instead of memorizing formulas.

flowchart LR
  A["one-position representation<br/>uneven value scale"]
  B["layer normalization<br/>recenter and rescale"]
  C["normalized representation<br/>stable next input"]

  A --> B
  B --> C

The important point in this diagram is that layer normalization does not choose new relationships among tokens or add a path for preserving the original information. It adjusts the value range inside one position representation so the next self-attention or feed-forward step does not receive an overly shaky input.

So residual connection and layer normalization often appear together inside the Transformer block, but they do not do the same job. If residual connection leaves a path for information, layer normalization aligns the computational baseline so the representation that passed through that path does not shake too much in the next computation.

Why layer normalization is a value-baseline adjustment inside one position representation, not meaning selection, and how it differs from batch normalization, is separated into P5-14.8 Supplementary Reading: Why Does Layer Normalization Align The Value Baseline?.

If we bundle the four components at once, the questions asked about the same token representation are different.

If We Draw It Very Simply¶

flowchart TD
  A["input token representation"]
  B["self-attention<br/>read relationships with other positions"]
  C["residual + normalization<br/>add the original representation and organize scale"]
  D["feed-forward network<br/>process the current-position representation"]
  E["residual + normalization<br/>pass the processed result stably"]
  F["updated representation for the next block"]

  A --> B
  B --> C
  C --> D
  D --> E
  E --> F

This diagram compresses one Transformer block at an introductory level. Read the flow as relationship reading -> add and organize the original representation -> process the current position representation -> pass it onward stably again.

Cases And Examples¶

Case. Reading The Restart Position Representation Through The Four Components¶

Consider the work-permit sentence Restart is held while the pressure remains unreleased. The current position of interest is restart. If a person reads only the word quickly, it is easy to read restart as a simple execution action. But the sentence also contains the condition pressure unreleased and the judgment held.

The judgment that is too easy to make is the word restart appeared, so this is an execution request. If we read through the Transformer block by role, this judgment does not change all at once. The same position representation answers different questions as it passes through several components.

If we first split the input by token position, it can be read as follows.

If we isolate only the restart position in this table, it looks like an execution action. But the central axis of this section is not what restart finally means; it is how that position representation changes while passing through Transformer block components.

At the self-attention stage, the restart position refers together to pressure unreleased and held. The result to confirm here is that restart no longer stays as an isolated execution word, but sees the earlier condition and the later judgment together. But at this point the current position representation has not yet been fully organized as a conditionally blocked action.

At the feed-forward network stage, the context mixed by attention is processed again inside the current position. The restart representation becomes clearer as an action that should be held because of the pressure condition, rather than as a simple action name. The core of this stage is not choosing a new word to refer to, but changing the already imported context into the current position representation.

At the residual connection stage, only the newly processed blocking meaning is not left behind. The original action axis of restart must also remain so that later blocks do not lose what is being held. So residual connection is not a path that skips new computation, but a path that passes the new computation result and the original input representation together.

At the layer normalization stage, meaning is not selected again. The current representation, whose several numeric axes have been mixed after attention, feed-forward, and residual, is organized into a baseline the next block can handle. Aligning the value range and judging meaning are not the same job.

If we narrow the flow to only the change in the restart position representation, it becomes the following.

So the output of this case is not just the conclusion restart is held. The more important output is the distinction that relationship reading, position-wise processing, preserving original information, and stabilizing value range are jobs of different components. Once this distinction is fixed, the Transformer block can be read not as a device where attention gives the answer, but as a representation-update unit where several roles repeat.

If we close the same case from the output viewpoint, it becomes the following.

The result to confirm in this case is that the restart representation does not change all at once as if by magic. Self-attention reads relationships, feed-forward processes the current position representation, residual connection leaves the original information flow, and layer normalization aligns the value baseline for the next computation. Only when these four questions are separated can the Transformer block be read not as attention alone, but as a repeating unit with divided roles.

Practice And Example¶

Example. Following The Representation Movement Of An Action Token With Numbers¶

If we shrink the same role distinction into another operations-log scene, we can directly see that the same action token moves in a different direction when its attention row changes. Here we follow only input -> after attention -> after feed-forward -> after residual, without calculating layer normalization. Value-range organization is handled separately in the next stabilization section.

When reading the code, do not try to memorize the whole matrix at once. First look only at how strongly the action token refers to other cues.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

# This example compares how the action-token representation moves through attention, feed-forward, and residual depending on whether rollback is confirmed.
import numpy as np

tokens = np.array([
    [1.0, 0.2],   # symptom token: urgency high
    [0.8, 0.5],   # deploy clue token: cause evidence medium
    [0.3, 1.0],   # action token: recovery status important
])

attention_cases = {
    "rollback_confirmed": np.array([
        [0.6, 0.3, 0.1],
        [0.2, 0.5, 0.3],
        [0.1, 0.3, 0.6],
    ]),
    "rollback_not_confirmed": np.array([
        [0.6, 0.3, 0.1],
        [0.3, 0.5, 0.2],
        [0.3, 0.5, 0.2],
    ]),
}

ff_weights = np.array([
    [1.1, 0.4],
    [0.2, 1.0],
])

for name, attention_weights in attention_cases.items():
    contextual = attention_weights @ tokens
    ff_output = contextual @ ff_weights
    residual_added = ff_output + tokens
    action_trace = [
        ("input", tokens[2]),
        ("after attention", contextual[2]),
        ("after feed-forward", ff_output[2]),
        ("after residual", residual_added[2]),
    ]

    print(f"[{name}]")
    print("action attention row =", np.round(attention_weights[2], 3))
    print("action token stage trace")
    for stage, values in action_trace:
        print(f"{stage:24s}", np.round(values, 3))
    print("---")

Read the sample output like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[rollback_confirmed]
action attention row = [0.1 0.3 0.6]
action token stage trace
input                    [0.3 1. ]
after attention          [0.52 0.77]
after feed-forward       [0.726 0.978]
after residual           [1.026 1.978]
---
[rollback_not_confirmed]
action attention row = [0.3 0.5 0.2]
action token stage trace
input                    [0.3 1. ]
after attention          [0.76 0.51]
after feed-forward       [0.938 0.814]
after residual           [1.238 1.814]
---

Explanation: the two scenes start from the same input tokens, but the action token's attention row changes, so the representation movement path also changes. In rollback_confirmed, the recovery-status axis remains larger after attention. In rollback_not_confirmed, the symptom/cause axis remains relatively larger. This difference remains as the block output direction after feed-forward and residual.

To check it yourself, change the action-token row of rollback_not_confirmed from [0.3, 0.5, 0.2] to something like [0.2, 0.4, 0.4] while keeping the sum at 1. You can compare how the recovery-status axis changes after after attention as the action token refers to itself more strongly.

Practice. Naming The Role¶

Judge which component the description below is most directly connected to.

Explanation: The core of this practice is not memorizing component names, but distinguishing that even inside the same block, what to refer to, how to change the current representation, how to leave the original information, and how to stabilize computation are different questions.

Practice. Fix Misleading Sentences¶

The sentences below are only partly right or mix roles. Rewrite them more accurately.

Explanation: The reason for correcting these sentences is to create a sense of boundaries, not to memorize terms. All four components are inside the same representation flow, but if relationship selection, within-position transformation, information preservation, and value stabilization are mixed together, the computational flow becomes blurry when reading the Transformer structure later.

Checklist¶

• Can you explain the role difference between self-attention and feed-forward?
• Can you explain residual connection as a device that leaves the original information flow?
• Can you explain layer normalization as a stabilization device for deep block repetition?
• Can you explain what kind of representation movement after attention, after feed-forward, and after residual show in the action-token stage trace?

Sources And References¶

• Ashish Vaswani et al., Attention Is All You Need, NeurIPS 2017, checked on 2026-07-19. https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
• Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning, MIT Press, 2016, checked on 2026-06-29. https://www.deeplearningbook.org/