P5-8.4 Supplementary Reading: How A Large Initialization Scale Shakes The Computation Range¶

Section ID: P5-8.4 Version: v2026.07.20

In P5-8.3, we grouped together the conditions that make deep computation actually shake less: initialization, numerical stability, and batch normalization. Now we confirm that claim with actual numbers.

The core question is simple.

When the same activations pass through several layers, how much does a large initialization scale actually amplify the values?

This section does not reproduce full learning. Instead, it assumes that the activation values coming from the previous layer pass through the same linear transform across three layers, and checks how a small scale and a large scale change the output range and variance at each layer. The single scalar multiplied at each layer is enough to read as a toy experimental value that represents the weight scale of that layer in simplified form. Then we also check how the output range is reorganized into an easier-to-handle form once batch normalization is applied after each layer.

The Question Of How Large Initialization Scale Shakes Ranges¶

• How does a large initialization scale increase the range and variance of raw activations during deep repeated computation?
• If batch normalization is applied after each layer, how does the same computational flow change?
• Even though this example does not stand in for a full neural network, how does it help us hold onto the intuition of computational stabilization?

As the final stage of Chapter 8, this section is the place where we confirm with numbers the computation-stabilization devices gathered in the previous section. That is also why we do not discuss optimizer updates, loss reduction, or actual dataset training performance here. Those topics are passed to the learning-loop explanation in P5-6.1 and the optimizer explanations in P5-7.1 and P5-7.2.

How To Read The Example¶

The input in the example below is not an arbitrary table of numbers, but activation values assumed to have come from the previous layer for three samples.

The weight scale is divided into four cases.

The important point here is not the large values mean richer representation. It is that when the same pattern is repeated through deep layers, the range of values received by the next layer and the gradient path can shake together.

It is better not to run this example only once and stop, but to change the values below and see which outputs become sensitive first.

One more point should be fixed first here as well. The batch-normalization results below do not mean it is fine even if initialization becomes arbitrarily large. This toy experiment separates out only what kind of distribution reorganization appears when we change only the scale while keeping the same sample shape. So it is safer to read batch normalization as a device that reorganizes a distribution that has already appeared into a range that is easier to handle, not as something that replaces the starting-point problem handled by initialization.

Python Example¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

# This example reads a CSV activation log and compares whether initialization scale expands raw range and variance in deep layers and whether batch normalization reorganizes the range.
from csv import DictReader
from pathlib import Path


def batch_norm(values, eps=1e-5):
    mean = sum(values) / len(values)
    variance = sum((v - mean) ** 2 for v in values) / len(values)
    normalized = [(v - mean) / ((variance + eps) ** 0.5) for v in values]
    return mean, variance, normalized

csv_path = Path("docs/assets/part-05/chapter-08/deep-scale-activation-log.csv")

rows = []
with csv_path.open(encoding="utf-8") as file:
    for row in DictReader(file):
        rows.append(
            {
                "case_name": row["case_name"],
                "weight_scale": float(row["weight_scale"]),
                "layer": int(row["layer"]),
                "sample": row["sample"],
                "raw_activation": float(row["raw_activation"]),
            }
        )

case_order = ["small_init", "medium_init", "large_init", "very_large_init"]
layer_order = [1, 2, 3]

for case_name in case_order:
    case_rows = [row for row in rows if row["case_name"] == case_name]
    weight = case_rows[0]["weight_scale"]
    print(f"[{case_name}] weight = {weight}")

    for layer in layer_order:
        layer_rows = [
            row for row in case_rows
            if row["layer"] == layer
        ]
        raw_values = [row["raw_activation"] for row in layer_rows]
        _, raw_variance, bn_values = batch_norm(raw_values)

        print(
            f"layer {layer}: "
            f"raw_range=({min(raw_values):.3f}, {max(raw_values):.3f}), "
            f"raw_variance={raw_variance:.3f}, "
            f"after_bn_range=({min(bn_values):.3f}, {max(bn_values):.3f})"
        )
    print("---")

This code reads the 36 values recorded in deep-scale-activation-log.csv and recalculates the range and variance by case and layer. It does not implement an entire real neural-network layer; it is a compressed experiment that looks only at where the weight scale pushes value range and variance as it passes through layers. So what we should look at first is not exact training performance, but the direction of repeated computation and scale accumulation.

An example output looks like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[small_init] weight = 0.8
layer 1: raw_range=(0.400, 1.600), raw_variance=0.249, after_bn_range=(-1.069, 1.336)
layer 2: raw_range=(0.320, 1.280), raw_variance=0.159, after_bn_range=(-1.069, 1.336)
layer 3: raw_range=(0.256, 1.024), raw_variance=0.102, after_bn_range=(-1.069, 1.336)
---
[medium_init] weight = 1.2
layer 1: raw_range=(0.600, 2.400), raw_variance=0.560, after_bn_range=(-1.069, 1.336)
layer 2: raw_range=(0.720, 2.880), raw_variance=0.806, after_bn_range=(-1.069, 1.336)
layer 3: raw_range=(0.864, 3.456), raw_variance=1.161, after_bn_range=(-1.069, 1.336)
---
[large_init] weight = 3.0
layer 1: raw_range=(1.500, 6.000), raw_variance=3.500, after_bn_range=(-1.069, 1.336)
layer 2: raw_range=(4.500, 18.000), raw_variance=31.500, after_bn_range=(-1.069, 1.336)
layer 3: raw_range=(13.500, 54.000), raw_variance=283.500, after_bn_range=(-1.069, 1.336)
---
[very_large_init] weight = 9.0
layer 1: raw_range=(4.500, 18.000), raw_variance=31.500, after_bn_range=(-1.069, 1.336)
layer 2: raw_range=(40.500, 162.000), raw_variance=2551.500, after_bn_range=(-1.069, 1.336)
layer 3: raw_range=(364.500, 1458.000), raw_variance=206671.500, after_bn_range=(-1.069, 1.336)
---

The first thing to look at in this output is the very_large_init numbers. In the first layer the raw range is (4.500, 18.000), but by the third layer it has grown to (364.500, 1458.000). The same input pattern was simply passed through repeatedly, yet with a large scale the value range truly widens dramatically as the network gets deeper.

For beginners, it is better not to leave the example output as a block of numbers, but to read it back through the following three lines.

Reading It Through Graphs¶

The first graph shows the raw activation range at each layer. large_init and very_large_init spread the value range widely between the same samples as the network becomes deeper.

The second graph compresses the same phenomenon into the variance. Because variance shows the spread of values as a single number, it becomes easier to read how quickly a large scale can create an unstable range in deep layers.

The third graph shows the output range after applying batch normalization behind each layer. Raw activations differ greatly by case, but after normalization the range is reorganized into a similar scale that is easier for the next layer to handle. The important point here is not the numbers are always fixed to exactly the same values, but rather even if the input distribution differs greatly, the center and spread are brought back into a comparable range. In this toy example, the ranges across cases look almost identical because the relative shape of the three samples is preserved while only the scale is changed, so it would be too strong to read this figure alone as saying batch normalization almost erases the initialization problem. The graph is also drawn as a point comparison by case, rather than as a line graph for tracking change, so readers first notice that the values gather into nearly the same range.

The Conclusion We Should Read¶

Therefore initialization and batch normalization are not devices that solve the same problem in the same place. Initialization sets the starting scale, and batch normalization reorganizes the activation distribution that has already appeared between layers. Numerical stability is the common question that explains why the two are discussed together in deep networks.

The conclusion this experiment should let us hold onto first is simple. Its role is to let us see with our own eyes how quickly the raw activation range and variance become unstable as the weight scale grows, and why batch normalization pulls the scale received by the next layer back into a comparable range.

Checklist¶

• Can you explain that a large initialization scale can actually enlarge the raw activation range as values pass through layers?
• Can you explain that raw variance can serve as a simple observation value for reading instability in deep repeated computation?
• Can you explain that batch normalization is not removing large values, but a device for resetting the standard of the distribution?
• Can you distinguish that this example is not a full training process, but a small experiment for confirming the intuition of numerical stability?

Sources And References¶

• Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola, Dive into Deep Learning, 5.4 Numerical Stability and Initialization, 8.5 Batch Normalization, checked on 2026-07-14. https://d2l.ai/
• Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning, Part II Modern Practical Deep Networks, checked on 2026-07-14. https://www.deeplearningbook.org/