P5-4.1 Loss Functions¶

Section ID: P5-4.1 Version: v2026.07.20

In Chapter 3, we saw that activation functions insert nonlinearity into neural networks and increase expressive power. The next question then follows immediately.

How, then, does a neural network judge how wrong its current output is?

The standard that answers this question is the loss function.

A loss function is the rule that turns how far the model's current output differs from the target into one number.

However, textbooks and framework documents sometimes separate loss from objective or cost a little more carefully. Often, loss refers to the per-sample discrepancy or its average, while the full target that is actually minimized is explained as an objective/cost that includes the batch average together with regularization. In this section, for beginner flow, everything is first explained under the single name loss, but when gradients and the optimizer are connected in later sections, we return to the question what is actually being minimized?

When the role of loss needs to be checked again briefly in later sections, return to the loss function entry in the concept glossary.

Here it is enough to fix the following three sentences first.

• Learning has not started yet if we only look at the output.
• The mistake must be turned into a number before the next calculation can continue.
• That number is exactly the loss.

The Question That Needs Loss Functions¶

• What does the loss turn into a number?
• Why does the loss function become the central standard in deep learning?
• How is the loss function different from an evaluation metric?
• What does it mean in learning when the loss value becomes smaller?

The concrete kinds of loss functions and their differences by problem type continue in P5-4.2, while how differentiation and gradient calculation actually connect to backpropagation returns in P5-5.1 and P5-5.2. In other words, this section is the place to first hold onto the role of loss as turning the gap between prediction and target into a number that learning can use.

Standards For Reading Error As A Number¶

• You can explain the loss function as the standard that turns the current prediction error into a number.
• You can distinguish the loss function from an evaluation metric.
• You can explain what it means in learning to reduce the loss.
• You can explain which wrong answers the difference between prediction and target causes the model to correct more strongly first.

Why Is A Loss Function Needed¶

Neural networks produce outputs. But output alone does not yet make learning happen. The model has to be able to ask itself:

How bad is the current result?

If that question is not turned into a number, the next step is impossible.

• The model does not know in which direction the weights should change.
• It does not know whether the current model is better than the previous one.
• It becomes hard to compare which output candidate is worse than another.

In other words, the loss function is the reference point by which the neural network checks its own result.

This scene can be restated as follows.

Without loss, the model cannot compare for itself whether things have become better or worse.

What Does Loss Measure¶

Loss usually compares the following two things.

• the current output made by the model
• the target we expected

It is enough to understand it like this.

Loss is a number that compresses the gap between prediction and target.

For example:

• if the correct answer is 10 and the prediction is 9, the loss may be relatively small
• if the correct answer is 10 and the prediction is 2, the loss may be larger

In other words, loss does not stop at simply right or wrong. It lets us read how far it is off through a more continuous number.

An important feeling readers often gain here is the following.

• whether it is classification or regression, loss deals with the degree of wrongness
• loss usually gives a smoother number than a hard judgment such as 0 or 1
• and precisely because of that, gradients can later be calculated

If this flow is compressed very briefly, it becomes the following.

flowchart TD
  A["Model prediction"]
  B["Target"]
  C["Compare the gap"]
  D["Loss value"]
  E["Gradient signal"]
  F["Parameter update"]

  A --> C
  B --> C
  C --> D
  D --> E
  E --> F

The result to confirm first in this diagram is that loss is not just a bad score, but the connection point that passes the difference between prediction and target into the next update.

How Is It Different From The Metric In Part 4¶

In Part 4, we looked at evaluation metrics such as accuracy, precision, recall, F1, and RMSE. At this point, it is easy to understand the loss function and the evaluation metric as though they were the same thing.

But the two have different roles.

So it is better first to distinguish that the loss function is the standard by which the model learns, while the metric is the standard by which the model is evaluated and selected.

Of course, the two are not completely separate. A good loss design should usually connect in some way to the performance direction we care about. But the two do not have the same role.

Why Is Loss Gathered Into One Number¶

Neural networks have many parameters. In order to change those parameters, the current state has to be gathered into one comparable standard.

For example, across several samples:

• some predictions may be slightly wrong
• some predictions may be very wrong
• some predictions may be almost correct

If that state is left as it is, it becomes hard to decide an update direction. So the loss function gathers many discrepancies into something like one number. In actual learning code, that number is reduced into a batch average or sum, and sometimes a regularization term is added so that the whole thing becomes the full objective/cost that is optimized.

In other words, the loss function plays the role of the compass number of the learning process.

If this is reduced into one sentence for readers, it can be remembered like this.

Loss is the signal that compresses into one number the direction in which the model is currently going wrong.

What Does It Mean When The Loss Decreases¶

This can be explained by the following sentence.

When the loss decreases, it means that the current model is creating less discrepancy from the target, at least on the training data.

But even here there is a point that needs care.

• just because the training loss decreased
• it cannot be concluded immediately that generalization improved

This point connects directly to the explanation of overfitting and generalization in Part 4. In other words, loss is very important, but the whole model cannot be judged from loss alone.

Cases And Examples¶

Case 1. Predicting Batch Energy Use¶

Suppose we are predicting the total energy use of one mixed batch. If the actual usage is 5.0kWh, model prediction A is 4.8kWh, and model prediction B is 2.0kWh, then a person first looks at which one is closer to the actual usage? Even by that standard alone, it is immediately visible that prediction A is less wrong than prediction B. But learning has to repeat this judgment with the same rule across all samples, so slightly off and far off must be turned into consistent numbers. The loss function turns exactly that difference into numbers that learning can use, so that it becomes possible to compare which prediction is worse and which is less bad. In other words, in regression problems, the first task is to turn how far off is it? into a number, rather than merely is it wrong?

If the same difference is read through the simplest rule, such as squared error, it becomes immediately visible why the loss spreads out.

The final result to confirm in this case is clear. Even if both predictions are wrong, loss does not stop at both are wrong. It gives a much larger number to prediction B, which missed by more, so that it will be corrected more strongly first.

Case 2. Classifying Inspection States¶

Classification works similarly. Suppose the correct answer is fine scratch, but model output A is normal 0.51, fine scratch 0.49, and model output B is normal 0.99, fine scratch 0.01.

People can easily say that both are wrong and move on. But from the viewpoint of learning, those two wrong answers cannot have the same weight. A almost got it right, while B judged the opposite very confidently. So the loss function also reflects how confidently it was wrong, and distinguishes cases where the same wrong answer still needs a much stronger correction. In other words, in classification problems, more than simple right-or-wrong, it also matters with what confidence did the model get it wrong? So the result to confirm in this case is whether, even though both are wrong answers, B receives a much larger penalty in the loss because it predicted the opposite with confidence.

Even without deriving cross-entropy in detail, the example the lower the probability assigned to the correct answer, the larger the loss becomes can be checked immediately.

If these two cases are turned back into graph shapes, regression loss grows as it moves farther away from the target value, while classification loss grows as the probability assigned to the correct class becomes smaller.

The purpose of these two graphs is not to memorize the formulas. It is to see along which axis does the loss grow larger? In regression, the key axis is the distance from the target value. In classification, it is the probability assigned to the correct class.

The final result to confirm in this case is also clear. In classification, even for the same wrong answer, a prediction such as B, which assigned almost no probability to the correct class, receives a much larger loss and is corrected more strongly first.

The core shown together by these two cases is the same. In regression, how far did it miss? matters more. In classification, how confidently was it wrong? matters more. Those things have to be reflected more strongly in the loss number so that the next update becomes more accurate.

If these two cases are tied into one line, the common role of the loss function is the same.

Turning the model's current discrepancy into a number that learning can use

And in this section, the explanation has to go one step further and close, at the level of cases, why is loss needed separately from the metric?

The core this table shows is simple. Even in cases that look like the same wrong answer when we look only at the metric or the final judgment, loss more finely distinguishes which wrong answer is more dangerous and should be corrected more strongly first.

flowchart TD
  A["Results that look equally wrong"]
  B["Human or metric view"]
  C["Both are just wrong or low-scoring"]
  D["Loss view"]
  E["Separate distance and confidence gaps"]
  F["Larger loss gets higher update priority"]

  A --> B
  B --> C
  A --> D
  D --> E
  E --> F

The final result the reader should hold in this flow is that loss is not an auxiliary score added after right/wrong judgment. It is the number that decides update priority.

One reason loss functions always appear early in machine-learning and deep-learning education is that learning is not simply repeating rules, but an optimization problem.

The core flow of deep learning usually continues like this.

put in an input -> create an output -> calculate the loss -> change the parameters in the direction that reduces the loss

In other words, the loss function is not an optional extra attached after the output. It is the standard that starts the whole learning calculation.

From the curriculum viewpoint as well, only when the loss function is understood first does it become natural to read:

• why backpropagation is needed
• what the optimizer is trying to reduce
• what number the learning rate is following

Practice And Exercise¶

The goal of this exercise is to read the simplest possible process in which the difference between prediction and target turns into a loss number. Rather than looking only at the average loss, the sample-by-sample errors are also checked so that it becomes visible on which item the large error occurred.

Input:

• 3 target values
• 3 prediction values

Output:

• squared error of each sample
• the sample that was most wrong
• average loss

Problem situation:

• the loss function should not be read only through the final average value; it must also be checked how the sample-by-sample errors accumulate

Concepts to confirm:

• average loss is a number made by gathering the error of each sample
• if the sample with the largest error is found, it becomes easier to interpret why the loss became large

Input:

Use the sample-by-sample target values and prediction values organized above.

Before looking at the table, it is useful first to predict which sample will pull up the average loss the most.

The purpose of this table is not to calculate the exact squared error in advance, but to hold first that average loss is not pulled equally by all samples; it is pulled more by the items that are more wrong.

If the same three batches are organized through the viewpoint of loss, they can be read as follows.

If these three values are gathered into the average, mean_loss = 0.35, and the batch that missed the most is read as restart_delay_batch.

What matters in this exercise is how the average loss number 0.35 was made.

• The first sample was slightly wrong.
• The second sample was also slightly wrong.
• The third sample was more wrong.

The loss function gathers these discrepancies into one number so that the model can read how bad its current state is. At the same time, if the sample-by-sample errors are examined, it also becomes visible which case is pulling up the average loss the most.

What the reader must read in this example is the following.

• The degree of error differs by sample.
• The loss function does not discard that difference, but reflects it numerically.
• The mistakes of several samples can be gathered into one average standard.

But if it stops there, it still remains only at the level of I looked at one average loss number. A loss example must also immediately make visible which sample should be corrected more strongly first.

Once this table is read, it becomes clearer that the loss function is not a procedure for calculating one average number, but the standard that identifies which wrong answer must be corrected more strongly first.

From the viewpoint of learning, it has to go one step further here. The sentence the reader should be able to say directly is not only does the average loss decrease?, but also which batch remains the worst case? and what weak input region does that batch represent? That is because the loss function is both the rule that summarizes numbers and the rule that attaches correction priority.

The same situation becomes more direct when it is tested briefly in Python to see which prediction, when corrected, lowers both the average loss and the worst case? The example below separates the case of strongly correcting restart_delay_batch from the case of correcting only night_shift_batch, whose error was already relatively small.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

# This example compares per-batch squared errors to see how mean loss and the worst case set prediction-fix priority.
samples = [
    {"name": "night_shift_batch", "target": 3.0, "prediction": 2.5},
    {"name": "stabilized_batch", "target": 1.0, "prediction": 1.4},
    {"name": "restart_delay_batch", "target": 2.0, "prediction": 1.2},
]


def squared_error(sample):
    error = sample["prediction"] - sample["target"]
    return error * error


def replace_prediction(samples, name, new_prediction):
    updated = [sample.copy() for sample in samples]
    for sample in updated:
        if sample["name"] == name:
            sample["prediction"] = new_prediction
    return updated


def summarize(label, samples):
    rows = [(sample["name"], squared_error(sample)) for sample in samples]
    mean_loss = sum(loss for _, loss in rows) / len(rows)
    worst_name, worst_loss = max(rows, key=lambda row: row[1])

    print(f"[{label}]")
    for name, loss in rows:
        print(f"{name}: squared_error={loss:.3f}")
    print(f"mean_loss={mean_loss:.3f}")
    print(f"worst_sample={worst_name} ({worst_loss:.3f})")
    print()


summarize("current", samples)

summarize(
    "fix_restart_delay_batch",
    replace_prediction(samples, "restart_delay_batch", 1.7),
)

summarize(
    "fix_night_shift_batch",
    replace_prediction(samples, "night_shift_batch", 3.0),
)

When executed, it can be read as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[current]
night_shift_batch: squared_error=0.250
stabilized_batch: squared_error=0.160
restart_delay_batch: squared_error=0.640
mean_loss=0.350
worst_sample=restart_delay_batch (0.640)

[fix_restart_delay_batch]
night_shift_batch: squared_error=0.250
stabilized_batch: squared_error=0.160
restart_delay_batch: squared_error=0.090
mean_loss=0.167
worst_sample=night_shift_batch (0.250)

[fix_night_shift_batch]
night_shift_batch: squared_error=0.000
stabilized_batch: squared_error=0.160
restart_delay_batch: squared_error=0.640
mean_loss=0.267
worst_sample=restart_delay_batch (0.640)

The value to manipulate here is the last number in replace_prediction(...). If restart_delay_batch is corrected, the average loss falls a lot and the worst case also changes, but if only night_shift_batch is corrected, the average falls a little while the largest weakness remains unchanged. So this code confirms that loss is not merely an average value, but a signal that reveals what should be corrected more strongly first.

In the graph, when restart_delay_batch is corrected, both the average loss and the worst loss fall together. By contrast, when only night_shift_batch is corrected, the average loss falls but the red bar, that is, the largest loss, remains unchanged. So the core point reinforced by this graph is that did the average decrease? and did the largest weakness decrease? must be read separately.

If the same output is turned into operational judgment, it becomes even more direct.

Is A Model Always Good If The Loss Is Small¶

Readers can easily feel that once the loss number becomes smaller, everything is solved. But the following must also be remembered.

• when reading loss, you should check not only whether the number itself became smaller, but also whether it fell only on the training data or remained stable on the validation data as well
• even with the same small loss, if you do not distinguish from which data region the value came, overfitting can easily be missed
• if you trust the model only from training loss, overfitting can be missed

In other words, loss is the central standard of learning, but it still has to be read together with the validation split, test set, and metrics seen in Part 4.

This is exactly why deep learning inherits the common machine-learning principles of Part 4 without change.

When Should The Loss-Function View Be Read First¶

The moment when the loss-function section needs to be brought in is when the explanation the model produced an output is still not enough to make it clear that learning has started.

Checklist¶

• Can you explain what the loss function turns into a number?
• Can you distinguish the role difference between loss and metric?
• Can you explain that the loss function is the rule that turns the gap between prediction and target into one number?
• Can you say why learning has not yet started with output alone, and why a loss number is additionally needed?
• Can you explain that loss is the internal standard by which the model learns, while the metric is the external standard read by people?
• Can you explain that learning is the process of changing parameters in the direction that reduces the loss?
• Can you say that a reduced loss does not automatically guarantee generalization?
• When an output was produced but the explanation is still missing what standard the model learns by, can you think of the loss-function view first?
• Do you know the flow that differences among loss types and backpropagation are passed to the next section and next chapter?

Sources And References¶

• Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning, MIT Press, 2016, date checked: 2026-06-29. https://www.deeplearningbook.org/
• Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006, date checked: 2026-07-19. https://link.springer.com/book/9780387310732