Investigating SDPA Numerical Errors and GPU Memory Remapping on ROCm
After migrating an inference environment from CUDA to ROCm, we investigated NaNs and deviations from reference values in attention operations. With identical finite inputs, the default SDPA path sometimes produced results that differed from the math SDPA path, or produced finite outputs that differed across repeated runs.
During the investigation, we also observed differences in latent values depending on the model’s memory allocator configuration. A separate diagnostic that did not use PyTorch subsequently reproduced memory corruption associated with GPU virtual address reuse. The SDPA numerical errors and memory remapping corruption are separate observations; there is no evidence that they share the same cause.
This article traces the investigation from SDPA through model boundaries to driver and kernel comparisons. The results are observations limited to the specified hardware, inputs, and repetitions, and cannot be generalized to every AMD GPU or model.
Starting with SDPA numerical errors
SDPA (scaled dot-product attention) computes attention output from Q, K, and V tensors. PyTorch selects among implementation paths according to the environment and input conditions. In this investigation, we repeatedly ran the default path with the same Q, K, and V and compared it with the math path for finite values and numerical differences.
On a Radeon 8060S, finite BF16 Q, K, and V captured from a Wan VAE had shape [1, 1, 64, 384]. The default outputs from both encoder and decoder were finite, but their mean absolute errors relative to the math path were 2.10054 and 0.23022, respectively. The three default-path outputs also differed from one another at each boundary. These observations apply to the tested inputs and conditions; the exact GPU kernel selected and behavior for other model inputs were not verified.
In a separate HeartMuLa codebook decoder case, default SDPA returned NaNs for finite input while math SDPA returned finite values for the same input. This path uses a different model and input shape from the Wan VAE, so the same kernel defect cannot be assumed. Conversely, in a case where Q, K, and V already contained NaNs, both the default and math paths returned NaNs. Input corruption and the result of an operation on finite inputs must therefore be examined separately.
Math SDPA was applied specifically to Wan VAE encoding and decoding, but subsequent generation still showed visual corruption and non-finite values. The workaround alone was not considered a resolution of the complete generation problem.
Test environment and problem definition
Animate2, used in the model tests below, is a motion-transfer workflow using the Wan Animate-2 Base model and Diffusers’ WanAnimate2ModularPipeline. It takes a character reference image and a human motion video and generates a video applying that motion. The VAE (variational autoencoder) converts video data into latent representations processed by the model and decodes generated latent representations back into video.
The test machine ran Ubuntu 26.04.1 LTS on an AMD Ryzen AI Max+ PRO 395 with Radeon 8060S (gfx1151). The baseline kernel was Ubuntu’s official 7.0.0-43-generic.
When a 65-frame Animate2 input was processed by the VAE, latent values differed according to PyTorch’s expandable_segments setting. Compared with the reference produced using False, two earlier True results differed in 901,738 and 971,113 of 1,577,600 latent elements, respectively.
NaNs appeared in some cases, but the absence of NaNs did not establish that values were correct. Instrumentation conditions, including hook usage, were not identical between the two baseline 7.0 results and the 7.4 comparison, so the mismatch counts were not treated as a controlled comparison between kernels.
From SDPA to the driver investigation
Non-finite values appeared again in complete generation even after comparing SDPA paths. The investigation was therefore extended to earlier outputs, including VAE outputs, and to attention input boundaries. The observed differences in latent values under different expandable_segments settings made it necessary to examine model computation separately from memory allocation and address reuse.
The arrows below indicate the evidence that led to the next investigation step. They do not claim that memory remapping corruption caused the SDPA numerical errors.
flowchart TD
A["SDPA output anomalies observed after ROCm migration"] --> B["Compare default and math SDPA with identical Q, K, and V"]
B --> C["SDPA path differences confirmed<br/>Generation issues remain after VAE math path"]
C --> D["Compare VAE boundaries and allocator settings"]
D --> E["Latent values differ with expandable_segments setting"]
E --> F["HSA remapping diagnostic without PyTorch"]
F --> G["Corruption on GPU address reuse<br/>Not observed with a different address"]
G --> H["Compare and trace AMDGPU paths on the same kernel"]
H --> I["Remove or add TLB calls and compare a delay control"]
I --> J["Evidence implicates invalidation and synchronization on address reuse"]
expandable_segments is a PyTorch setting that affects GPU memory allocation management. Changing it can also change allocation and address-use patterns, making the difference a lead for driver investigation. The setting alone, however, did not identify a failing driver operation. Address reuse was therefore controlled directly in a separate test.
GPU address reuse and the TLB
GPU virtual addresses are connected to physical memory through page tables. A GPU uses a TLB (translation lookaside buffer) to cache address translations. When an allocation is released and the same virtual address is mapped to a different physical allocation, the driver must update the page table and manage translation-cache invalidation and synchronization so that subsequent GPU work sees the changed mapping correctly.
flowchart LR
subgraph BEFORE["Previous mapping"]
A["GPU virtual address X"] --> B["Physical allocation A"]
end
subgraph AFTER["New mapping after unmap"]
C["Same GPU virtual address X"] --> D["Physical allocation B"]
end
BEFORE -->|"Page table update<br/>invalidation and synchronization"| AFTER
This diagram explains the relationship required during remapping. The test did not directly read physical GPU TLB entries, so it does not show which entries actually remained. The HSA diagnostic retained an alias to the previous physical allocation to check whether its data changed as well as checking the new allocation.
Testing address reuse with an HSA diagnostic
To isolate the problem, we ran an HSA remapping diagnostic without PyTorch. It unmapped a 20 MiB allocation, mapped a new allocation, and compared values in both the alias of the previous physical allocation and the new allocation. We separated cases where the new allocation reused the same GPU virtual address from a control using a different address.
The latest independent diagnostic with the baseline driver produced the following results:
| Address of new allocation | Previous allocation mismatch | New allocation mismatch |
|---|---|---|
| Same address reused | 19/20 runs | 2/20 runs |
| Different address | 0/20 runs | 0/20 runs |
This points to a memory visibility problem associated with address reuse, beyond model output alone. The test does not directly reveal the physical GPU TLB state or establish a single line of code as the initial defect.
Kernel and driver test results
A Pass below applies only to the stated test. HSA passes when there are no mismatched bytes; VAE passes when latent values match the reference bit for bit and contain no NaN or Inf. Complete generation passes when an 832×480, 65-frame, 20-step run completes, the inspected boundaries contain no NaN or Inf, and the hashes of all 65 PNGs match the comparison reference. Not every intermediate tensor was inspected.
| Environment | HSA: same address reused | VAE False / True |
Complete generation True |
|---|---|---|---|
| Ubuntu baseline 7.0 driver | Fail: previous 19/20, new 2/20 mismatches | False reference result / True mismatches observed |
Success under this condition not confirmed |
| Same 7.0 + experimental TLB fence backport | Pass: previous and new each 0/20 mismatches | Each matched reference 10/10 times | Pass: one run |
| Same 7.0 + unmodified AMDGPU 31.70 test module | Pass: previous and new each 0/20 mismatches | Each matched reference 2/2 times | Pass: one run |
| Ubuntu mainline 7.4-rc3 comparison kernel | Pass: previous and new each 0/60 across three independent processes | Each matched reference 2/2 times | Pass: one run |
The HSA control using a different address had no mismatches in each environment. The 7.4-rc3 complete generation result was compared with a False run on the same kernel. Complete generation with the backport and 31.70 was also run only once each, so those results do not establish long-term stability.
7.4.0-070400rc3-generic is an upstream Linux mainline test build provided by Ubuntu, not an official Ubuntu update kernel. Ubuntu states that mainline builds are unsupported and do not receive security updates. Linux 7.3.5 was examined only at the source level and was not tested on this GPU.
Comparing changes to the TLB path
After comparing the baseline module and 31.70 in separate boots of the same 7.0 kernel, we traced the HSA unmap interval. In the specified diagnostic thread and interval, a compute TLB function call was not observed with the Ubuntu baseline module but was observed 20 times with 31.70. This difference narrowed the candidate path for intervention, but the presence or absence of a function call alone cannot establish hardware state.
To investigate further, we added or removed the call and compared the result with a delay-only control.
- An experimental backport of the upstream TLB fence change passed the repeated HSA and VAE tests and one complete generation run.
- A separate experimental module that added only a compute TLB call to the baseline 7.0 source passed 20 runs for each HSA condition and 10 VAE runs for each allocator setting. This minimal intervention is not a production patch: general asynchronous update and multiple-XCC handling are incomplete.
- Removing the compute TLB call from the GEM VA path of the 31.70 test module resulted in previous-allocation mismatches in 19/20 runs and new-allocation mismatches in 3/20 runs when the same address was reused. In a separate repeated VAE test,
Truediffered from the reference in 10/10 runs, none containing NaNs;Falsematched in 10/10. A NaN was observed in one of two other VAE runs. - A control replacing the call with a delay of approximately 170 µs still corrupted the previous allocation in 15/20 runs. Waiting alone did not produce the same effect.
The last two results apply to experimentally modified 31.70 and should not be interpreted as failures of the unmodified 31.70 test module.
The baseline driver’s CPU page table update path has a branch that skips TLB fence creation when there is no input fence; that condition was observed during a failing interval. Taken together, the interventions provide strong evidence that the driver’s invalidation and synchronization path is involved after GPU address reuse. Physical TLB contents were not read directly, so the initial hardware state remains unconfirmed.
Results of the intervention tests
The diagram shows the number of runs with a previous-allocation mismatch when the same HSA address was reused. Each item represents 20 runs in a separate boot; the tests used different modules and execution conditions.
flowchart LR
subgraph RESULTS["HSA previous-allocation mismatches · 20 runs each"]
direction TB
A["Baseline 7.0: 19/20"]
B["31.70 unmodified: 0/20"]
C["31.70 TLB call removed: 19/20"]
D["31.70 170 µs delay in place of call: 15/20"]
E["7.0 TLB fence backport: 0/20"]
end
classDef fail fill:#fce8e6,stroke:#b3261e,color:#42110d
classDef pass fill:#e6f4ea,stroke:#137333,color:#0b3214
class A,C,D fail
class B,E pass
The 19 baseline 7.0 mismatches are from the latest independent diagnostic; the 19 mismatches after removing the 31.70 call are from a separate intervention test. Corruption returned when the call was removed and persisted when the call was replaced by an approximately 170 µs delay. Conversely, corruption was not reproduced in the experiment that added a TLB-related path to 7.0. These findings support the interpretation that the driver’s invalidation and synchronization path affects the result of address reuse, beyond a simple elapsed-time difference. The tests have limited repetitions and do not directly prove physical TLB state or a single defect location.
Operational response and scope
At the time of testing, the Animate2 worker enforced expandable_segments:False and checked for non-finite values at the VAE and inference boundaries. This is a workaround for the observed model path. Since the issue also appeared in the HSA diagnostic without PyTorch, the setting cannot be considered a fix for the driver defect itself.
The unmodified 31.70 test module passed within the stated scope, but long-term stability and general desktop use were not tested. AMD’s ROCm compatibility matrix specifies the built-in kernel driver of a supported Ubuntu release for supported Ryzen APUs including Radeon 8060S. The isolated result was therefore not treated as a general installation recommendation or an AMD-supported configuration for this machine. The default system module was not replaced.
After an official kernel or driver update, the plan is to rerun the HSA remapping diagnostic while the GPU is idle and perform regression tests on latent values and generated output using actual model inputs. The workaround should not be removed without separate review even if those tests pass.
Experimental code for future regression testing
For SDPA, preserve identical Q, K, and V, mask, scale, dtype, shape, and stride, then compare repeated default-path results with the math path. Check input and output finiteness, numerical error, and consistency across runs. Record that copies and synchronization for instrumentation can alter execution conditions. Separately, after a driver or kernel update, VAE latents can be preserved under identical input and model settings and compared with a reference. The following example compares two tensors already extracted; reference and candidate are PyTorch tensors with the same shape and dtype. It does not run the model or extract tensors.
import torch
def compare_latents(reference: torch.Tensor, candidate: torch.Tensor) -> dict:
"""Check finiteness and bitwise equality against reference latents."""
if reference.shape != candidate.shape or reference.dtype != candidate.dtype:
raise ValueError("Latents must have the same shape and dtype.")
if not reference.is_floating_point():
raise ValueError("Floating-point latents are required.")
reference = reference.detach().cpu().contiguous()
candidate = candidate.detach().cpu().contiguous()
finite = torch.isfinite(reference) & torch.isfinite(candidate)
# Compare bytes to distinguish signed zeros and NaN bit patterns.
byte_diff = reference.view(torch.uint8) != candidate.view(torch.uint8)
element_diff = byte_diff.reshape(reference.numel(), reference.element_size()).any(dim=1)
return {
"elements": reference.numel(),
"reference_nonfinite": int((~torch.isfinite(reference)).sum()),
"candidate_nonfinite": int((~torch.isfinite(candidate)).sum()),
"different_elements": int(element_diff.sum()),
"both_finite": bool(finite.all()),
}
This test passes when neither tensor contains NaN or Inf and different_elements is zero. To establish a reproducible reference, record the input, seed, model and weights, dtype, software versions, and instrumentation method. Normal numerical differences can occur between environments, so bitwise equality is a regression criterion under identical conditions, not a universal accuracy criterion across implementations.
Set expandable_segments before GPU initialization and run each condition in a separate process. For example, with an existing model execution script:
PYTORCH_ALLOC_CONF=expandable_segments:False python run_vae.py
PYTORCH_ALLOC_CONF=expandable_segments:True python run_vae.py
Here, run_vae.py represents a script supplied by the reader; this article does not provide a complete reproduction program.
The HSA remapping test must be performed separately from this tensor comparison. Repeat both the same-address and different-address conditions and check mismatched bytes in the previous and new allocations. Run as a regular user while the GPU is idle, and record unexecuted conditions or missing results as unverified. The code above does not replace an HSA diagnostic or a driver fix.
Future response procedure
- Retain operational settings: Until verification, retain
expandable_segments:Falseand non-finite checks at VAE and inference boundaries. Do not treat finite values alone as proof of accuracy. - Establish a pre-change reference: Record latent values and output for the same input and seed, along with kernel, driver, ROCm, and PyTorch versions. Include instrumentation methods in the comparison conditions.
- Verify in stages after a change: Compare repeated default SDPA results with the math path using preserved finite Q, K, and V. Separately, while the GPU is idle, test HSA address reuse and the control, then compare model latents and run complete-generation regression tests. Record repetitions, failures, and unverified scope.
- Review deployment: Separately confirm official driver support and operational stability. Passing limited regression tests alone does not justify removing the workaround or deploying an experimental module.
Additional observations and scope
A graphics queue timeout and GNOME session termination were also observed during a long generation run. Whether the display termination and HSA remapping corruption share a cause remains unconfirmed. Even when observations occur in the same environment, they require independent reproduction and comparison.
This investigation separated model results, the memory allocator, and driver paths to narrow the problem. Preserving reference values for identical inputs, controlling address reuse, and comparing the effects of adding or removing a specific path provided more concrete evidence for interpreting the observations and potential causes.
Go Home