UE 5.7 and 5.8 Pixel Streaming crashes, VRAM leaks and disconnects: the NVENC D3D12 bug and its fix

Who is affected#

It applies to Windows builds using DirectX 12 and an NVIDIA GPU with hardware encoding, which is the standard Pixel Streaming setup on cloud and on-premises GPU servers. It is independent of the plugin generation (Pixel Streaming or Pixel Streaming 2) because the change sits in the engine’s encoder layer underneath both.

What changed in Unreal Engine 5.7#

Pixel Streaming captures each rendered frame into a texture and hands it to NVENC to encode as H.264, AV1 or another hardware codec before WebRTC sends it to the browser.

Up to UE 5.6, the captured D3D12 texture was first copied into memory owned by CUDA, and NVENC read that CUDA copy. The copy costs a little GPU time but gives the encoder a buffer the renderer never touches again.

From UE 5.7 the engine takes a more direct path: NVENC registers the D3D12 texture itself and reads from it. One copy less per frame, in principle faster. In practice two problems appeared with it, and the 5.8 console variable AVCodecs.NvEnc.D3D12UsesCUDA exists to bring back the 5.6-style CUDA copy.

Failure mode 1: memory grows until the app dies#

What viewers see: the stream starts normally and runs fine, then after tens of minutes (sooner at high resolution or frame rate, and faster on lossy networks) the image freezes, input stops responding and the viewer is disconnected. A new session on a freshly launched app works again, until it does not.

What you see on the server:

• The app’s memory climbs steadily and never plateaus. The public reports describe the process’s committed memory growing under nvEncodeAPI64.dll (one UE 5.8 report measured about 1 GB per minute per encoder session) while physical RAM and VRAM stay flat, ending with Windows terminating the process. Sample both the commit size and the dedicated GPU memory of the process with Task Manager or nvidia-smi every minute; whichever climbs, the shape is the same: linear growth for as long as the stream runs.
• The end of the session is often an “Out of video memory” message from the engine, which the reports call misleading, a device removal, or a hang with no log line at all. The Windows System event log may record a Resource-Exhaustion-Detector event naming the game executable.
• It reproduces with the H.264 hardware encoder and disappears on VP8 or VP9.

The analysis in issue #900 ties the growth to rate control: WebRTC changes the target bitrate and frame rate frequently, each change calls nvEncReconfigureEncoder, and memory accumulates on every call. That also explains why the growth accelerates with packet loss, which makes the rate controller busier. It is the form most operators hit first, because it only needs a long-running session.

Failure mode 2: the fast crash at connect#

What viewers see: the app launches, the viewer connects, and within seconds the stream dies. It is intermittent, roughly every other launch in the builds where we saw it, and more likely on scenes that stream in a lot of content right after start.

What you see in the log:

• LogD3D12RHI: Error: ... DXGI_ERROR_DEVICE_HUNG, followed by a device-removed report.
• With DRED (Device Removed Extended Data) or NVIDIA Aftermath enabled, a page fault whose faulting address belongs to the Pixel Capture texture, FPixelCaptureCapturerRHIRDG Texture.
• The “active pass” and “recently freed resource” named in the DRED breadcrumbs change from crash to crash: a water pre-pass one time, Nanite culling the next, the FX system after that.

That last point is the trap. The breadcrumbs point at whatever the renderer happened to be doing when the encoder read the texture, not at the real culprit, so teams spend days disabling Nanite, water or transient allocation. None of it helps, because the faulting access comes from the encoder, which DRED cannot attribute.

Two quick checks confirm the encoder is involved:

• Launch the packaged executable directly, without Pixel Streaming. If it never crashes, the renderer is fine.
• Switch the stream to VP9. VP9 in Pixel Streaming is encoded in software from a converted frame and never registers a D3D12 texture with NVENC. If VP9 is stable, you have this bug.

The fixes, in detail#

UE 5.8 and later: turn on the CUDA path#

Add the flag to the app’s launch arguments, next to your other Pixel Streaming flags:

YourApp.exe -PixelStreamingURL=ws://... -RenderOffScreen -AVCodecs.NvEnc.D3D12UsesCUDA=true

If you would rather bake it into the build than rely on launch arguments, the form posted on the Epic forums is an entry in the project’s game configuration:

[AVCodecs.NvEnc]
D3D12UsesCUDA=1

UE 5.7: no switch, choose a trade-off#

In 5.7 the variable is not recognised; an unknown console variable on the command line is silently ignored, so passing the flag changes nothing. Check yours the same way as above: if the console reports the variable as unknown, it is not in your build. Your options:

Option A: DirectX 11. Launch with -dx11. The DX11 encoder path is unaffected. You lose DX12-only features such as hardware ray tracing and anything that depends on them; check that Lumen, Nanite and your post-processing still look acceptable, since some fall back to software or lower-quality modes.

Option B: VP9. Set the Pixel Streaming codec to VP9. On the original plugin the argument is -PixelStreamingEncoderCodec=VP9; Epic’s reference notes that several settings changed or were removed in Pixel Streaming 2, so check the flag your build ships. VP9 is encoded in software, so it avoids NVENC entirely. The cost is CPU load, lower frame rates at high resolutions, and more bandwidth for the same visual quality. It is a stop-gap, not a long-term setting.

Option C: fix the encoder bitrate. The memory growth is driven by rate changes, so removing them slows it down. One reporter measured roughly a tenfold reduction by pinning the target bitrate, for example -PixelStreamingEncoderTargetBitrate=20000000 (bits per second; the original plugin’s argument name). You give up adaptive bitrate, so viewers on weak connections will see stalls instead of a lower-quality picture, and the growth is reduced, not removed.

Option D: upgrade to 5.8 and use the flag. For most projects this is the real fix. Moving from 5.7 to 5.8 is a minor-version upgrade; plan normal regression testing, particularly for plugins.

UE 5.6 and earlier#

Nothing to do. These versions already use the CUDA copy path.

Choosing the fix: quick decision guide#

1. On 5.8 or later? Add -AVCodecs.NvEnc.D3D12UsesCUDA=true, then confirm a viewer can connect and stream on each GPU model you run.
2. On 5.7 and can upgrade within weeks? Upgrade to 5.8, then step 1.
3. On 5.7, cannot upgrade, and do not need ray tracing? -dx11.
4. On 5.7, cannot upgrade, and need DX12 features? VP9 or a fixed bitrate as a temporary measure, and plan the upgrade.

Will a driver update fix it?#

Do not wait for one. NVIDIA’s forum staff replied to a report of this crash that Pixel Streaming is a third-party plugin and referred users to Epic, so there is no public NVIDIA bug to track. The reconfiguration analysis points at engine code (FEncoderNVENC::ApplyConfig calling nvEncReconfigureEncoder for rate-only changes), and the crash-at-connect race is in how the capture texture is handed to the encoder. Both are Epic-side. A community member on the Epic forums wrote in May 2026 that “there should be a fix for this in the near future”; until an engine release says so, keep the CUDA path on where it works.

What we could not verify#

• The exact engine change in 5.7. No Epic release note describes it; the mechanism above is inferred.
• Whether the variable exists in late 5.7 hotfix builds. Check with the console as described.
• The RTX 50-series failure. We have one public report and no reproduction.
• Whether the slow leak is entirely explained by reconfiguration calls or also by the direct D3D12 path. The flag fixes the reports that used it; the fixed-bitrate mitigation reduces growth without the flag, which suggests both matter.

FAQ#

Does this affect AV1 or only H.264?#

The memory growth is reported with H.264. The fast crash at connect is not tied to a codec: we reproduced it on an AV1 stream. Both go away with D3D12UsesCUDA=true on the hardware where the flag works, because the encoder no longer reads the D3D12 texture directly.

Does the flag reduce frame rate?#

It adds one GPU copy per frame. On our builds that was not measurable against rendering and encoding. Measure on yours with the method in the latency article.

I added the flag on UE 5.7 and nothing changed. Why?#

The variable was added for 5.8. On the 5.7 builds we have seen it is not recognised and is silently ignored. Use -dx11, VP9, a fixed bitrate, or upgrade.

My DRED report blames Nanite, water or particles. Is it still this bug?#

Probably, if the faulting address is in FPixelCaptureCapturerRHIRDG Texture and the blamed pass changes between crashes. Confirm by running the app without Pixel Streaming, or on VP9.

Is UE 5.6 affected?#

No. 5.6 and earlier use the CUDA copy path by default.

Does it matter whether I use Pixel Streaming or Pixel Streaming 2?#

No. The change is in the engine’s NVENC integration, which both plugins use.

Should I enable -d3d12gpuvalidation to debug it?#

Only on a machine with the Windows “Graphics Tools” optional feature installed. Without it, the app fails at start-up with a D3D12 adapter error, which looks like a new crash.

The flag killed my encoder instead of fixing it. What now?#

That matches the RTX 50-series report. Remove the flag, pin the encoder bitrate to slow the growth, restart sessions before memory runs out, and report your GPU model and driver on the Epic infrastructure repository so the combination is on record.