Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Remote Desktop "An internal error has occurred": certificate, security layer and service checks

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

You open Remote Desktop Connection to a server, and it stops with "An internal error has occurred." Sometimes it fails before the credentials prompt, sometimes straight after. The message names no cause.

Microsoft's troubleshooting page for this error covers direct RDP connections, with no Remote Desktop Gateway in the path (Microsoft Learn, "An internal error has occurred" when connecting to a remote machine). It works through the checks in a fixed order: network or server, then security negotiation, then the server's Remote Desktop certificate or protocol settings.

What you see

  • "An internal error has occurred" in the Remote Desktop Connection app (mstsc), before or after you enter credentials.
  • The error may appear from one client or from all of them; the console test in step 1 shows whether the server is the cause.
  • A variant Microsoft describes separately: the connection hangs for up to about a minute, then the error appears, and the second attempt works.

Why it happens

Microsoft says the error usually occurs while the two computers negotiate security protocols, ciphers or other encryption settings. Its page names these causes:

  • The Remote Desktop self-signed certificate cannot be created or renewed. The server makes its own certificate for RDP when no other certificate is set. Creating it needs certain services running and the right permissions on the machine key folder.
  • Security protocol negotiation fails. A listener set to use a certificate that no longer matches, a cipher-order policy, or damaged TLS (SCHANNEL) or cryptography settings in the registry.
  • The network, not the server. If the server accepts a connection from itself, the fault is between client and server.
  • Audio settings (the one-minute hang). Microsoft describes a blocking call while the client reads audio settings, which holds up the connection until the server resets it after about a minute.

How to prove the cause

Work through these in order. Each step tells you which branch you are on.

1. Network or server? Sign in to the server's console (for example through the Hyper-V console, as Microsoft suggests, or your hypervisor's equivalent), run mstsc and connect to localhost.

2. Is the listener up? In an elevated PowerShell on the server, run qwinsta. Microsoft's page says the listener is working if rdp-tcp is listed with the state Listen. Check that the Remote Desktop Services (TermService) and Remote Desktop Services UserMode Port Redirector (UmRdpService) services are running.

3. Is it the security negotiation? Microsoft's test is to set two values to 0 temporarily on the affected server. Disruptive test: this lowers the security of every Remote Desktop connection to the server while it is in place, so run it in a maintenance window. Back up the registry first, and note the current values (or that they do not exist):

reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" /v SecurityLayer
reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" /v UserAuthentication

Then set both to 0:

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" /v SecurityLayer /t REG_DWORD /d 0 /f
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" /v UserAuthentication /t REG_DWORD /d 0 /f

If you can now connect, the cause is the self-signed certificate or the protocol negotiation. Put the values back as soon as the test is over. Set each one to the value you noted, with the same reg add command and that number. If a value did not exist before the test, remove it again:

reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" /v SecurityLayer /f
reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" /v UserAuthentication /f

4. Certificate or negotiation? Open the Certificates snap-in for the Computer account on the server. Under Remote Desktop > Certificates, delete the RDP self-signed certificate, restart the Remote Desktop Services service and refresh the snap-in. Disruptive step: restarting Remote Desktop Services disconnects everyone working on that server over Remote Desktop, including you if that is how you are connected. Do it in a maintenance window, from the console or another management route.

  • Certificate not re-created: follow the certificate checks in the fix.
  • Certificate re-created: follow the negotiation checks in the fix.

The fix

The self-signed certificate is not being created

From Microsoft's page:

  • Registry values. Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations, SelfSignedCertStore holds Remote Desktop. Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList, ProgramData holds %SystemDrive%\ProgramData.
  • Services. These must be running to create the certificate: Certificate Propagation (CertPropSvc), CNG Key Isolation (KeyIso), Cryptographic Services (CryptSvc) and Remote Desktop Configuration (SessionEnv). If one will not start, look for its dependencies in the System event log, source Service Control Manager.
  • MachineKeys folder permissions. On C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys, only: Builtin\Administrators Full control; Everyone Read, Write. Caution: this folder holds the private keys of every machine certificate on the server, not only the RDP one, so a wrong change here can break other services that use certificates (web sites, for example). Save the current permissions first, for example icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys /save machinekeys-acl.txt, and change only the folder entries Microsoft lists.
  • The RDP machine key's permissions. In that folder, the key whose name starts with f686aace6942fb7f7ceb231212eef4a4: make an administrator the owner, then allow only Network Service Read and System Full control.

After each change, restart Remote Desktop Services (disruptive, as in step 4) and check that the certificate comes back.

The security negotiation fails

  • The listener's certificate thumbprint. Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp, compare the certificate thumbprint value with the Thumbprint of the certificate in certlm.msc under Remote Desktop > Certificates. Microsoft's internal-error page calls the value SSLCertificateSHA1HashValue; its listener certificate page names it SSLCertificateSHA1Hash (Microsoft Learn, Remote Desktop listener certificate configurations). If the values do not match, delete the registry value, restart the server and test again. Disruptive step: restarting the server disconnects everyone working on it and stops every service it runs, so do it in a maintenance window, from the console or another management route. Microsoft says not to do this on a server with the Remote Desktop Connection Broker role.
  • Two listener values. fAllowSecProtocolNegotiation set to 1, and MinEncryptionLevel set to 3.
  • Cipher-order policy. If Computer Configuration > Administrative Templates > Network > SSL Configuration Settings > SSL Cipher Suite Order is set, change it to Not configured. Then delete the Functions value under HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002 if it is still there.
  • Damaged TLS settings. Microsoft's last step is to export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL, and separately HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002, from a working server on the same Windows version. Back up the affected server's keys, then replace them with the exported ones.

If you want the listener to use your own certificate

Microsoft's listener page covers this for servers that are not part of a Remote Desktop Services deployment. Import the certificate into the computer's Personal store first, then set the thumbprint with PowerShell:

Get-WmiObject -class "Win32_TSGeneralSetting" -Namespace root\cimv2\terminalservices | Set-WmiInstance -Arguments @{SSLCertificateSHA1Hash="THUMBPRINT"}

Microsoft warns that a thumbprint copied from the certificate dialog can carry a hidden character; remove all spaces and check the string at a command prompt before you use it. The page also says NETWORK SERVICE needs Read permission on the certificate's private key (All Tasks > Manage Private Keys).

The one-minute hang

Microsoft's workaround is to disable the Windows Audio service on the remote server, and on the client computer too if needed.

How to confirm the fix

  • Connect from the console to localhost, then from a normal client computer.
  • qwinsta shows rdp-tcp in the Listen state.
  • SecurityLayer and UserAuthentication are back at the values you noted in step 3 (or absent, if they did not exist).
  • If you replaced the certificate, the client's certificate prompt, if any, shows the certificate you expect.

What to watch afterwards

  • Group Policy. The two test values live under the Policies key, which Group Policy manages. This follows from the key's location; check with gpresult /H which policy sets them if they change back.
  • Certificate expiry. The listener points at one certificate by its thumbprint, so a certificate you set by hand needs renewing and setting again before it expires. This follows from how Microsoft describes the setting.
  • TLS setting changes. The cipher-order policy and the SCHANNEL settings are on Microsoft's cause list. Test RDP after changing either.

Risks and limits

  • Setting SecurityLayer and UserAuthentication to 0 lowers the connection's security. Use it only as a short test, then restore the values you noted.
  • Replacing the SCHANNEL or cryptography keys changes TLS for the whole server, not just RDP. Back up the keys first and copy from the same Windows version only.
  • Do not delete the thumbprint value on a Connection Broker server.
  • Deleting the self-signed certificate is part of Microsoft's test; if it is not re-created, connections can keep failing until the certificate checks are fixed. Keep console access open.
  • If a Remote Desktop Gateway is in the path, this page does not cover it; check the gateway first.

Sources

Next step

While this error stands, Remote Desktop to that server does not work for the people affected. NetLeverage has looked after business servers since 2002. If you are seeing this error, talk to us through managed support with the result of the console localhost test and whether the self-signed certificate comes back.