I was trying to install the DPM agents from System Center Data Protection Manger 2012 to Windows Servers and got the above error message.

I could fix this error by installing manually by following the steps,

• Shared the path, C:\Program Files\Microsoft System Center 2012\DPM\DPM\Protection\RA\4.0.xxxx and made to access from \\dpm-server\DPMAgent\$\
• Installed the agents manually from command prompt
• Copied to the local disk from the above path to the servers.
• Opened Command Prompt (I used run as Administrator), install the agent manually by running as below c:\Downloads\amd64\DPMAgenInstaller_x64 dpm-server.domain.local  ; where dpm-server.domain.local is my DPM server.
• Added the servers to take the backup from DPM Management Console.

If you are still experiencing the problem, I found in a forum mentioned to add the DPM Server to your Domain Admins Group to allow the DPM Server to have the communication properly.  Once I added that the agent started working fine after refreshing in the DPM Management Console.