UAG DA client cannot connect, Error : ERROR_IPSEC_IKE_AUTH_FAIL in the network captures.

  • Article

Friends, for now i would say there are few prerequisites to understand following material i.e. you should have basic understanding of how to set up UAG DA and its concepts, so target audience is admins and peers who are already working on the UAG DA and can run into following scenario. I m planning to put links about basics of UAG DA here for everybody's understanding later.

Scenario discussion

I was setting up lab for a UAG DA Scenario and found that DA client was not connecting.So i checked if tunnels were by running following command

netsh advf monitor show mmsa

and found none of the two tunnels were up as i did not see main mode SA(security Associations) established, since Main mode did not establish ,quick mode cant. which meant, we had to start focusing on the IPSEC part, of the whole UAG DA technology which we know uses multiple technologies to implement DA, including IPSEC, apart from others like IPV6 ,Advance windows firewall etc.

so before digging deeper i took network captures on the client machine and ran following two commands in the command prompt.

net stop iphlpsvc (to stop IP helper service)

net start iphlpsvc (to start IP helper service)

purpose of doing this is to restart the IP helper service as it would try to start DA connectivity and would try to establish infrastructure tunnel and then corp tunnel. This would also mean that IPSEC SAs at main mode would also try to establish and that's where we wanted focus to start with.

In network captures got following error

ERROR_IPSEC_IKE_AUTH_FAIL in the network captures. Where i filtered the trace with protocol.authip i.e. with AuthIP protocol in network monitor .

This was not very descriptive so i took scenario tracing as below

Run following two commands in the command prompt

  •  Netsh trace start scenario=directaccess capture=yes report=yes tracefile=C:\client.etl
  •  Netsh wfp capture start

Then 

  • net stop iphlpsvc (to stop IP helper service)
  • net start iphlpsvc (to start IP helper service) 

 to initiate the DA connectivity again.

Then stopped the traces by running following two commands in the command prompt

 

  •  Netsh wfp capture stop
  •  Netsh trace stop

Then opened the client.etl file with network monitor and found following in it

 

i.e.  

 WFP:IPsec: Main Mode Failure - Error: ERROR_IPSEC_IKE_AUTH_FAIL
 WFP WFP:IPsec: Main Mode Failure - Error: CERT_E_WRONG_USAGE

From Ipconfig details of the client , since client was on public internet it was using 6to4 adapter, so my client with IPV6 IP address 2002:2828:2803::2828:2803 was sending that error regarding the machine certificate usage to the UAG DA server.

 **********************************************************************

 Tunnel adapter 6TO4 Adapter:

   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Microsoft 6to4 Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2002:2828:2803::2828:2803(Preferred) 
   Default Gateway . . . . . . . . . : 2002:2828:280b::2828:280b
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Disabled

 **********************************************************************

 That made me to check the machine certificate on the UAG DA server, which looked right, ran following command in the command prompt on the UAG DA server.

 certutil -store my

 and it showed me that certificate was OK.

  

 just to be on the safer side, i removed this machine certificate from the computer certificate store of the UAG DA server. 

 Then requested this machine certificate again from the certificate Authority and then again restarted the

  IP helper service as explained above to restart the DA connectivity.

 This time DA connected fine, Network captures were clean our AuthIP traffic in the network captures showed that authentication was successful.

 Tested DA by connecting to internal resources and it worked.