Troubleshooting AirWatch Tunnel/MAG and API/AWCM Trust Errors

Tunnel to API and AWCM

As part of the network requirements the AirWatch Tunnel requires access to both the AirWatch API and AirWatch Cloud Messaging (AWCM) services. The Tunnel makes requests to both these endpoints to retrieve and update a list of devices that have permission to access internal resources. If the API or AWCM connections are failing, the Tunnel may be unable to get a list of approved devices and so will block devices attempting to reach internal resources. 

The Tunnel services that require communication to the API and AWCM endpoints are:

  • Proxy services on both Windows and Linux*
  • Per-App VPN service on Linux (VPND)

Note: The Content-Gateway on both Windows and Linux does not currently make connections to API or AWCM, so this article is not relevant for troubleshooting Content-Gateway errors. 

The requests made by Tunnel to retrieve this information are done over a secure HTTPS connection; this requires that the Tunnel be able to trust the SSL certificates bound on those servers. If this trust fails we will encounter SSL Handshake errors in the log files. This article provides steps on how to resolve those errors.

For the Tunnel Proxy (on both Windows and Linux) the API and AWCM requests will be made from the Relay server (if using a relay-endpoint configuration) unless the “EnableOutboundCallsViaProxy” option found in the config.xml file is set to true, in which case the requests are proxied to the Endpoint Tunnel server, and from there potentially to another outbound proxy. The Per-app VPN service will only make these requests from the Relay Tunnel server (in a relay-endpoint configuration). If the requests are being made from the Relay server then the resolution steps listed below need only be done on the Relay server. If using a basic Endpoint only configuration the steps need only be done on the Endpoint server.

 

Error in logs

As the following examples show, failures to establish a secure connection to API and AWCM are easily identified in logs. Please note the letter (A-D) next to the error you are encountering to guide you through the resolution steps below:

 

A: Error when connecting from API to Tunnel Proxy on Windows or Linux (from mag.log):

WARN  [main] [RelayMag:184] - Unable to get latest config from API Server! Reason: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification 

 

B: Error when connecting to AWCM from Tunnel Proxy on Windows or Linux (from mag.log):

WARN  [AwcmIdler] [AwcmIdler:130] - Communication to AWCM failed! Reason: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

 

C: Error when connecting to API from Tunnel Per-app VPN on Linux (from tunnel.log):

INFO: TCPSocket::Connect Resolving address {API_HOSTNAME}

….(lines omitted for brevity)….

ERROR: SSLClient: Handshake with AWCM/API returns returns=-1 error=1 error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

 

D: Error when connecting to AWCM from Tunnel Per-app VPN on Linux (from tunnel.log):

INFO: TCPSocket::Connect Resolving address {AWCM_HOSTNAME}

….(lines omitted for brevity)….

ERROR: SSLClient: Handshake with AWCM/API returns returns=-1 error=1 error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

 

Resolving the SSL Certificate Trust failure

To resolve any of the four errors mentioned above we will import the root certificate (and in some cases also the intermediate certificate) from the API or AWCM server into the correct certificate trust store on the Tunnel server. If using a Relay-Endpoint Tunnel configuration these steps need only be done on the Relay server. If using a basic Endpoint configuration the steps will be done on the Endpoint Tunnel server. The steps to accomplish this will vary based on platform and for which component is affected, and are outlined below. Before proceeding please note which error you are encountering, A-D, as shown in the above section. 

If the AirWatch API and AWCM services are on the same server they may both be using the same SSL certificate, but this is not always true. It is recommend to browse to your API or AWCM address from a desktop browser and verify which certificates are being used. If you notice that the certificate being used is invalid (expired or revoked certificate, address mismatch, etc.), the steps below will not resolve the issue and the certificate invalidity must be addressed first. 

 

Resolving Errors A & B: Tunnel Proxy to API or AWCM on Windows

1. Download a Base-64 encoded X.509 copy of the root and intermediate certificates from your API or AWCM site. This can be done by browsing to your API or AWCM URL from a desktop browser, viewing the certificate details, and then copying to a file. If you are unsure of what URL the Tunnel is using for API or AWCM you can find it here:

    • Open the file at: C:\AirWatch\MobileAccessGateway\conf\config.xml and view the “ApiUrl” and “AwcmUrl” values

Copy the certificate file(s) to: C:\Program Files\Java\jre{version}\lib\security. This path will vary based on the JRE version installed. If you have multiple folders at the C:\Program Files\Java\ directory, we will first confirm on the specific version of Java being used by viewing the “wrapper.java.command” value found in the file at: 

C:\AirWatch\MobileAccessGateway\service\mag-conf\magServiceWrapper.conf. Note that the path listed in this folder is a shortname and will appear slightly different than the path shown in the Windows Explorer window.

3. Open a Command Window into the same …\lib\security folder where we copied the certificate file(s)

4. In the command line, run the following command:

    • keytool -importcert -keystore cacerts -trustcacerts -alias ALIAS_NAME -file FILE_NAME -storepass KEYSTORE_PASSWORD –noprompt
      • …where ALIAS_NAME can be any name to help you identify this certificate within the Java Keystore at a later time
      • …where  FILE_NAME is the name of the root or intermediate .cer/.crt file that we copied into the folder in step 2
      • …where KEYSTORE_PASSWORD is the Java Keystore password. By default the password is “changeit”.

5. If importing multiple certificates into the Java Keystore (such as both the root and intermediate certificates), complete step 4 for each of these certificates.

6. Restart the “AirWatch Mobile Access Gateway” service and confirm if the error is still present in the logs. If the error is no longer occurring, you have completed this troubleshooting guide. 

 

Resolving Errors A & B: Tunnel Proxy to API or AWCM on Linux

1. Download a Base-64 encoded X.509 copy of the root and intermediate certificates from your API or AWCM site. This can be done by browsing to your API or AWCM URL from a desktop browser, viewing the certificate details, and then copying to a file. If you are unsure of what URL the Tunnel is using for API or AWCM you can find it here:

Open the file at: /opt/airwatch/tunnel/proxy/conf/config.xml and view the “ApiUrl” and “AwcmUrl” values

2. Copy the certificate file(s) to: /usr/java/default/jre/lib/security

3. Change your working directory to the same path as shown in step 2 and run the following command:

    • keytool -import -keystore cacerts -trustcacerts -alias ALIAS_NAME - file FILE_NAME -storepass KEYSTORE_PASSWORD –noprompt
      • …where ALIAS_NAME can be any name to help you identify this certificate within the Java Keystore at a later time
      • …where  FILE_NAME is the name of the root or intermediate .cer/.crt file that we copied into the folder in step 2
      • …where KEYSTORE_PASSWORD is the Java Keystore password. By default the password is “changeit"

4. If importing multiple certificates into the Java Keystore (such as both the root and intermediate certificates), complete step 3 for each of these certificates. 

5. Restart the Tunnel proxy service and confirm if the error is still present in the logs. If the error is no longer occurring, you have completed this troubleshooting guide.

 

Resolving Errors C & D: Tunnel Per-App VPN to API or AWCM on Linux

1. To confirm that the error is due to a certificate missing from the trusted CA bundle, run the following command. If either command returns a certificate error proceed to step 2 to resolve.

For API endpoint:

    • curl https://{API_URL}/api/help --cacert /etc/pki/tls/certs/ca-bundle.crt
    • The expected response is a HTTP 401 unauthorized with no certificate errors

For AWCM endpoint:

    • curl https://{AWCM_URL}/awcm/status --cacert /etc/pki/tls/certs/ca-bundle.crt
    • The expected response is a HTTP 200 with “OK” returned and no certificate errors

2. Download a Base-64 encoded X.509 copy of the root and intermediate certificates from your AWCM site. This can be done by browsing to your API or AWCM URL from a desktop browser, viewing the certificate details, and then copying to a file. If you are unsure of what URL the Tunnel is using for AWCM or API you can find it here:

    • Open the file at: /opt/airwatch/tunnel/vpnd/server.conf and view the “api_address” and “awcm_address” values. For AWCM, please also note the “awcm_port” value.

3. Open the certificate file into a text editor and select and copy all of the text between and including the “-----BEGIN CERTIFICATE-----“ and “-----END CERTIFICATE-----“ lines.

4. Navigate to the path containing the ca-bundle used by Tunnel Per-App VPN. By default this is found at: /etc/pki/tls/certs/ca-bundle.crt. You can confirm or change the bundle by viewing the “api_ca” value in the server.conf file mentioned in step 2.a.

5. Paste the text copied in step 3 to the end of this file. If you are importing multiple certificates you can paste the text for those certificates in any order into the file. Save the file with these changes. 

6. Restart the VPND service and confirm if the error is still present in the log. 

 

*Note: The Proxy service on Windows Server is commonly referred to as Mobile Access Gateway (MAG). Throughout this article we will refer both the Windows and Linux services as the AirWatch Tunnel but will make a distinction between Windows and Linux when relevant. Whether you are using the MAG on Windows or the Tunnel on Linux this article will still apply to your implementation. 

Have more questions? Submit a request

0 Comments

Article is closed for comments.