NTLM and Kerberos protocols

This article explains how two most common authentication mechanisms (NTLM and Kerberos) work. Both authentication protocols are based on symmetric key cryptography. The protocols themselves are platform independent.

NTLM Authentication

NTLM is the default authentication protocol prior to Windows 2000 and still prevalent today as backup to Kerberos. It is based on challenge/response mechanism.

Authentication starts with user trying to log on from a client computer providing the username and password. The following steps will occur:

1. The client application hashes user’s password (with HMAC-MD5) and then discard the password. Then the client only sends the username to the server;
2. The server generates a challenge (16-byte random number) and sends it to the client;
3. The client encrypts this challenge with the hash (of user’s password from step 1), and send the result back to the server as response (encrypted challenge);
4. The server sends the following three items to the domain controller:
   • username (in plain text)
   • the challenge it had sent to the client
   • the response from the client (encrypted challenge)
5. The domain controller has the hash of user’s password stored, and retrieved (by username). It uses the hash to encrypt the challenge by itself and get its own version of encrypted challenge. It then compares it with the one passed on from the server. If identical, the user is authenticated and the domain controller notifies the user;
6. The server sends the appropriate response back to the client.

Essentially, NTLM mechanism is comparing the result encrypted by the client the result encrypted by the credential stored in itself. The results are expected to be identical because the encryption key – hash of user’s password – should be identical. In this whole process, there is no password transmission on the wire. However, the disadvantage with this challenge-response based mechanism is that it does not let client authenticates the server, and is therefore considered less secure. This is what Kerberos is built to address.

Kerberos Authentication

Kerberos is a ticket-based authentication mechansim. In Kerberos, a Key Distribution Centre (KDC) consists of AS (Authentication Service) and TGT (Ticket Granting Service). Authentication takes place in the following steps:

The authentication starts with user trying to log on from a client computer. Note that the “client” here refers to Kerberos client relative to Kerberos server (KDC). The client machine can serve as application server that runs kerberos library. The user provides username and password.

1. The client sends KRB_AS_REQ as plain text to AS including:
   • username
   • timestamp
2. AS verifies timestamp, and validates username exists. If timestamp is too far (i.e. over 5 min) from current time, or user is not a legal principal, KRB_AS_REQ will be rejected. Otherwise, AS generates a random TGS session key and uses it to build a TGT. In KRB_AS_REP AS sends two messages to the client:
   • Message 1 is the TGT, which is encrypted with TGS secret key (so the client cannot decrypt TGT). It includes:
     • username
     • TGS name
     • timestamp
     • client network address
     • lifetime of TGT
     • TGS session key
   • Message 2 is encrypted with client secret key (stored in AS)
     • TGS name
     • timestamp
     • lifetime
     • TGS session key (same as message 1)
3. The client receives both message 1 (TGT) and message 2. It decrypts message 2 with its own secret key and obtains TGS session key. For message 1 (TGT), the client cannot decrypt it. The client simply stores it in the credential cache. Then the client prepares two messages to send to the KDC:
   • Message 3 contains:
     • TGS name
     • lifetime
     • TGT (message1, encrypted with TGS secret key)
   • Message 4 (aka Authenticator) is encrypted with TGS session key and contains:
     • username
     • timestamp
4. KDC processes message 3, determines TGS name is valid and forward TGT to TGS. TGS decrypts TGT using its own secret key and obtains TGS session key, along with username and timestamp. Then TGS decrypts message 4 using the newly obtained TGS session key, in order to get username and timestamp from message 4 as well. At this point, TGS has two sources of username and timestamp: one from TGT; the other from message 4. The TGS make sure they are identical, check if TGT is expired, and confirm that authenticator is not in the cache (to prevent replay). If all checks pass, the TGS then generates a random service session key. It will send two messages back to the client:
   • Message 5 (the service ticket) is encrypted with service secret key (stored in TGS) and contains:
     • username
     • service name
     • timestamp
     • client network address
     • lifetime
     • service session key
   • Message 6 is encrypted with the TGS session key containing:
     • service name
     • timestamp
     • lifetime
     • service session key
5. The client receives message 5 and message 6 but it cannot decrypt message 5. The client cached TGS session key from previous step so it can decrypt message 6 and obtain service session key. Now the client contact the server by sending the following two messages:
   • Message 7: a new authenticator message encrypted with service session key that contains:
     • username
     • timestamp
   • Message 8: the same as message 5, encrypted with service secret key
6. The server now receives message 7 and message 8. It decryptes message 8 to get service session key, along with username, service name, timestamp, etc. Using the service session key it descrypts message 7 to get a second source of username and timestamp. Similar to what happened in TGS, now the server compares username from the authenticator and from the ticket, checks if ticket is expired, and confirms that authenticator is not already in cache (to prevent replay attack). If all checks turn out okay. The service confirms its identity to the client with:
   • Message 9: an authenticator message encrypted with service session key that contains:
     • service name
     • timestamp
7. Lastly, the client receives message 9 and decrypts it with the service session key in cache. The client then confirms the service name and timestamp are valid/expected. If they are good, the authentication is completed and the client starts to communicate with the server.

The steps above borrowed some information from this page. The Keberos authentication process involves many steps and several keys:

• session (shared) keys:
  • TGS session key
  • Service session key
• secret keys:
  • client secret key
  • TGS secret key
  • Service secret key

By using these keys, no password is ever transmitted across the wire in the clear. The client and the server authenticate each other (mutual authentication). With a trusted third party, Kerberos ensures that the service ticket is only used by the intended client, and that only the intended server can validate the requested service ticket. Although, this sounds similar in two way authentication in TLS handshake, Kerberos does not encrypt the application traffic, neither is it intended to.

Kerberos Implementations

The most popular implementation is MIT Kerberos 5. The other well-known implementation is Heimdal. In addition, it is supported as API in GSS-API. In Windows applications, SSPI (Security Support Provider Interface) provides similar functionality to the GSS-API. SSPI can be viewed as Microsoft’s implementation of GSS-API, which can be virtually regarded as Kerberos API.

Kerberos is supported by many application protocols through GSS-API. If you build a Linux application, the server where the application is hosted acts as Kerberos client (i.e. requiring krb-libs package and /etc/krb5.conf configured correctly) to interact with customer’s Active Directory environment. If you need to configure Kerberos servers, you will need other packages such as krb5-admin-server, krb5-kdc, and krb5-user.

This page lists some pros and cons of Kerberos. Some important takeaways are:

• Kerberos is legacy protocol, complex to set-up and maintain. It requires user accounts, user clients and the services on the server to all have a trusted relationship to the Kerberos token server. All must be in the same Kerberos realm or in domains that have a trust relationship between each other.

• Kerberos cannot be used in a scenario where users want to connect to services from unknown/untrusted clients as in a typical Internet or cloud computing environment, where authentication provider typically does not have knowledge about the users client system. This implies Kerberos does not work will with modern REST applications and Authentication Methods such as SAML, and OAuth 2.0

• Kerberos was created to accomplish authorization back in the days when no-one used a secure network connections.