Configuring an F5 Load Balancer for Mutual TLS

In a previous post, Mutual TLS: Stuff you should know, I discussed Mutual TLS and how it enables your Connect (or eventNotification) listener to authenticate the HTTPS client (the DocuSign platform).

Connect webhook listeners should always authenticate the client and use access control to ensure that the client is DocuSign. Either Mutual TLS or the notification payload’s digital signature can be used.

In this post, I’ll cover the configuration of an F5 Load Balancer for Mutual TLS and access control.

Acknowledgment

My thanks to George Balafoutis, DocuSign Director of Technical Architecture, for his F5 expertise and help in preparing this post.

Network Configuration

Typically, F5 Big-IP load balancers are placed between the Internet and a pool of one or more servers. Clients (web browsers and computers such as the DocuSign Connect system) make requests via the Internet to the F5. The F5 passes the requests onwards to the servers in the pool, balancing the requests amongst the servers.

For HTTPS requests, the F5 can act as the terminus for the SSL/TLS session, offloading the SSL/TLS cryptography work from the servers. This is the “offloading” configuration. An F5 can also be configured to “pass-through” the HTTPS conversation to the back-end servers. Offloading configurations are more common and this post focuses on that configuration.

In the offloading configuration, the F5 “clients” are the clients on the Internet. The F5 acts as a server to them. The F5’s “servers” are the pools of servers that will ultimately handle the incoming requests. The F5 acts as a client to its servers.

A Disclaimer: Rely on your experts

F5 systems are complex. This post is designed to help an F5 expert configure Mutual TLS. Your F5 configuration should always be reviewed and tested by your networking and Information Security departments. Don’t rely on this post to create a secure system.

F5 Software Versions

For the tests this post is based on, we used an F5 running software version 11.5.1. Version 12 should also work. Versions 9 and 10 will not work. A recent customer problem was solved when they upgraded their F5 from version 10.2.4 to 11.5.4. The earlier version did not support client certificates that use SHA2 Signing Hash Algorithms.

Configure the F5 as a vanilla load balancer

Step 1 is to configure your F5 to load balance your pool of one or more servers that host your Connect listeners. The resulting configuration should enable, for example, a URL of https://listener.example.com/listener that is served by the F5. The F5 will handle the TLS work, and load balance the requests to a pool of servers: http://listener1.it.example.com, http://listener2.it.example.com, etc. Note that the it.example.com domain does not need to be visible outside of the company.

Configure the Connect configuration to use https://listener.example.com/listener and check that it works properly.

Configure Mutual TLS: Download the root cert

On the F5, open the Import SSL Certificates and Keys screen.

Enter data into these fields:

• Import type: Certificate
• Certificate Name: “DS Connect root” or similar
• Certificate Source: copy/paste the Connect 2048 bit root certificate from Symantec.

An alternative 1024 bit root cert is also available: use the "Root 2" cert from https://www.symantec.com/theme/roots. If the 2048 bit cert doesn’t enable verification, try the 1024 bit cert.

Enable Mutual TLS on DocuSign

Check or update your Connect or eventNotification configuration on DocuSign to enable Mutual TLS. The Mutual TLS blog post includes the details.

Enable Mutual TLS on F5

Open the Client SSL Profile

Update the Client Authentication section as shown below. The other sections can be left with their default values.

The Trusted Certificate Authorities field is set to the F5 default CA bundle. The standard bundle includes a root cert that can verify the DocuSign Connect client cert. In response to the certificate request, DocuSign sends its cert and two intermediate certs.

The Advertised Certificate Authorities field is set to the Connect root certificate that you downloaded previously. This certificate’s distinguished name will be sent to the client (DocuSign) via the Certificate Request message during the TLS handshake.

The Access Control iRule

Mutual TLS authenticates the client. But that’s not enough. To add security, you must also use access control to limit access to just DocuSign.

We used the following iRule set. It includes several diagnostic log commands that can be activated (uncommented-out) during installation and testing.

The iRule stores the MD5 hash of the incoming certificate in the cert_hash variable, and the client’s Common Name in the cSSLSubject variable using format CN=common_name.

Next, the iRule sets the Expected_hash value from the AuthThumb lookup table on the F5.

If the Expected_hash doesn’t match the cert_hash, then the connection is rejected.

when CLIENTSSL_HANDSHAKE {
    set subject_dn [X509::subject [SSL::cert 0]]
    set cert_hash [X509::hash [SSL::cert 0]]
    set cSSLSubject [findstr $subject_dn "CN=" 0 ","]
    #log local0. "Subject = $subject_dn, Hash = $cert_hash and $cSSLSubject"
    # Check if the client certificate contains the correct CN and
    # Thumbprint from the AuthThumb lookup table
    set Expected_hash [class match -value $cSSLSubject eq AuthThumb]
    #log local0. "Check Expected Hash = $Expected_hash, Hash received = $cert_hash"
    if { $Expected_hash != $cert_hash } {
        #log local0. "Thumbprint presented doesn't match mythumbprints. Expected Hash = $Expected_hash, Hash received = $cert_hash"
        reject
    }
}

Add the AuthThumb lookup table

Create the lookup table AuthThumb by using the iRules:Data Group List screen.

We’ll fill in a key/value pair later.

Add the iRule for Access Control

Add the iRule shown above to the F5. Then add the rule to the configuration by using the Resources tab:

Send a test Connect notification

Send a test Connect notification message and check the F5 log. You should see a log entry with the certificate’s MD5 hash value and its common name:

The presence of the DocuSign certificate’s common name in the log shows that Mutual TLS worked.

Copy the MD5 hash value and the lookup string “CN=demo.connect.docusign.net” to your AuthThumb lookup table:

Test and Go!

Conduct some more tests to ensure that notifications are being delivered successfully.

Check that a certificate with the wrong common name or wrong MD5 hash value will be rejected. When the F5 rejects a connection, the Connect failure log will show “Error - The request was aborted: Could not create SSL/TLS secure channel.”

Troubleshooting

iRule problems

The F5 iRule compiler checks for the presence of the AuthThumb lookup table at compile time. Therefore, you need to create the table before you can start your tests to determine the MD5 hash value of the correct certificates.

Certificate problems

In our configuration, we’re sending the Distinguished Name from a root cert for the Connect certificate, and verifying the Connect certificate by using the standard F5 ca_bundle. Using the 2048 root cert to verify the Connect certificate did not work on F5, perhaps because the F5 was trying to verify the entire intermediate cert chain sent by DocuSign. It did work on Apache and Nginx. We did not try the 1048 bit cert.

Client Certificate not Sent

The first issue is for DocuSign to send its client certificate in response to the F5’s CertificateRequest TLS handshake message. Test this without the iRule on. If DocuSign doesn’t send its certificate, check that the F5 version is 11.5.1 or more recent.

Logging

You can activate logging on the F5 to help debug problems. Look at the TLS handshake protocol which is used to set up the encrypted channel.

Comments? See you in San Francisco!

Comments on this post are welcomed. And don’t miss out: the DocuSign Momentum Developer Conference is the place to be May 3-4. The registration fee for developers is $0, zero, free, you’re welcome! Tell you manager that you’ll learn new techniques for integrating DocuSign with your line of business systems, saving your company tens of thousands of dollars, euros, pounds sterling, etc. (It’s true, ask your account team for the statistics.) Visit the DocuSign Momentum conference site and view the Developer Track Agenda. See you there!