Building best practices webhook listeners, part 4: Behind the firewall with AWS

By Larry Kluger and Chen Ostrovski

This blog post is part 4 of a multi-part series on how to build webhook listeners. If you haven’t already read our previous installments, use the links below to catch up on parts 1, 2 and 3.

In part 2 of this series, we discussed how you can use an asynchronous microservices architecture to process Connect webhook notifications. The following diagram shows the path for a notification message with this architecture.

Messaging path in a webhook-driven architecture

Part 3 discusses the benefits of implementing this architecture.

In this post, we’ll cover an AWS PaaS (Platform as a Service) implementation of this architecture to receive Connect and envelope-level (eventNotification) webhook notifications from behind the firewall.

Costs

Quickly receiving webhook notifications from behind the firewall sounds good, but how much does it cost? Good news: the two AWS services used, Lambda serverless functions and the Simple Queuing Service (SQS) each offer free usage tiers.

The Lambda free tier covers the costs of processing a million webhook notifications a month. Together with the SQS free tier of 1 million messages per month, your net AWS cost is $0.00 per month for the first million notification messages (taxes included). Need more than a million notification messages per month? Costs are still very low: well under $5 per million notifications. Note that the AWS free tiers are available month after month; it is not trial pricing. Since AWS can change their pricing plans at any time, consult the AWS website for current information.

DocuSign environments and integration keys

First, install and test your application using the DocuSign developer sandbox, the demo platform. During this phase of development, you will also create an integration key if your application doesn’t already have one.

After you have completed your testing, use the Go Live process to promote your integration key to the production systems. You can then create another instance of your application for the DocuSign production system. The original application can be left intact for regression testing and further development. This means that you will have two sets of AWS functions, queues, and related objects: one for use with the DocuSign demo platform, and one for use with the DocuSign production systems.

Installing and configuring the AWS Connect code example

The Connect code example for receiving webhook notifications from behind the firewall via AWS has two components, the Listener and the Worker. Multiple versions of the worker component are provided for different languages.

The Listener

The listener is a serverless function that runs on the AWS Lambda serverless PaaS. It receives the notification messages from DocuSign, checks their validity, and enqueues them on a private SQS queue. The code example is written in JavaScript for the Node.js Lambda environment. See the code repository, connect-node-listener-aws.

The Worker

The worker is a command-line application that receives the notifications from the SQS queue and then downloads the envelope’s documents from DocuSign via the eSignature REST API.

Code examples for the worker application are available from GitHub for:

Creating the queue

First, download the listener code example. Its Readme and Long form installation instructions provide full information on installing and configuring the example.

After creating an AWS account, open the Simple Queuing Service (SQS) console and create a queue. You can create either a regular or FIFO (First In, First Out) queue. To create a more flexible integration with DocuSign, a regular queue is recommended.

A regular queue may sometimes (rarely) deliver a notification message more than once, and a notification message can arrive in a different order than what was received from DocuSign. Architecting and building your application to handle these situations is often easily done and will enable you to create a less brittle, more self-healing application, better able to handle the occasional transient network problem or hiccup.

For example, don’t use an envelope-complete notification to directly trigger shipment of goods. Instead, envelope-complete should trigger an examination of the order: ship the goods if they are not yet scheduled to be shipped.

Listener installation

Next, use the AWS Lambda console to create your Lambda microservice. It will receive the incoming notifications from DocuSign and enqueue them. You will configure the listener with the SQS information you received in the prior step.

You will also add an AWS API Gateway trigger to the Lambda function. The long form instructions provide step-by-step information. Record the public URL for your function.

Connect Configuration

Configure the Connect subscription (or eventNotification object):

To not include the envelope documents with the notification messages
To use the AWS serverless function’s API Gateway trigger URL
To use a single HMAC message signature
To include Basic Authentication

The Connect subscription is also configured to select the envelope and/or recipient events that will trigger event notifications to your application. Most applications only select the Envelope Complete event.

Documents cannot be included in the notification messages because that would cause them to exceed the maximum SQS message size, 256 KB. If the other data (tab values and meta-data) in the notification message will cause the SQS message to be greater than 256 KB, you must update the Listener serverless function to delete some of the webhook data to ensure that the maximum SQS message size is never exceeded. Remember that your application, using the envelope ID, can look up any quantity of data about the envelope via the eSignature API.

Update the listener configuration in AWS with the Basic Authentication and HMAC settings from the Connect configuration.

The Worker application

Install one of the worker code examples listed above on your development system. The development system must have regular access to the Internet and DNS lookup capabilities from behind your firewall. It does not need any other special networking configuration, it does not need its own IP address on the public Internet, nor to be on your network’s DMZ, etc.

As discussed in each of the worker code examples' Readme files and long form instructions, the worker must be fully configured with settings for DocuSign and for the SQS queue. This includes a DocuSign integration key and a user GUID from the DocuSign account.

The user will be impersonated by the worker application as it downloads each envelope’s documents to the worker’s system. For most use cases, the user GUID will need administrator privileges to enable it to download the documents from any user’s envelopes.

Testing

Start the worker application on your development machine. Then send an envelope and update it to trigger a Connect or eventNotification message. You should then see a logging message from the worker application, and the envelope’s documents should be downloaded from DocuSign to your development machine.

During testing, you may want to set Connect to send a notification message when the envelope is sent. This will enable a notification message to be sent as soon as you send a test envelope, minimizing your work to test the overall application. Of course, be sure to update the Connect trigger events when you have completed notification message testing.

The worker code examples include queue testing software that will rapidly send test messages to the serverless function and check that they are correctly received by the worker. See the README files in the worker repositories on GitHub for more information.

Debugging

In an organized fashion, check each leg of the message flow in order:

Is Connect attempting to send the notification message? Check the Connect logs.
Is Connect succeeding in sending the notification message? Check the Connect log for errors; check your Lambda function’s log to see if the messages have been received.
Has the Lambda function accepted the message’s Basic Authentication and HMAC values? Check the Lambda function’s logs.
Has the notification message been enqueued onto the SQS queue? Check the queue statistics via the SQS management console.
Has the worker been able to communicate with the SQS queue and dequeue waiting messages? Check the SQS queue statistics for de-queued messages and the worker’s diagnostic output.

Congratulations!

Congratulations on receiving webhook notifications from behind your firewall. This architecture will enable you to build more agile and performant applications at a low cost.

See you in San Francisco!

Please join me and my colleagues for a great developer day at the DocuSign Momentum conference on March 4. Registration for developers is free!

Larry Kluger is the Lead Developer Evangelist for DocuSign. He has many years of tech industry experience as a software developer, product manager, and evangelist. An award-winning speaker, he enjoys giving talks and meeting members of the developer community.

Chen Ostrovski is an Engineering student at Ariel University in Israel. Chen wrote many of the Connect AWS code examples during her Engineering Internship at DocuSign.

Additional resources

Developer posts about Connect webhooks:

Additional developer resources:

Author

Larry Kluger

Lead Product Manager, Partner Platform

Published

January 27, 2020