NServiceBus Message Property Encryption with AWS KMS

Brad Jolicoeur

06/29/2020

One of the many features of NServiceBus is the ability to encrypt specific properties/fields in your messages. NServiceBus property encryption ensures sensitive data in messages are encrypted at-rest as well as obscured from system administrators when they end up in your error queue. Property encryption also has the benefits of leaving non-sensitive data in the message available for debugging purposes.

NServiceBus applies property encryption/decryption as part of the NServiceBus message handling pipeline and once you have your conventions configured will automatically encrypt/decrypt your message properties on send/receive of messages. This is convenient since it keeps encryption logic out of your business logic and will ensure encryption is implemented consistently.

While NServiceBus automatically applies message property encryption/decryption based on the conventions you configure, it leaves it up to you to provide a mechanism for persisting the array of keys it needs to perform the encryption. This may seem like a lack of functionality at first, but in reality every company has a unique and detailed strategy for securing encryption keys. Enabling you to easily implement the specific requirements of your company is a much more useful feature than providing an implementation in this case.

In this article we will implement a Proof-of-concept NServiceBus IEncryptionService that leverages AWS Key Management Service (KMS) to manage the keys as well as perform the encryption. KMS is designed to manage your cryptographic keys and control their use inside of the AWS ecosystem. If you are leveraging AWS, then KMS is the perfect choice to store your encryption keys for NServiceBus. Other cloud providers have key management services like KMS that can be implemented in a similar way.

Since this is a proof-of-concept we will be utilizing LocalStack to make it easy to pull down the example and test it. If you need more information on getting started with LocalStack, check out our article Using LocalStack for NServiceBus Development.

Important Note: this article will reference keys that were generated for demonstration purposes. Do not use these keys for anything important.

Configuring local KMS

We are going to use Local LMS (LKMS) docker image instead of the KMS service included in LocalStack. There is better documentation on the configuration of LKMS and as a result easier to work with.

You will need to include this section to your overall docker-compose file for LKMS. The init folder will contain your seed.yaml file that you will configure next.

  kms:
    image: nsmithuk/local-kms
    volumes:
      - ./init:/init
      - ./data:/data
    ports:
      - 8081:8080

Now create a seed.yaml file with encryption key in init folder that you mapped in the docker-compose file. This file contains the test keys that will be used to complete the encryption.

Keys:
  - Metadata:
      KeyId: bc436485-5092-42b8-92a3-0aa8b93536dc
    BackingKeys:
      - 5cdaead27fe7da2de47945d73cd6d79e36494e73802f3cd3869f1d2cb0b5d7a9

Aliases:
  - AliasName: alias/testing
    TargetKeyId: bc436485-5092-42b8-92a3-0aa8b93536dc

This is a simple seed.yaml file with some example entries. It is possible to set aliases for your keys as well as simulate key rotations with KMS using the seed.yaml file that you may want to try out. See the LKMS project page for details.

Once you have docker-compose.yml and seed.yaml configured, you can start everything up by using the docker-compose up command while in the same path as your docker-compose.yml file.

Implementing IEncryptionService

Now that our infrastructure is running, we can work on implementing the NServiceBus IEncryptionService interface.

The first step will be to add the following NuGet packages to our project

  • NServiceBus.Encryption.MessageProperty
  • AWSSDK.KeyManagementService

After we have added the NuGet packages we will create a class named KMSEncryptionService that inherits the NServiceBus IEncryptionService interface. This is where we will implement the KMS encrypt/decrypt.

To start our implementation, we are going to Inject IAmazonKeyManagementService and an encryption key identifier into the service.

private readonly IAmazonKeyManagementService _client;
private string _encryptionKeyIdentifier;

public KMSEncryptionService(string encryptionKeyIdentifier, IAmazonKeyManagementService client)
{
    _encryptionKeyIdentifier = encryptionKeyIdentifier;
    _client = client;
}

Now we will implement the Encrypt method of IEncryptionService. This method will send the unencrypted string value to KMS for encryption. Once the results are returned from KMS the results are converted to Base64String and returned.

public EncryptedValue Encrypt(string value, IOutgoingLogicalMessageContext context)
{
                
    var encrypted = _client.EncryptAsync(new EncryptRequest
        { KeyId = _encryptionKeyIdentifier, 
            Plaintext = new System.IO.MemoryStream(Encoding.UTF8.GetBytes(value)) 
        } ).GetAwaiter().GetResult();

    string base64Value = encrypted != null ? Convert.ToBase64String(encrypted.CiphertextBlob.ToArray())  : null;

    context.Headers[EncryptionHeaders.RijndaelKeyIdentifier] = _encryptionKeyIdentifier;

    return new EncryptedValue
    {
        EncryptedBase64Value = base64Value
    };
}

Then we implement the Decrypt method. This method gets the encryption key id from the NServiceBus message headers. The NServiceBus encryption implementation has an unfortunate name for the KeyIdentifier message header RijndaelKeyIdentifier. There may be a way to override this name, but we are going to forge ahead and ignore it for now since NServiceBus automatically populates that header for us.

Once we have the Key id, then we can convert it from a Base64String to a byte array and send the encrypted value to the KMS service to be decrypted with the key id. The returned byte array is then converted to a string and returned.

public string Decrypt(EncryptedValue encryptedValue, IIncomingLogicalMessageContext context)
{
    if (encryptedValue == null || String.IsNullOrEmpty(encryptedValue.EncryptedBase64Value))
        return null;

    if (!context.Headers.ContainsKey(EncryptionHeaders.RijndaelKeyIdentifier))
        return null;

    var decryptlabel = context.Headers[EncryptionHeaders.RijndaelKeyIdentifier];

    var decryptRequest = new DecryptRequest { KeyId = decryptlabel };
    var value = Convert.FromBase64String(encryptedValue.EncryptedBase64Value);
    decryptRequest.CiphertextBlob = new System.IO.MemoryStream(value);

    var response = _client.DecryptAsync(decryptRequest).GetAwaiter().GetResult();

    if(response != null)
    {
        return Encoding.UTF8.GetString(response.Plaintext.ToArray());
    }

    return null;
}

The full implementation of KMSEncryptionService is available in the example project GitHub repo.

Configuring Endpoint

Once you have your Encryption Service implementation, adding it to your endpoint configuration is relatively straight forward.

Remember that all of the endpoints that produce and consume messages within your system will need to use the same configuration so that they can all encrypt/decrypt your messages in a consistent way.

There are two parts of the configuration. The first part of the configuration is the connectivity with KMS and setting the default key to use. The second part is implementing an unobtrusive convention that will automatically pick up properties that contain a specific name and encrypt/decrypt them.

In our case we are going to encrypt/decrypt any property that ends with 'Encrypted' or ends with 'AccountNumber'. You can really use any naming convention that makes sense for you system. If you have a consistent naming convention this can be really powerful since you can create a convention that includes any property your company policy requires to be encrypted.

Note: Since we are running everything local and to keep things simple, the UseHttp=true flag is being used. You would never want to do this in production since it would transmit the data you are trying to protect without transport encryption.

private static void ConfigurePropertyEncryption(EndpointConfiguration endpointConfiguration)
{
    //this is a key stored in KMS; for reference this same value is in the init\seed.yaml file for kms simulator config
    var defaultKey = "bc436485-5092-42b8-92a3-0aa8b93536dc"; //Todo: should not be hardcoded

    var kmsClient = new AmazonKeyManagementServiceClient(new AmazonKeyManagementServiceConfig
    {
        //This is for localstack and is not needed if targeting AWS directly
        UseHttp = true, //do not use this option for anything important!
        ServiceURL = "http://localhost:8081"
    });

    var kmsEncryptionService = new KMSEncryptionService(defaultKey, kmsClient); 

    //message property convention will encrypt/decrypt any property that 
    // ends with Encrypted or AccountNumber.
    endpointConfiguration.EnableMessagePropertyEncryption(
        encryptionService: kmsEncryptionService,
        encryptedPropertyConvention: propertyInfo =>
        {
            return propertyInfo.Name.EndsWith("Encrypted")||propertyInfo.Name.EndsWith("AccountNumber");
        }
    );
}

Inspect Encrypted Property

For our first test, we are going to send a message with encryption disabled and show that AccountNumber is not encrypted. To inspect the message start the web project alone without the saga and worker endpoints. This will produce a message that we can view using the AWS CLI as shown below.

Executing the sqs recieve-message command against the LocalStack service by using the following command:

aws --endpoint-url=http://localhost:4576 sqs receive-message --queue-url http://localhost:4576/123456789012/Example-PaymentSaga

This is the raw message returned from SQS that does not have property encryption enabled. Notice that there is no RijndaelKeyIdentifier header in the message.

"MessageId": "d10a1282-546b-4bb4-bd9f-8b7dbc46a2f6",
"ReceiptHandle": "d10a1282-546b-4bb4-bd9f-8b7dbc46a2f6#b5a1c04b-16f9-474f-8f5b-6e365078772b",
"MD5OfBody": "ddb5888b1ad0549f871c66598ab27d1b",
"Body": "{\"Headers\":{\"NServiceBus.MessageId\":\"577a99ac-47ac-4020-9f7c-abb80103165d\",\"NServiceBus.MessageIntent\":\"Send\",\"NServiceBus.ConversationId\":\"ea2cf31b-46b9-4179-a887-abb80103165d\",\"NServiceBus.CorrelationId\":\"577a99ac-47ac-4020-9f7c-abb80103165d\",\"NServiceBus.OriginatingMachine\":\"LAPTOP-3ACB4PKS\",\"NServiceBus.OriginatingEndpoint\":\"Example.WebApp\",\"$.diagnostics.originating.hostid\":\"d7d63ca165b846f0d986d655dd47ebfc\",\"NServiceBus.ReplyToAddress\":\"Example-WebApp-InstanceID\",\"NServiceBus.ContentType\":\"application/json\",\"NServiceBus.EnclosedMessageTypes\":\"Example.PaymentSaga.Contracts.Commands.ProcessPayment, Example.PaymentSaga.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null\",\"NServiceBus.Version\":\"7.2.0\",\"NServiceBus.TimeSent\":\"2020-05-11 15:43:18:493845 Z\"},\"Body\":\"77u/eyJSZWZlcmVuY2VJZCI6ImU1ZmU2ZTgwLTFjMDktNDljOC1hMDI1LWMwMmY0MzQ3YjI2MSIsIkFtb3VudCI6MTAwLjQ1LCJBY2NvdW50TnVtYmVyRW5jcnlwdGVkIjoiMTIzNDU2IiwiUm91dGluZ051bWJlciI6IjU1NTU1NSIsIlJlcXVlc3REYXRlIjoiMjAyMC0wNS0xMVQxNTo0MzoxOC40OTMwMzA4WiJ9\",\"S3BodyKey\":null}"

Once we decode the message body we can see that the account number is not encrypted.

{
    "ReferenceId":"e5fe6e80-1c09-49c8-a025-c02f4347b261",
    "Amount":100.45,
    "AccountNumberEncrypted":"123456",
    "RoutingNumber":"555555",
    "RequestDate":"2020-05-11T15:43:18.4930308Z"
}

Now enable the encryption by adding the following line in your endpoint configuration. This calls our property encryption configuration method we created earlier.

ConfigurePropertyEncryption(endpointConfiguration);

Now run the same test you completed before and use the AWS CLI to observe your new message.

aws --endpoint-url=http://localhost:4576 sqs receive-message --queue-url http://localhost:4576/123456789012/Example-PaymentSaga
{
    "Messages": [
        {
            "MessageId": "56e4bb5c-f645-4f48-b5cb-769ec0cf1b0d",
            "ReceiptHandle": "56e4bb5c-f645-4f48-b5cb-769ec0cf1b0d#e18f91be-5df0-4583-83b4-1679aa225e03",
            "MD5OfBody": "c185917fdbf58a53b7e417fd7993cf38",
            "Body": "{\"Headers\":{\"NServiceBus.MessageId\":\"83ddb66b-8ef7-4571-b5cf-abb80100db40\",\"NServiceBus.MessageIntent\":\"Send\",\"NServiceBus.ConversationId\":\"b8fce273-e5c1-4156-862e-abb80100db4b\",\"NServiceBus.CorrelationId\":\"83ddb66b-8ef7-4571-b5cf-abb80100db40\",\"NServiceBus.OriginatingMachine\":\"LAPTOP-3ACB4PKS\",\"NServiceBus.OriginatingEndpoint\":\"Example.WebApp\",\"$.diagnostics.originating.hostid\":\"cbf2c09b3652d8c563d48395a8b58c5d\",\"NServiceBus.ReplyToAddress\":\"Example-WebApp-InstanceID\",\"NServiceBus.RijndaelKeyIdentifier\":\"bc436485-5092-42b8-92a3-0aa8b93536dc\",\"NServiceBus.ContentType\":\"application/json\",\"NServiceBus.EnclosedMessageTypes\":\"Example.PaymentSaga.Contracts.Commands.ProcessPayment, Example.PaymentSaga.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null\",\"NServiceBus.Version\":\"7.2.0\",\"NServiceBus.TimeSent\":\"2020-05-11 15:35:11:356034 Z\"},\"Body\":\"77u/eyJSZWZlcmVuY2VJZCI6IjRjN2RhNzQzLTkxODMtNDRiNC05MDFkLWVhYTIwNzIxYzZhZCIsIkFtb3VudCI6MTAwLjQ1LCJBY2NvdW50TnVtYmVyRW5jcnlwdGVkIjoiUzJGeWJqcGhkM002YTIxek9tVjFMWGRsYzNRdE1qb3hNVEV4TWpJeU1qTXpNek02YTJWNUwySmpORE0yTkRnMUxUVXdPVEl0TkRKaU9DMDVNbUV6TFRCaFlUaGlPVE0xTXpaa1l3QUFBQUJCZGtmZnFCOGxiQVdJRk9uaTI0U0hudlBBclZCMmpVb3hzTm5EWWRCM1VRM1JAIiwiUm91dGluZ051bWJlciI6IjU1NTU1NSIsIlJlcXVlc3REYXRlIjoiMjAyMC0wNS0xMVQxNTozNToxMS4xMzc2NzUzWiJ9\",\"S3BodyKey\":null}"
        }
    ]
}

Notice the key id in the header RijndaelKeyIdentifier\":\"bc436485-5092-42b8-92a3-0aa8b93536dc\". If this header is not in your message the decrypt method is not triggered.

Once we decode the message body we can see that it contains the Account Number as an encrypted and base 64 encoded string.

{
  "ReferenceId":"4c7da743-9183-44b4-901d-eaa20721c6ad",
  "Amount":100.45,
  "AccountNumberEncrypted":"S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAABBdkffqB8lbAWIFOni24SHnvPArVB2jUoxsNnDYdB3UQ3R@",
  "RoutingNumber":"555555",
  "RequestDate":"2020-05-11T15:35:11.1376753Z"
}

Resources