Serverless Secrets Management

Getting started securing secrets in AWS Lambda is confusing at best and downright frightening at worst. You are faced with understanding and comparing KMS, Parameter Store, Secrets Manager, and Secure Environment Variables. You need to consider whether you are going to be retrieving secrets at run time, deploy time or a hybrid. And when you do retrieve the secrets you also are faced with deciding on whether to retrieve them decrypted or encrypted for later manual decryption.

With so many options, it’s way too easy to get this wrong. To make matters worse, there are many solutions, plugins, blogs, and resources out there that conflict in their recommendations. In this post we’ll look at some common sense security goals, the options available to the Lambda developer using the Serverless Framework, and how these options stack up to the security goals.

The Secret Management Problem

When developing an application, whether it is serverless or not, eventually you are confronted with how to deal with secrets that are needed for your application to run but you don’t want exposed to anyone else. These are things like API keys or database credentials. Common solutions to secret management are:

1. Hard code your secret in source control. You’re not really considering this are you? Albeit, if you store encrypted secrets in source control this is OK.
2. Read your secrets from Environment Variables. Popular with The Twelve-Factor App. Commonly the Environment Variables are populated from the ‘secure variables’ section that most, if not all, Continuous Integration services provide. This keeps the secrets out of source control and are fairly easy to manage via the CI provider.
3. Retrieve your secrets from a secret management service at runtime. Very secure but can be complex to setup. This includes things like AWS Parameter Store, AWS Secrets Manager, and Hashicorp Vault.

The Secret Management Goals

The requirement for secret management will vary based on application to application. For the purposes of this analysis I’ll be looking at the following functional and non-functional requirements:

1. Secrets must be encrypted at rest
2. Secrets must be encrypted in transit
3. Ease of Use
4. Pricing

Keep in mind, that this is not an exhaustive list! Certainly, other things like PCI compliance, principle of least privilege, auditable history, and performance impacts could be applicable for your application.

Deciphering AWS Secret Management Services

AWS provides three services for serverless secret management: Lambda itself via Environment Variables, System Manager Parameter Store, and Secrets Manager. All three of these are built upon the AWS Key Management Service (KMS). Let’s start by looking at KMS.

KMS

A common misconception is that KMS can store your arbitrary secrets. This is NOT what KMS does. KMS primarily does two things:

1. It provides an extremely secure way to store symmetric encryption keys.
2. It provides APIs that allow an authorized IAM user to utilize those encryption keys without ever exposing the actual keys.

Note that important difference, KMS stores encryption keys, not the secrets themselves. This is why KMS is the common foundation to other AWS based secret management solutions.

The encryption keys stored in KMS really are designed to be accessible by nobody. The encryption keys live in a FIPS 140-2 Hardware Security Module (HSM). They have external audits by the NIST Cryptographic Module Verification Program. There is Ethan Hunt proof physical security that will zero out all keys if the pick-resistant locks are broken into. The encrypted secrets all utilize the AES-256 algorithm that would take all of the computing power on earth more time than the age of the universe to brute force crack¹.

The security in KMS is appropriately intense as it really is a foundational component in the AWS security story with integrations across more than 40 other AWS services.

How Secret Management Solutions use KMS

As far as the options for secret management, each will eventually invoke the KMS APIs to do the same two things²:

1. Send the plaintext secret provide by the user to the KMS Encrypt API to get an encrypted secret back for storage.
2. Send the stored encrypted secret to the KMS Decrypt API to return the decrypted secret back to the authorized caller.

In addition, all communication with KMS is fully encrypted in transit. The security risks in serverless secret management aren’t going to be with KMS directly, they are going to be how the secrets are managed when they have been decrypted as part of the deployment or runtime process. Let’s look at how Lambda, Parameter Store, and Secrets Manager can be used and misused for secret management.

Lambda Environment Variables with Plaintext Secrets

Lambda provides out-of-the-box functionality for specifying Environment Variables at deploy time that become available to the Lambda at invocation. Lambda environment variables are always encrypted via KMS while at rest within the AWS data centers. However, there is gap: There is no out-of-the-box way to ensure the secrets are encrypted while in transit during deployment of the Lambda function.

Here is a simple serverless.yml definition via Serverless Framework using environment variables:

  deploytime-env-var-function: #NOT RECOMMENDED
    handler: envvar.handler
    environment: 
      SECRETKEY: ${env:SECRETKEY} 

This is a common scenario where the secret is stored in the “secure environment variable” section of the CI provider or the developer machine. At serverless deploy time Serverless Framework will retrieve the secret via ${env:SECRETKEY} and then inject that secret into the Environment section of the auto-generated CloudFormation template used for ultimately deploying the Lambda. That right there is already has some reason for concern.

CloudFormation is not a good place for secrets. CloudFormation is not stored at rest with KMS encryption at either the origin machine or the destination AWS data center. This breaks our first security goal of always having secrets encrypted at rest.

It is possible to deploy Lambdas by other means besides CloudFormation, either by invoking the Lambda APIs or using the Lambda UI in the AWS Console. However, these alternative forms of Lambda deployment are also not recommended with plain text secrets as explained by these AWS docs:

All the environment variables you’ve specified are encrypted by default after, but not during, the deployment process… If you need to store sensitive information in an environment variable, we strongly suggest you encrypt that information before deploying your Lambda function.

AWS isn’t making any promises that it keeps the environment variables secure at deploy time no matter which deployment method is used.

Summary of specifying plaintext secrets as Lambda Environment Variables:

• Always Encrypted at Rest: No, due to CloudFormation
• Always Encrypted in Transit: No, due to AWS admitting they aren’t doing it within the datacenter.
• Ease of Use: High
• Pricing: No additional cost

Lambda Environment Variables with Ciphertext Secrets

As noted in the excerpt from the AWS docs above, Lambda does make a suggestion on how to store secrets: Encrypt them before putting them in the Environment Variables. From within the Lambda UI in the AWS Console, there are even helpers to encrypt the secret using KMS so the environment variable stores encrypted ciphertext.

The helpers then provides an autogenerated code snippet that can be copy/pasted into the application to decrypt the secret at run time.

Although it’s not practical to be using the Lambda UI for any sizable project for secret storage, it is possible to do the same approach in Serverless Framework by doing the KMS encryption manually and then store the ciphertext in the Environment Variables.

  deploytime-env-var-function: 
    handler: envvar_encrypted.handler
    environment:
      ENCRYPTED_SECRETKEY: AQICAHh429eYwvaw/MRcoBXJA3ZIzfZyHO5u4cZeDSlMNJdN8wFvQCBWjyFPWzzHsOXPiKxCAAAAZjBkBgkqhkiG9w0BBwagVzBVAgEAMFAGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM1M3h1hp78u5K+0nNAgEQgCPyGN2+YTfb+G9bwnGRCQx0v+MaqNrhKoXOp+8J3tYhnyZFDQ==. ## Generated by calling KMS Encrypt API

    iamRoleStatements:  ## Give Lambda permission to call KMS Decrypt at run time.
      -
        Effect: Allow
        Action:
          - 'kms:Decrypt' 
        Resource: !GetAtt ServerlessDemoKey.Arn

import boto3
import os
from base64 import b64decode

def kms_decrypt(secret, arn):
    client = boto3.client('kms')
    resp = client.decrypt(CiphertextBlob=b64decode(secret))
    return resp['Plaintext']

def handler(event, context):
    secret = kms_decrypt(os.environ['ENCRYPTED_SECRETKEY'])

Summary of specifying ciphertext secrets as Lambda Environment Variables:

• Always Encrypted at Rest: Yes
• Always Encrypted in Transit: Yes
• Ease of Use: Medium
• Pricing: Low ($1/mo for Custom KMS key)

Although this meets all the security requirements, it does lacks in usability. It can be difficult for an authorized user to manage the plaintext of the secrets by needing to manually deal with KMS encryption/decryption each time the secret needs to be viewed or updated.

Parameter Store

AWS Systems Manager (SSM) has a hidden gem of a service called Parameter Store. Within Parameter Store you can store hierarchical configuration data and secrets for your application. Each secret stored in Parameter Store is securely encrypted using KMS and can then be selectively shared with other AWS resources via IAM policies and APIs. The service handles all the KMS operations for you so you never have to deal with ciphertext directly but can just reference the hierarchical path of the secret.

Parameter Store has other nice benefits like providing a very useful history view showing all previous values, when the value was modified, and by whom. And to top it off, Parameter Store is free to use.

It’s a great way to store secrets but there are many ways to get it wrong when using in a Serverless context.

Retrieve Decrypted Parameter Store Secrets at Deploy Time

Serverless Framework has built-in support for Referencing Variables using the SSM Parameter Store. To decrypt the values at deploy time specifying a ~true at the end of the key will get the plaintext value of the secret for deploying to Lambda. However, this is NOT recommended for the same reasons as stated in Environment Variables section. Since the secret is being decrypted at deploy time it is going to be shoved into CloudFormation in plaintext.

  deploytime-parameter-store: # NOT RECOMMENDED
    handler: envvar.handler
    environment:
      SECRETKEY: ${ssm:/my/path/to/secretkey~true}

Unfortunately, this is a recommended way of storing secrets on the serverless blog.

Summary of Retrieving Decrypted Parameter Store Secrets at Deploy Time:

• Always Encrypted at Rest: No
• Always Encrypted in Transit: No
• Ease of Use: High
• Pricing: No additional cost if using default KMS key

Retrieve Encrypted Parameter Store Secrets at Deploy Time

This approach is similar to using using Environment Variables with Ciphertext Secrets talked about earlier. The benefit here is that you can utilize the Parameter Store to easily view/manage and update your secrets. In Serverless framework when specifying ~false at the end of the ssm key, or omitting the flag altogether, the secret will be retrieved from Parameter Store encrypted which will satisfy our encryption requirements.

  runtime-parameter-store:
    handler: kms.handler
    environment:
      ENCRYPTED_SECRETKEY: "${ssm:/my/path/to/secretkey~false}"
      SECRETKEY_ARN: "arn:aws:ssm:${self:provider.region}:920535593515:parameter/my/path/to/secretkey"  # Will explain this..
    iamRoleStatements: ## Give Lambda permission to call KMS Decrypt at run time.
      -
        Effect: Allow
        Action:
          - 'kms:Decrypt'
        Resource: !GetAtt ServerlessDemoKey.Arn

import boto3
import os
from base64 import b64decode

def kms_decrypt(secret, arn):
    client = boto3.client('kms')
    resp = client.decrypt(CiphertextBlob=b64decode(secret), EncryptionContext={"PARAMETER_ARN": arn})
    return resp['Plaintext']

def handler(event, context):
    print(kms_decrypt(os.environ['ENCRYPTED_SECRETKEY'], os.environ['SECRETKEY_ARN']))
    return "OK"

As long as the Parameter Store was configured to use the same KMS key for encrypting the secrets, the Lambda should be able to decrypt the values at runtime.

This wouldn’t be too bad of an approach, but there is one obscure detail that makes this difficult. When encrypting something using KMS there is an optional argument that can be specified called “Encryption Context” that consists of arbitrary key/values. Parameter Store happens to internally call the KMS encrypt APIs using the Encryption Context of:

{"PARAMETER_ARN": <full parameter store arn of the secret being stored>})

Now here is the tricky thing, in order to decrypt the secret via KMS, the same encryption context must be supplied. So for each secret, the ARN of the secret in parameter store must be supplied in addition to knowing the path to the secret.

Summary of Retrieving Encrypted Parameter Store Secrets at Deploy Time:

• Always Encrypted at Rest: Yes
• Always Encrypted in Transit: Yes
• Ease of Use: Low
• Pricing: No additional cost if using default KMS key

Although this is secure, the benefits of using Parameter Store in this way are offset by the complexity of passing in the custom encryption context.

Retrieve Parameter Store Secrets at Runtime

It’s possible to just leverage Parameter Store Decryption directly within the application code at runtime only. This removes the need to know of the secrets ARN as Parameter Store will handle all the KMS decryption for you when the lambda is invoked. The Lambda function will just need IAM permissions to both get the parameter from Parameter Store and decrypt the value using KMS. In this example the environment variable to be passed into the application is just the Parameter Store path to lookup.

  runtime-get-parameter-store:
    handler: parameterstore.handler
    environment:
      SSM_PATH_TO_SECRETKEY: /my/path/to/secretkey
    iamRoleStatements:
      -  
        Effect: Allow
        Action:
          - ssm:GetParameter
        Resource: 'arn:aws:ssm:${self:provider.region}:*:parameter/my/path/to/secretkey'
      -
        Effect: Allow
        Action:
          - 'kms:Decrypt'
        Resource: !GetAtt ServerlessDemoKey.Arn

import boto3
import os
import logging

def parameter_store_decrypt(key):
    client = boto3.client('ssm')
    resp = client.get_parameter(
        Name=key,
        WithDecryption=True
    )
    return resp['Parameter']['Value']

def handler(event, context):
    secret = parameter_store_decrypt(os.environ['SSM_PATH_TO_SECRETKEY']))

This solution supports all the security requirements as the Lambda is never deployed with the raw secrets.

There is some complexity in the ease of use however with any run time retrieval of secrets that isn’t reflected in the sample code above. Should the secret be stored in the global scope to minimize API calls/latency when the Lambda is warm? If so, how to invalidate the cached secret when it changes?

Summary of Retrieving Parameter Store Secrets at Runtime

• Always Encrypted at Rest: Yes
• Always Encrypted in Transit: Yes
• Ease of Use: Medium
• Pricing: No additional cost if using default KMS key

There is another nice feature about Parameter Store in that it is possible to call the get_parameters (note the plural) API to get multiple configuration values/secrets in one API call based on the hierarchical Parameter Store path. This can be useful in optimizing for cold boots.

Secrets Manager

AWS Secrets Manager is yet another way to store secrets in the AWS ecosystem. Secrets Manager has one primary benefit over Parameter Store: Auto rotation of RDS credentials. If using RDS, Secrets Manager is a great choice. The nice thing about Secrets Manager is that it can be retrieved using the same SSM get_parameter API despite being a separate service:

client = boto3.client('ssm')
client.get_parameter(
        Name="/aws/reference/secretsmanager/DBPass",
        WithDecryption=True
    )

There are some downsides to Secrets Manager:

• It costs money per secret stored and there is cost per retrieval of the secret. This can add up in a serverless context when the secrets need to be retrieved on each lambda invocation.
• There is no audit log to show previous values of secrets and when they were changed in the AWS Console.
• It’s possible to only retrieve a single secret at a time. In a cold start optimization context this can be impactful if there are multiple secrets. (Update: It is possible to store multiple key/value pairs within a single Secret Manager secret to get all secrets via a single call)

Secrets Manager is a relatively new service, so there may be new functionality to leverage as time goes on.

Summary of Retrieving Parameter Store Secrets at Runtime

• Always Encrypted at Rest: Yes
• Always Encrypted in Transit: Yes
• Ease of Use: Medium
• Pricing: Medium ($.40/secret, $.05/10000 API calls)

Summary

Solution	Encrypted at Rest	Encrypted in Transit	Ease of Use	Pricing
Deploy Time: Env Var w/ Plaintext	No, not via CloudFormation	No, not within AWS	High	Free w/ default KMS CMK
Deploy Time: Env Var w/ manual KMS Ciphertext	Yes	Yes	Medium	$1/mo for KMS CMK
Deploy Time: Env Var w/ Plaintext from Parameter Store	No, not via CloudFormation	No, not within AWS	High	Free w/ default KMS CMK
Deploy Time: Env Var w/ Ciphertext from Parameter Store	Yes	Yes	Low	$1/mo for KMS CMK
Run Time: Parameter Store	Yes	Yes	Medium	Free w/ default KMS CMK
Run Time: Secret Manager	Yes	Yes	Medium	$.40/secret, $.05/10000 API calls

Although it may seem like using a secret management service at run time may seem like the best option at first, there are a number of other complications to consider with run time solutions:

• How to catch a malformed secret from breaking the application when it is updated in the service?
• How to handle caching of the secret as to not invoke the secret management service API each time it is needed?
• How to invalidate the cache when the secret is no longer valid?
• How to backoff with appropriate jitter when there are interruptions in the availability of the secret management service?

These concerns are less of a problem for deploy time solutions as they can be validated through a CI/CD pipeline before going live.

The ecosystem of secret management solutions for Lambda suffers from too many options that don’t all provide a very strong security story. It takes some careful consideration of the security goals specific to your application. For deploy time secret retrieval, Lambda Environment Variables with ciphertext via KMS is a solid option. For run time secret retrieval, SSM Parameter Store and Secrets Manager will both do the job with the former perhaps being a bit better suited for Lambda at this time.

What do you use for Secrets Management? What other considerations are there?

All of the code samples in this post can be found at piohhmy/serverless-secret-examples

---

---

1. https://www.eetimes.com/document.asp?doc_id=1279619 ↩

2. AWS Secrets Manager actually does this slightly different, it uses Envelope Encryption to get a Data Key from KMS and then uses that Data Key for the secret encryption. ↩