Ten Best Practices for Managing Application Secrets in AWS

“A full 80% of data breaches are caused by silly mistakes by those responsible for managing secrets,” according to Rashmi Jha, Microsoft program manager as quoted by InfoSecurity.  Clearly, managing secrets is vital and is oft-cited as a leading IT security challenge. So, although we have access to great tools for managing secrets and configurations when developing applications for the cloud, getting secrets management right is important. As a result, we’ll share in this article best practices for managing application secrets. While we use Amazon ECS and AWS Secrets Manager as our example, these best practices can be applied to other services as well.

1. Do Not Store AWS Access Key and Secret Key Credentials in Code

   This one is such a big no-no that we highlight it first. This is actually not specific to container applications or AWS; it is bad to store any credentials in Git. AWS scans public Git repositories and automatically revokes AWS credentials that are made public. Storing AWS access credentials in a private repository is still a terrible idea. If AWS credentials are stored in Git, it means they are exposed and that they are not frequently rotated. 
   
   Once your containers run in AWS, they can assume an IAM Role to obtain temporary credentials. 
   
    

2. Do Not Print Secrets

   While it may sound obvious, we explain more often than we’d like that people should not print secrets. If secrets are encrypted and securely stored in AWS Secrets Manager, printing them defeats the purpose. Once printed, secrets are aggregated with other logs and sent to AWS CloudWatch or another logging platform. At which point, people without access to the secret can now see it in plaintext — all without an audit trail of who was able to access it. 
   
   In addition, if secrets are stored in environment variables, do not print the environment variable when the application crashes as it would expose the secrets to application users.
   
    

3. Make your App Environment-Agnostic

   This one may be more controversial than it sounds at first. Most development frameworks (like .NET Core and Java Spring) have built-in management for multiple environments. It is very common for developers to use these features.
   
   Container images should be built once, tested, then promoted to higher environments. Re-building a container image for each environment is an anti-pattern. Why go through intensive testing in a lower environment and then take the risk to deploy something new to production? 
   
   When we help customers improve their cloud practices, a quick look at Amazon ECR can tell us if this best practice is followed. If we see repository names like “app1-dev”, “app1-staging”, or “app1-prod”, it means they deploy a new app in each environment and do not have a mature code promotion process.
   
   Some development frameworks have a convention for environment naming — e.g. .NET proposed “Development”, “Staging” and “Production”. However, your cloud infrastructure may host apps developed with other frameworks and these environments may be named with a different convention. You may even want to add a new environment in the future. 
   
   For each environment, your app will need a different set of both configs and secrets, AWS Secrets Manager is not just for secrets! You can use it to store configuration settings as well as secrets.
   
    

4. Make Code Region-Agnostic

   If your application uses any AWS Services from an AWS SDK, you will need to specify the Region you want to target — most often this is the Region where the app runs. Most code samples online will show you something like the following snippet because it is the easy way, not the right way:
   
   AmazonDynamoDBConfig clientConfig = new AmazonDynamoDBConfig();clientConfig.RegionEndpoint = RegionEndpoint.USEast1; AmazonDynamoDBClient client = new AmazonDynamoDBClient(clientConfig); 
   
   If your application is currently deployed in a single Region, there is a good chance you will need to move it to a different Region in the future. It could be to decrease latency to your customers or run it in multiple Regions for improved resiliency. Your app may also need to reach a service in another Region. Yet, depending on the language you use, the AWS SDK does not always make it easy to auto-detect in which Region the app has been deployed. In these cases, you can pass the Region to the container via the Task Definition as an environment variable to make your code Region-agnostic.
   
   

5. Cache Secrets

   If your app frequently needs to access secrets (e.g. on every API call) and retrieves them from Secrets Manager every time, performance will be impacted significantly given the time it takes to request, decrypt and retrieve them. Instead, you can cache them locally, allowing the app instant access.
   
   AWS Secrets Manager has the ability to rotate some secrets (e.g. for connecting to a managed database like RDS). In this case, you will need to retrieve new secrets once the older ones expire. 
   
   

6. Don’t Deploy Secret Keys with AWS CloudFormation

   You can use AWS CloudFormation to deploy Secrets Manager resources for your app but do not use it to create the secret keys inside the secret. If you were to make an update to this template, the pipeline would redeploy it and erase all secret values.
   
   

7. Unit Tests Should Not Depend on Secrets Manager

   Unit tests should be self-contained. You may be able to run unit tests locally which makes a call to Secrets Manager but the unit tests should also be executed as part of the application build and deployment pipeline. The environment building the container should not be allowed to access any secrets. Testing the integration with Secrets Manager should be part of integration testing.
   
    

8. Avoid Storing Secrets in Environment Variables

   Environment variables are accessible to everything running in the OS or container. If your app is a large monolith, everything will have access to them. If your app is a microservice, then the exposure is limited.
   
   It is a common practice for developers to print debug information which may include environment variables – hopefully not in production but we’ve all seen a webpage with this kind of information. It creates an unnecessary risk for credentials to be accidentally exposed.
   
   If you accept passing secrets as environment variables, you will not have to use the AWS SDK to retrieve those secrets which can save some development time. If you are going to do this with ECS, add a reference to Secrets Manager in your ECS Task Definition. Do not put the secrets values directly in the Task Definition environment variables, because they would be visible from the AWS Console.
   
    

9. Identify Secrets by ARN

   The AWS SDK allows you to retrieve a secret by its name or ARN (Amazon Resource Name). There are two reasons we recommend you use ARN. 
   
   First, it is possible for secrets to have identical names (although not in the same account and Region). In this case, you would be relying on a naming convention which is more error-prone than using an ARN.
   
   Second, the Secrets Manager resource for your app should be created before the Task Definition. In this way, we can configure the IAM Policy for the Task Definition to only access the secret it needs and not the others. This is called the principle of least privilege.
   
    
   
   
   
   The ARN itself is not a secret, and it can be passed to the container as an environment variable. Your container can assume the Task role to obtain temporary AWS access credentials to retrieve the secret by specifying the secret ARN.
   
    

10. Run Your App Locally Without Secrets Manager

    If you need to run containers locally as part of the development process, it is practical to limit external dependencies as much as possible. In this way, you have a self-contained environment and limit potential conflicts with other developers.
    
    That being said, you will need to write some code to integrate with Secrets Manager. As a result, you will need another local configuration that allows communication with AWS services.
    
    For additional reading on Secrets Management:

• AWS ECS and Vault Paired for HIPAA-Compliant Secret Management
• Payment System Provider Automates Security with Dynamic Secret Keeper
• A Technical Look at Managing Secrets using AWS’ Parameter Store
• Handling Secrets in Microservices
• AWS Youtube channel – Securing your .Net Core Container Secrets – Part 1
• AWS Youtube channel – Securing your .Net Core Container Secrets – Part 2

Or, for a deeper dive into avoiding common cloud security mistakes, subscribe to our blog for ongoing tips, tricks, and cloud analysis.