Financial Publisher Adopts HashiCorp Vault with Flux7 SmartStart

Print is the new digital and no one proves that more than one of our most recent customers who is a brand-name publishing house. Embracing a digital-first business model, this publisher’s IT, development and security teams sought to significantly reduce their on-premise footprint, moving to Docker and hybrid cloud architecture, using AWS and Google. As part of its move, the need for a secret management solution was identified and we were happy to get the call to help address the company’s need with HashiCorp Vault.

The firm was using a solution for its usernames and passwords which was not programmable, and it was without secure storage or secret store for its SSH keys, and KV pairs. As a result, it sought to switch to HashiCorp Vault, a full-featured secret management solution of choice when building highly secure and highly available systems. 

The first step in our engagement was an assessment in which we examined the company’s technical, business and process needs. In doing so, we identified three clear steps for the project: 

• Create a workflow using HashiCorp Vault and a custom security broker
• Deliver AWS CloudFormation templates to deploy HashiCorp Vault and HashiCorp Consul clusters
• Use AWS CloudFormation templates for AWS serverless application model (SAM) to deploy security broker

To start, our DevOps consulting team developed the AWS CloudFormation templates that would allow the publisher to deploy HashiCorp Consul — a highly available and distributed service discovery and KV store — and Vault in the publisher’s AWS environment. Flux7 used the Consul AWS auto-discovery feature to bootstrap the Consul cluster. We created a Vault Auto-Scaling Group (ASG) and added an ELB (Elastic Load Balancer) in front of Consul ASG and Vault ASG to ensure high system availability. 

We secured the HashiCorp Consul – Vault communication with TLS (transport layer security) encryption and used Consul’s snapshot endpoints for Backups and Disaster Recovery (DR). For accountability and an audit trail, we configured AWS CloudWatch to capture logs that were in turn forwarded to Splunk as the company’s central log service.

Security Broker Approach

While Vault authentication backend is not enough, we used role ID and secret ID to authenticate to Vault. However, getting this role ID and secret ID to the instances are a problem. This problem is referred to as the ‘Secret 0’ problem and it means that a third identity that can help establish trust between the client (containers and instances) and secret IDs is needed. 

To resolve this issue, AWS Lambda was used as the security broker to serve role and secret IDs; the security broker client is set to authenticate against the API Gateway and fetch credentials. It should be noted that while this problem can be solved using the AWS IAM authentication method, we could not deploy this approach as this customer has a hybrid cloud architecture, using Google and AWS. 

In the customer’s AWS Docker-based microservices architecture, the container calls a function that asks for credentials to Vault. The request goes through an automated validation process, where tags on the infrastructure are looked at, access is given and when approved, the container is given one-time credentials which it then hands back. 

This best practices layered security approach was supplemented with a skills training and knowledge transfer that assured the publisher’s technology teams would be able to apply and extend the tools and processes moving forward. Now the publishing house has a highly available federated installation of Vault and Consul that the team can actively manage themselves. 

Taking security by design approach allowed this firm to proactively build in a cloud security architecture as it moved its systems to the cloud. In the process, this firm has decreased risk from manual management while introducing automation that streamlines processes. Moreover, a highly secure and high availability design for Consul and Vault means that the system has zero downtime for applications and users alike.