Inspect Docker Images with Anchore for DevOps Security

At Flux7, we are passionate about sharing the power of DevOps. In that vein, we recently gave a workshop introducing developers to the power, ease of use, and governance that comes with moving to a DevOps model reinforced with well-architected tooling. The goal of the workshop was to teach developers more about AWS and Docker-based microservices architecture. And, how using Amazon services like EC2 Container Service, CodePipeline, and CodeBuild can come together to create a platform for developer teams to focus on their application. We highlighted the Anchore solution as part of our microservices architecture for security and will share in today’s blog why we deployed Anchore, how we used it to ensure DevOps security and policy compliance, and our overall experience with the tool.

What is Anchore?

Before we dive into the architecture we created for the workshop, for those of you new to Anchore, a quick introduction may be helpful. Anchore is a service that helps manage, secure, and troubleshoot containers with a security by design approach that ensures the contents of containers match corporate policy. As Anchore states in its marketing materials, the tools are transparent to developers, visible to production, accessible to security, and all designed for the fluid nature of containers. To achieve these goals, Anchore has three main features: 

• Discovery: Anchore allows you to search for Docker container images across multiple public and private registries. And, it allows you to drill down into repos to see all tags and images including image history and updates.
• Analysis: Once images are discovered, you can conduct analysis on them, including searchable lists of all packages, files and software artifacts.
• Policy Compliance: Last, Anchore allows you to create custom policies and whitelists that ensure your Docker container images meet specific requirements. For example, the Anchore policy can ensure that only images that meet your security and compliance requirements are deployed to your pipeline. The tests supported by Anchore include CVE analysis, detecting secrets in images, license compliance, image genealogy and more.

Why Choose Anchore?

Docker’s model of immutable containers makes security scanning very interesting. For example, Docker lets you easily conduct analysis at container build time, rather than at run-time. This allows Docker containers to easily fit into a CI/CD pipeline. While there are several tools that conduct static analysis on containers, we chose Anchore. We liked several Anchore features, particularly how well it integrates with CI/CD pipelines, its comprehensive tests, and open source model with a reasonable price plan for using paid features. We also like that Anchore was developed by the same team that created Ansible, a tool that the Flux7 DevOps team likes a lot.

Amazone ECS Cluster Setup

For the workshop, we created an Amazon ECS cluster and had attendees deploy (from a drop down option) their containers to our cluster using one of the template CI/CD pipelines we created. The pipelines were completely serverless using AWS native tools like CodeBuild and CodePipeline. The final pipeline we deployed in the workshop was also the most interesting because it demonstrated security by design in a DevOps environment. To do this, we used Anchore to provide static analysis of containers and included that in our serverless Amazon native tools pipeline.

In addition to the tools pipeline, we used Anchore as part of our custom built services pipeline. As services moved through the pipeline, images were given a security check with Anchore. If they received an Anchore error, they were automatically failed and an alert email was sent to the appropriate party. To remediate these errors, we:

1. Launched the stack
2. Viewed the failure from the Anchore test
3. Updated nginx base image version to eliminate errors
4. Reran the image as passing

In our serverless setup we used Anchore with AWS CodeBuild. Initially we tried running the Anchore container directly from CodeBuild but we hit an issue in that we needed to run Docker-in-Docker in order to examine the target image from ECR (EC2 Container Registry). Although we know it is possible to do this as Amazon supports building Docker images in CodeBuild, (something we did in other parts of our pipeline,) we were hesitant to try Anchore in the Amazon provided Docker build image because that used Ubuntu 14.04. And, Anchore has not yet been tested with Ubuntu 14.04.

So, we used Anchore’s pre-built container image from the CodeBuild image for building Docker containers. It was able to checkout and run Anchore seamlessly. And so we were able to run the Anchore commands from CodeBuild. (You may access the code we used for Anchore in our workshop here.)

One side benefit from running Anchore using Docker was that attendees were able to follow the same commands used in CodeBuild to test their image locally. It’s worth noting that attendees used a wide variety of systems, e.g. Linux, MacOS, and Windows, which is a testament to the portability of containers.

For the workshop we created an image based on the default nginx image. Unfortunately, the most up to date image included some Debian vulnerabilities that we had to whitelist. Fortunately we already had a CI/CD pipeline for putting our templates into S3. We added the known whitelists file to the same git repo and were able to easily pull the file from S3.

The last issue we ran into was with how Anchore returns its results: 0 for success, 1 for a failure, and 2 for a warning. However, CodeBuild treats any non-zero command as an error. This issue was easily fixed by running Anchore within a wrapper and suppressing the return status of 2 so that CodeBuild registered the build as a success and CodePipeline proceeded to the deployment stage.

One of the downsides of our implementation was that we had to use a database local to the container. This meant a lot of wasted compute time in analyzing the same base images. While Anchore supports an external database, we couldn’t have used one without making it publicly accessible since you cannot as of right now run CodeBuild within a VPC.

Using Anchore in CodeBuild, we were able to create a serverless DevOpsSec CI/CD pipeline that was easy to setup and maintain. Moreover, Anchore enabled us to easily alert and remediate any containers with security concerns. As continuous monitoring for security issues is paramount to ongoing risk management, compliance and adherence to internal security policy, Anchore will become an important tool for organizations looking to build and maintain secure container-based architectures.

Resources

Effectively Balancing DevOpSec How to Achieve Security with Agility in the Cloud