As we discussed in our first post in this series, with DevOps methodologies the goal is to speed up deployment using automation while increasing the predictability and manageability of the development cycle. Too often though, security is left for last, ultimately slowing down deployment, backing up the development process and fundamentally defeating DevOps objectives.
That’s why it’s best to bake security into every stage of the application development process. In our first post we covered DevSecOps when the build artifact is an application binary. In this post we’re taking a closer look at DevSecOps for containers, from the development stage, to pre-production, to production.
The dev stage
Right off the bat, it’s imperative your sandbox environment leverages the same automation framework as your pre-production and production environments. Otherwise there’s absolutely no way you can feel confident that any steps you take to ensure strong security will actually achieve their purpose once a build makes it all the way to a live deployment. Even slight configuration drift can lead to challenges further in your pipeline that can double your work, and greatly increase the annoyance levels of everyone involved in the launch.
All builds should be tested to pass your configuration hardening standards. Not just builds from interns. Not just builds from devs who’ve worked there less than a year. All builds. Continuing with this theme, all images should be audited for software vulnerabilities during the build process, with the output of these audits integrated into your build systems so both developers and devops have immediate visibility into security audit issues.
It’s crucial that your developers care about security at least as much as they care about whether other developers are using tabs or spaces to indent their code. You could simply stress that they care during all-hands meetings, (and no doubt most of them would), but you can make this all much easier by using your audit data to design a policy around the kinds of vulnerabilities that will result in build failures. This will make security part of every developer’s workflow, instead of an unexpected annoyance that is occasionally thrown at them by operations. Lock this all down by enabling trust control and signing your Docker images to cut down on the possibilities of replays, forged images, key compromises, and other man-in-the-middle attacks.
It’s not enough to stop here, though, since some images in your registries may land through an out-of-band process and entirely new vulnerabilities may be discovered that you weren’t on the lookout for previously. Ensure the entire registry infrastructure is compliant with your security guidelines and implement a continuous security audit program for all of your registries.
Registry access should be segmented by application teams and no access credentials for the CI/CD infrastructure should ever be checked-in. As annoying an extra step as they may consider it to be, developers always need to enter their credentials before gaining access. It might be highly unlikely that anyone who shouldn’t get access will be able to do so through a developer’s computer or connection, but every bad thing is pretty unlikely until that thing happens.
Testing the security of your application should be as high priority as testing its functionality, which is why you should complete security audits across the entire application structure while QA is hammering away on the application itself:
- Check for configuration hardening standards on your own docker and test infrastructure.
- Monitor for new vulnerabilities right up until the moment of launch.
- Check application logs for compliance with your own internal regulations, as well as any external ones that are applicable to your company.
As you get ready to push an application to production, ensure the containers are baselined and that none of the configuration drift mentioned earlier has reared its head. Have you profiled the running footprint? Checked access control? Gone through all of your processes? Check them. Then check them again. Then you can feel comfortable pushing to:
While getting the application live in production is a great moment of celebration for every developer, designer, and product manager who worked on that particular project, getting the application live in production is a great moment for ops to double-down on security:
- Always be on the lookout for any new vulnerabilities. Code that didn’t seem vulnerable six months ago can suddenly look vulnerable without much warning.
- Watch for unexpected activity on the container like changes in access control, processes, network configurations, and the like.
- If unsigned images aren’t allowed into the container registry during the development process, they certainly shouldn’t be allowed into production
- Keep an eye on your production hosts. There’s no rest for the weary.
Being vigilant and sticking to processes are the best ways to ensure you aren’t blindsided by security issues down the road.
In the next post in this series we’ll take a closer look at best practices for Virtual Machines / Instances.