Introduction
The term “shift left” refers to the efforts of a DevOps team to guarantee application security at the earliest stages in the development lifecycle, as part of an organizational pattern known as DevSecOps (collaboration between development, security, and operations).
DevSecOps is the philosophy of developing applications securely from ideation to deployment. Modern applications are typically split into multiple components and services that are packaged and deployed as containers. Those containers include runtime dependencies of a given application or service, like shared libraries, helper binaries, and more. Docker is one of the popular choices for building, publishing and running containers, whereas container orchestration frameworks like Docker Compose, Swarm, Rancher, and Kubernetes are becoming the standards for deploying, scaling, and managing containerized applications.
Shifting Left
The premise of DevSecOps for us is that in the Software Development Life Cycle (SDLC) of KUY.io Konnect™, we made each member of our team responsible for security with the goal to unify our operations and development teams in terms of security.
Over the past weeks we made a concerted effort to make sure that our CI/CD pipeline for building KUY.io Konnect™ into a containerized application stack checks for all known vulnerabilities of the application, the libraries that KUY.io Konnect™ depends on and the OS libraries and binaries at the base of the container environment.
Before even publishing the KUY.io Konnect™ container to our Docker registry from which our customers deploy and update their KUY.io Konnect™ access server instances, we need to make sure that the corresponding applications, services and its dependencies are not vulnerable to attacks by malicious actors. But how do we know what vulnerabilities exist and whether our application or dependencies are affected?
The Common Vulnerabilities and Exposures (CVE) system provides a reference-method for publicly known information-security vulnerabilities and exposures. The United States' National Cybersecurity FFRDC, operated by The Mitre Corporation, maintains the system, with funding from the US National Cyber Security Division of the US Department of Homeland Security. The Security Content Automation Protocol uses CVE, and CVE IDs are listed on Mitre's system as well as in the US National Vulnerability Database.
To leverage the CVE data, we integrated security measures into each step of our development process, and made security and vulnerability scanning part of our DevOps process, effectively turning it into a DevSecOps process.
The Agile DevOps process model provides a blueprint for continuous monitoring of code quality and security.
Development Cycle
We started our left shift in the top left corner of the Agile DevOps process diagram, the development cycle. As part of our day-to-day development work on KUY.io Konnect™, we included static code checking rules for common OWASP security problems that run automatically after each source code change and before changes are checked into our source code version control system.
The security checks that we leverage are part of the excellent RuboCop system that we already have in place to ensure consistency and idiomatic code for our dev team. RuboCop runs every time a developer presses "Save" in their code editor and provides immediate feedback about any code issues directly in the editor - think squiggly lines when you are writing a document in Word. You can learn more about the RuboCop security checks we implemented here.
As part of the left shift in the development cycle, we also added a new rule to our source code repository that executes RuboCop as a pre-commit hook, ensuring that only source code that passes the test suite can be checked into our version control system.
Continuous Integration
Next, we added a left shift to our CI cycle by leveraging the excellent brakeman security scanner. Brakeman is a static analysis security tool that goes well beyond the security checks implemented by RuboCop.
Example of a security issues discovered by the brakeman static security analysis tool.
However, since brakeman is a substantially more complex tool than RuboCop, it's also heavier to run - so much so that we don't want to execute it every time a developer hits "Save" in their editor. Brakeman is designed to catch problems early on during development, so we opted to run the tool as close to the development cycle as possible and made it part of our CI pipeline that checks every every commit in our version control system and provides immediate feedback and quality metrics to developers working on a feature or bugfix branch.
Continuous Testing
Every commit to a feature or bugfix branch triggers an automated run of our comprehensive test suite for KUY.io Konnect™. Our test battery includes a broad rang of test types, ranging from Unit tests, to full end-to-end feature test using Selenium driving a headless Chrome browser to interact and click through our application the same way a user would. Our test suite also produces a screenshot for every tested application state that we use to manual top-hatting of our user interface.
We integrated the automated vulnerability and security scanner Trivy from Aqua Security into our test suite, and made sure it executes and alerts about application and dependency vulnerabilities before our source code branches can be merged into our main code branch that we use to build official releases. Trivy is a state-of-the-art scanner used by companies like GitHub, GitLab and Docker.
Continuous Delivery
Every time a source code branch passes all automated tests and a manual code review by our team members, the branch can be merged into our main branch. The main branch always contains a "release-grade" version of KUY.io Konnect™. Our CI/CD system automatically builds a KUY.io Konnect™ container image every time the main source code branch changes.
To shift our CD process left, we again relied on the Trivy scanner by Aqua Security and integrated the Docker image scanning capabilities into our CI/CD pipeline. After the CI/CD pipeline builds a new KUY.io Konnect™ container image, we execute a vulnerability scan on the resulting container image to ensure that the container OS and runtime dependencies inside the container a free of known vulnerabilities.
Only when the container image passes the vulnerability scan, do we upload the release container to our Docker registry, where it is ultimately delivered to our customers.
Where Do We Go From Here?
One of the main drivers of KUY.io Konnect™ is that we never stop because we consider something is "good enough". So where will our left-shift journey take us next? We believe there are two areas where we can make an immediate impact to our application security:
Threat Modelling: Adding threat modeling is the ultimate shift-left approach, where potential threats, vulnerabilities can be identified, enumerated and mitigations are prioritized early on in the design phase.
Penetration Testing as a Service: Penetration testing is an authorized simulated cyber-attack by experts performed to evaluate the resilience of the system's security control. Penetration Testing as a Service is a continuous approach to manual penetration testing, where the system is continuously challenged by a group of security experts to provide continuous assurance as the development team continuously pushes new code to production.
A big shout-out to the great folks at Software Secured who continue to provide us with amazing insights into the left-shift approach!
Summary
DevOps is all about increasing the feedback loop from idea to customer and back again by leveraging automation and tools to drastically remove the amount and error of manual labor. The main driver for us in staying on top of and leveraging state-of-the-art DevOps practices is to ensure high code and application quality and stability. KUY.io Konnect™ is a mission-critical tool in a competitive market, with our customers relying on KUY.io Konnect™ to provide them with secure access to their most important networks, any error or failure that brings the system down has tremendous negative impact on our customers and us.
But KUY.io Konnect™ is no only mission-critical in a sense that it must always work: since our customers use KUY.io Konnect™ to secure their most precious on-premises and cloud networks, we need to make sure that the KUY.io Konnect™ application itself, the dependencies it uses are free of known attack vulnerabilities. But the ball doesn't stop here: it doesn't matter if the KUY.io Konnect™ application and its dependencies are secure if the final container image that includes the OS and runtime dependencies and OS libraries has holes. So we made sure that any time our customers download an updated version of the KUY.io Konnect™ Docker image, that container image is free from known CVEs.
As a result of our efforts we are proud to announce that we are making a full report of all fixed vulnerabilities for KUY.io Konnect™ part of our release notes, and ensure that no KUY.io Konnect™ image gets pushed to our Docker registry with any known vulnerability at the time of publishing.
We shifted left, so your networks can be secure.