From Devops to DevSecOps!

"... If you do so, you will be hacked, and
you will bring shame to your entire company or department. We all want to be on TV someday, but there are better ways to go about that."

Kubernetes: Up and Running. Kelsey Hightower, Brendan Burns, and Joe Beda

Unless using such super secure frameworks, your production running application could somehow be hacked. Don't be shocked! Even the most robust applications/platforms contain numerous security holes and bugs. This is an absolute fact in computer science and a truth that cannot be denied. It also highlights that the journey to security requires everyone involvements, and starts from the moment you write the very first line of code and continues even after deployment to production, with the aim to make attacks difficult, ideally impossible.

DevSecOps, embraces DevOps practices, and advocates the necessity to integrate security into our CI/CD pipelines and extend the existing devops tools and practices to include security checks.

Integrating security into our pipelines

Security means hundred of things! and while trying to integrate it in our pipeline we can easily get overwhelmed and lost! Understanding the workflows and tools the team is using is an essential part toward adding effective security checks and controls to our pipeline.

As I've briefly mentioned in a previous article, we can decouple our continuous integration, continuous delivery and continuous deployment pipelines into 6 main stages:

Dev: Development phase, before the code lands into the source code repository.
Build: Building and performing basic automated testing of the system.
Test: After a successful build, the artifacts are deployed into staging and test environments.
Host: This stage involves any configuration and/or updates need to be applied to the infrastructure.
Run: If all green, the application is ready to be deployed into production environment.
Observe: Continuously measuring and monitoring production activity.

In this post, we’re going to discuss how we can integrate security practices in each of the six activities listed above. Let’s dig in.

Stage 1: Dev

Devops practices emphasis writing good code. Code that works well, easy to change and to understand. DevSecOps extends these practices by adding security checks for writing good & secure code.

Practices like static code analysis, code reviews, pre-commit hooks can be extended to include security as well, without impacting developers productivity. It enables to find and fix common security issues before the code is committed into the source code repository.

Code reviews: (Peer) code review can improve security in important ways. It increases developers liability and transparency, which reduce risks of internal threats (someone trying to include a backdoor or logic bomb in code) and helps to improve code quality. This practice can (and should) also be extended to infrastructure code (Puppet manifests, Ansible playbooks, Dockerfiles ..).
Static analysis tools in IDE: A variety of tools exist, both open source and commercial, that can be integrated into your favorite IDE and help to check for code consistency, maintainability, clarity, bug patterns... In fact, security checks can also be added with a limited number of rules to identify security vulnerabilities.

A non exhaustive list of tools that can be used at this stage include: Gerrit, Phabricator, SpotBugs, PMD, CheckStyle, Find Security Bugs, ..

Stage 2: Build

Once the code is committed to source repository, the build and basic automated testing of the application are performed to ensure that code is compilable and buildable at all times.
Again, checks can be added at this stage enabling the detection of critical and high security issues. if serious problems are found, the build should fail and send alert notifications.

Software Component Analysis: According to Sonatype, 1 in 8 of used open source components contain known security vulnerabilities. Running SCA tools helps identify out-of-date libraries and frameworks with known security vulnerabilities.
SAST: Another way to improve security at this stage and give quick feedback to the team is by running static analysis software testing. These tools can find hidden bugs that reviewers will sometimes miss, and that might be hard to find through other kinds of testing.
Unit tests: UTs are important in catching regressions as we refactor code or make other changes. Adding unit security tests to functions is something worth investing on.

... developers downloaded more than 300 billion open source components in the past 12 months, and that 1 in 8 of those components contained known security vulnerabilities.

Sonatype’s 2018 State of the Software Supply Chain Report

Tools that can be used at this stage include: The well known Sonarqube, OWASP Dependency Check, sourceclear, Retire.js and snyk, ...

Stage 3: Test

The Test stage is triggered after a successful build by picking the generated artifact(s) and deploy it to staging and testing environments. All tests, including functional, integration, performance, advanced SAST, Security and DAST are executed in this stage.

This phase generally takes more time and resources to execute, and following a fail fast approach is generally preferable here! This means that the more expensive and time-consuming tests are left until as late as possible, so that they are only executed if other tests have already passed.

Targeted dynamic scanning: Once deployed, the application is exposed to an other category of possible attacks, such as cross-site scripting, SQL injection or broken authentication flaws... This is where Dynamic Application Security Testing (DAST) comes into play by analyzing the running application against common critical and high severity issues like those outlined in the OWASP Top 10.
Fuzz testing: Fuzzing is a Black Box, brute-force reliability testing technique, which basically consists in finding implementation bugs using malformed/semi-malformed data injection in an automated fashion.
Automated Attacks: We can go even further and try to simulate attacks on running application, by executing basic set of targeted automated pen tests against the system as part of our automated test cycle.

Here again, many tools (Open source and commercial) are available: BDD Security, JBroFuzz, Boofuzz, OWASP ZAP, Arachi, IBM AppScan, Mittn, GAUNTLT, SecApp suite ...

Phase 4: Host

Securing the runtime environment is as important as securing the application running on top of it. Many best practices, patterns, guides and tools have been developed throughout the years to help hardening the infrastructure and make it more resilient.

Least Privilege: The PoLP states that every module (such as a process, a user, or a program, depending on the subject) must be able to access only the information and resources that are necessary for its legitimate purpose.
Configuration Automation: Configuration management tools make it easy to provision secure infrastructure repeatedly and at scale. By standardizing the configuration, CM tools reduce issues associated with patch management, minimize the risk that hackers can exploit one unpatched server, and helps reduce any differences between the different environments. Worth nothing to mention that using CM tools allows configuration information to be tracked in central repositories and under version control.
Immutable infrastructure: With immutable
infrastructure, once an artifact is created in the system it does not change via user modifications, there are no incremental changes: Rather than a series of incremental updates and changes, a new server instance is created and replace the old one.
Security standards: There are many community efforts providing guidelines and recommendations for hardening your infrastructure. Such standards include Center for Internet Security (CIS) benchmarks and NIST configuration checklists.

Tools here include: Ansible, Puppet, HashiCorp Terraform, Chef, Docker, ...

Phase 5: Run

If all the stages above run successfully, we're now ready to go to production. The goal here is to verify if there isn't any mistakes during configuration or deployment time that could make the system less reliable and resilient, and open it up to attack under failure.

This is where chaos engineering and it principales play an important role for automating runtime checks and tests. The most obvious example we can give here is the Netflix's Simian Army and their monkeys, especially the Security Monkey that finds security violations and vulnerabilities and highlights risks, like changes to access control policies or firewall rules.

Phase 6: Observe

Security doesn’t end after systems are in production. In DevSecOps, automated security checks and monitoring feedback loops are essential parts of production operations.

Continuous monitoring allows to gain insights into the types of traffic an application is receiving and helps identify patterns of malicious users.

Runtime Application Security Protection: RASP technology identifies and blocks application security threats in real time. By adding detection and protection features to the application
runtime environment, RASP enables applications to “self-protect” by reconfiguring automatically, without human intervention, in response to certain conditions.
Security Monitoring: It starts by automating the process of collecting and analysing indicators of potential security threats, define which types of behavior should trigger alerts, and take serious action on alerts.
Red Team: The Red Team dig deep to fully understand the realistic level of risk, weaknesses in the system and vulnerabilities. They are generally given freedom to act short of taking the system down or damaging or exfiltrating sensitive data

As for tools, many runtime defense solutions exist in the market: Prevoty, Alert Logic, HALO, EVIDENT, ...

Final words

In this post I scratched the surface of some practices that we could adopt to start our DevSecOps journey. Security should be an essential part of our pipelines, and our teams must own it as we do with development and operation!