Software is assembled from many component parts to quickly address customer needs. The end-to-end process of delivering value through software starts with ideation and ends with a finished product or service that significantly improves the lives of its customers. Some depict the Software Supply Chain from right to left enumerating the Continuous Delivery of software as it flows through a number of phases from design, through implementation, then build and ultimately deployment. The Continuous Delivery of software is supported by build and deployment automation commonly called a Continuous Integration/Continuous Deployment (CICD) pipeline. This pipeline is used to support fast deployments and a continuous feedback loop that allows for continuous software improvement. The CICD pipeline makes it possible for many people to contribute to the success of a project and make rapid changes daily to address customer needs and demands.
With the introduction of Lean, Agile, Scrum and DevOps, a shift-left paradigm exists which requires the operational aspects of software support to move earlier in the Supply Chain using collaborative methods. Critical tools and processes are also being migrated closer to enabling platforms, such as the Public Cloud, to make it easier to address these operational concerns earlier within the Software Supply Chain. This makes it possible to remove gating processes that cause bottlenecks and reduce speed to innovation. Some would consider security to be the last frontier for this transformation. The rest of this post describes some of the top-level requirements and considerations for shifting security to the left.
Migrating operational aspects of the Software Supply Chain to the left ensures that these concerns are represented as design constraints before they represent too much burden, debt or complexity. For security practitioners, like myself, shifting left is a complete nirvana because it represents the opportunity to see better security in products sooner. Essentially, security becomes a design constraint. The shift-left paradigm is also consistent with messaging that requires security to be built into software instead of being bolted on.
The FTC has provided a Public white paper last summer which discusses adding security from the start with the following excerpt:
“ Companies that consider security from the start assess their options and make reasonable choices based on the nature of their business and the sensitivity of the information involved. Threats to data may transform over time, but the fundamentals of sound security remain constant. “
The FTC white paper enumerates a number of use cases within the more than 50 enforcement actions it has taken to produce better security, identifying 10 common issues among these enforcement actions. Many of these issues can be translated into codified security requirements and artifacts that could be consumed earlier within the Software Supply Chain. With security requirements represented earlier in the software development process, it also makes enforcement part of the Continuous Delivery pipeline with improved testing, monitoring and response to support security drift detection.
Sounds easy? It’s not and there is a lot that the technology industry must do to change. If we consider for a moment that most of the -ilities are design constraints and commonly the cause of technical debt then it is not difficult to understand. Teams must find ways to innovate and deliver value without creating too much technical debt. Let’s examine a bit more about software security:
Typically, security requirements are described via paper policies and standards up-front then checked towards the end of the Software Supply Chain. Security documents are often cumbersome to navigate and contain a great deal of detail that ultimately burdens a software developer interested in playing their part. In the traditional approach, software is developed and then later inspected by the security team to ensure approvals are obtained before a release to production. This is consistent with most gated processes used to ensure stakeholder buy-in via checks and balances. Because of the complexity of policies and standards, the checks are commonly performed via manual inspection. Time constraints make it difficult if not impossible to perform all checks which causes a coverage problem. Further, many of the checks that need to be performed require software to be published into an end-to-end environment before it can effectively be certified compliant with the policies and standards provided upstream. This ultimately creates a chicken and egg scenario that often causes the relationship between Developer and Security to erode.
That said, security can be shifted left in the Software Supply Chain with great perseverance. Using the shift-left approach, software is developed with security as a design constraint and software defined platforms as enablers. This makes it possible to implement advanced testing, threat intel and ultimately end to end investigations (red team exercises). More importantly, it creates longer-lived and more resilient software.
To effectively shift security to the left and create security as code for inclusion from the start, we’ve enumerated the following 5 elements of a successful program:
1) Culture of Collaboration
Understanding security feedback is critical towards reducing software attack surface and for continually tuning workloads to be resilient to attack. An internal security team is not an adversary. In fact, the internal security team has the ability to help reduce the complexity and false positives identified by traditional security tools by focusing on the behavioral anomalies that have the highest likelihood of attack. After all, a 350 page pdf doesn’t necessarily provide the details and help that DevOps teams need. Instead, a culture of collaboration and contributing value is necessary to create safer software. Take for example, the following diagram depicting the Secure Software Supply Chain with the necessary feedback loops and security capabilities:
Figure 1: The Intersection of Feedback and Intel in the Secure Software Supply Chain
Much of the job of making security design decisions can easily be transferred to an inline development role within a scrum team. Without a doubt, this is by far the best solution and allows for the DevOps team to address security in balance and in context. Like other design constraints, this places the onerous task of creating a hypothesis on the value creator. Yet the leading drawback to this approach is a lack of sufficient information to make informed decisions in order to create a successful hypothesis for creating resilient software. Although it makes a difference, it is not enough to think that signature based static and dynamic analysis will provide all the security testing required. Security as Code will speed up the transformation of controls and policies into the platform and application components. Instrumentation can be added to provide signal noise. Yet, Security is not equivalent to Quality because anti-pattern development can include non-software elements as part of the security kill-chain and a motivated adversary. In other words, the DevOps team members, their location(s), their partners and the component parts they trust are part of the attack surface of their application.
Further, having DevOps spend all of their time attempting to do a complicated job that requires significant depth generally fails because the concerns of the expert are commonly dropped in favor of productivity, availability, or latency. Developers are incentivized to create value as an implicit element of their job. Yet, the job of ensuring security is a very scientific job that often requires a completely opposite set of skills from those of value creation. While some of the job of security can be embedded into DevOps, it is very commonly a context bound job with behavioral analysis and forensics as a foundation. This work is non-trivial when done well, even with automation. And like other context work, security is commonly accomplished by professionals with significant experience.
What’s the answer? Culture. Shifting left requires everyone knows how to collaborate and understand enough of the context to ensure the safety of software. Teams must realize that making decisions is hard and that everyone has something to offer in the equation of building great software. DevOps teams build great innovations and Security teams ensure trust. The key ingredient is bringing people along and developing a joint set of measurements that everyone can learn from. Iteration can also improve everything that goes into software by not over-reaching for an ideal state set of decisions to avoid the act of continuous collaboration. Sometimes simply being transparent about an iterative release plan with a set of assumptions can reduce the friction for everyone involved. For example, what if you can’t build the perfect security controls from the start but the controls you do build are commensurate with the risks you’ll take during that phase of a project. True iteration needs to be spread throughout design to really enable everyone to participate in the DevOps culture.
2) Translate Paper into Code
Security is an enabling concern for most software development projects. Commonly, it also represents a set of features that must exist for customers to feel comfortable using an application. We’ve alluded to this earlier, but more explicitly stated: where developers build value for an organization; security builds trust. Building better security requires three major roles to be represented in the Secure Software Supply Chain: 1) Builders, 2) Enablers and 3) Breakers. I’d like to think that anyone can build security as code and the challenge in doing so is simply collaboration, tools, and organization. But security is increasingly complex and attack surface is becoming complicated beyond what tools can detect. Likewise, attackers are using the same agile methods and tools being used by DevOps to increase their speed of exploit and exfiltration. Essentially, this means that good security comes from a blended approach of all three critical roles in order to distribute defense in depth and stay ahead of attackers at speed and scale. Paper just won’t cut it in an continuously evolving environment.
In translating paper into code, we’ve stolen a page from Maslow and created a Hierarchy of Software Security Needs. The intention behind this model is to help builders understand top concerns from a security perspective and how to build controls with a layered approach. For example, if you build encryption into your workload but you fail to zone/contain the workload effectively, it’s highly likely that the encryption will fail because the environment is unable to protect either the data and/or the encryption keys. Similarly, an environment that has zoning but no asset management may fall victim to compromise when an unauthorized insecure asset is introduced in the wrong zone.
Figure 2: The Hierarchy of Software Security Needs
Using the Hierarchy of Software Security Needs, applications and enabling platforms can be united for full stack security. Applications become hardened from inception through runtime. Policies get added into each layer with some consistency developed through the hierarchy of concerns. Software defined platforms are used to provide protective controls and for defending applications from infrastructure attacks, making them more resilient. This however has to be balanced against the needs of the application and organization. There is no one-size-fits-all approach to securing innovation. The approach must be tailored and meet the culture, objectives and policies of the organization so that all team members can coordinate across boundaries. It is this level of coordination which then makes it possible to deliver secure, resilient software innovation at speed and scale.
Also, there is no single platform to help with ensuring security as code because end-to-end components are often created using different languages. A myriad of protective controls exist within the marketplace. The constraints that they introduce have the effect of reducing security issues but they often inflict an operating model and other control requirements. So it is important to ensure that a one-size-fits-all model is not the mindset because many of these protective measures don’t solve for common mistakes. Inspection/evaluation of controls and their drift from a baseline state is very important in ensuring that common mistakes do not make their way to production. Controls within each layer of a stack are not always compatible and commonly don’t get delivered with the same code. Essentially this means that each level within the full stack will require security to be codeified as an ecosystem of controls. Security as code in its minimum state should include a manifest, inheritance and resolution back to the same component baseline to ensure continuous feedback has a closed loop. In other words, put everyone to work and tie in processes so that everyone knows how to get feedback to the right people quickly.
3) Fanatical Testing and Instrumentation
The art of developing zero day exploits is alive and well. It is not something that will disappear in the future but will likely evolve to take advantage of Continuous Delivery, the Public Cloud and the DevOps era. Attackers are continuing to discover and exploit even the smallest mistakes and re-introduction of once solved security defects. They gain access and escalate their privileges through the trust relationships granted between software components and interfaces. In fact, small mistakes and unreliable trust relationships are consistently linked to catastrophic consequences. So it is of greater importance for organizations to invest in fanatical testing and increased instrumentation to detect and resolve issues before they can be fully exploited. Likewise, it is important that DevOps teams understand how to engage security resources to their full benefit as well as understanding their responsibility within the control ecosystem. Here’s a quick depiction of a Continuous Delivery lifecycle that supports DevOps + Security capabilities and potential tools (it’s an example and there are thousands of tools that could have been included):
Figure 3: Example of Continuous Delivery + Security
For many, it may be useful to focus on transforming gating processes to include policies and testing within their Continuous Delivery pipeline. Fanatical testing starts with understanding what needs to be tested and how. Test driven development should include security as a design constraint that gets progressively tested as part of the software that gets development. This means that application security tests like the OWASP Top 10 should be included in the tests that get run against software at the inception of the project. It makes no sense for things like Cross-site scripting and SQL Injection to make it all the way to production unless a test failed to enumerate an obscure attack case. This essentially means that the toolchain and skills for ensuring that the OWASP Top 10 essentially need to be moved early in the process of developing software. And it also means that security professionals should be evaluating the tests being included to enumerate some of the obscure edge cases that attackers dedicate their time to in order to create the ecosystem that closes the testing cycle for security.
Likewise in a full stack software defined environment, it is critical to add the equivalent of the OWASP Top 10 to a set of testing scenarios. Looking at our Hierarchy of Security Needs, the top issues are consistently zoning and blast radius issues, poor component management, feeble attempts at instrumentation/logging, authentication/trust failures, and misapplication of encryption/secrets management. Configuration management suites offered by Chef and Puppet have a variety of testing capabilities that make it possible to reduce these issues inline with Continuous Delivery. A greater emphasis on fanatical configuration testing makes it possible to reduce some of the most notorious attack surface.
Software defined environments also require specialty testing knowledge and an greater emphasis on component/service whitelisting. Using component parts without understanding attack surface is a good way to encounter a trust-based attack. Using high-fidelity components with the security required by your ecosystem is an essential ingredient in reducing attack surface. Cloud providers are unlikely to support this approach because increased transparency and disclosure of pattern based weaknesses is still largely new territory. Common vulnerability mechanisms like CVSS, CVE and CWE have not yet addressed the problem of pattern based attack models that will continue to surface from misapplication of trust models. As an example, you may have data that requires a high degree of trust among providers and mistakenly introduce a component or service with less trust than is required for your architecture to be “secure” enough.
Instrumentation is also essential and needs to be designed into a workload early to be additive and to avoid creating noise that detracts from great detection and response. With DevSecOps, speed is an critical ingredient largely provided by thorough instrumentation, correlation and attribution back to a component part. The faster a defect is discovered and resolved, the greater the chances for reducing risk to an overall stack. For context, instrumentation is part of the feedback loop that supports security “recalls” and with speed can eliminate attacks that require time for learning and exploration of the environment.
So what does fanatical instrumentation look like? Consider for a moment that you have developed an attack map and one of the controls you require is an alert to know when one of the security assumptions has been challenged. In this case, you might add logging or an event that triggers as part of the application to alert within a correlation sequence. As an example, an admin application is designed to be used between the hours of 8-5 and a user logs in at 9pm which triggers an alarm. This provides a baseline trigger that causes Security Operations to watch and evaluate the actions and behaviors of the user. And in many cases an application might be designed to be self-healing when a trigger like that happens where the policy answer dictates the use cases is not acceptable.
4) Provide Intuitive Security Measurement
Gene Kim does a great job of defining what successful value creation requires at speed and scale. One of the critical elements is a trust based environment whereby DevOps can achieve the mission without blame. The value of blameless environments is fast learning and adaption, or rather inspired innovation. To help us with achieving a blameless environment, let’s look at security which is notorious for being the blame & shame function. Blame comes from attempting to hold someone responsible for their decisions and actions. The opposite of this is praiseworthy behavior. And yet, security is difficult to transform because the information needed to make software secure isn’t always available until after it is designed or developed.
The reality behind the blameless mantra translates into a formula which requires decisions to be made with all the right constraints available up front. A compendia of security is largely developed as a number of common bodies of knowledge, OWASP, SANS, PCI, NIST, and many others. Security professionals contribute to this information and have organized it for their own consumption. It’s not entirely friendly. In fact, if you don’t know the terms then its a steep learning curve. Attackers use software in creative ways that were never part of the original intention of the software. In other words, attackers build anti-patterns and security is a deeply complex science of discovery. For example, cross-site scripting has been around for some time and when mixed with other parts of a kill chain can reduce trust in the original software. Did the original designer consider this? No. And yet, the term cross-site scripting isn’t entirely intuitive nor is it something a software designer would immediately consider as part of their ideation and development process. instead the designer is mostly interested in whether something will actually work.
A blameless transformation for security comes with arming DevOps with as much information as possible as soon as possible. But more importantly, a DevOps team needs a means of learning about security so that they can take on the responsibility of making better security decisions for their software. It was once believed that security issues could be turned into training modules and that software will ultimately get better after developers take secure code training. It is part of the equation but taken outside of the actual software that a DevOps team is working on, the information is hard to apply for the obscure and nuanced issues. If we consider that its easy to understand issues that take software offline and harder to understand anti-patterns that create abuse cases, then we can better anticipate what DevOps teams need for their training and feedback loops. Measurement can help to see the forest through the trees and it is part of the cycle of creating a blameless environment.
Innovative software comes from furthering the needs of a customer by making choices about their requirements and balancing the continuation and maturity of the product. A hypothesis deserves good instrumentation and measurement, not blame. This means that measurement is a critical element towards taking a scientific approach to developing software, and definitely when security is used as a design constraint up front.
Figure 4: Example for Measuring Security defects
So this information is great but it is also a huge tax on DevOps teams. Having useful and actionable intelligence allows DevOps teams to quickly discern the biggest problems that require immediate correction. And one-size-fits-all doesn’t work with this approach. Software that solves for power grid automation is likely to need a different investment in security than a mobile game that will be relevant for 18 months. Likewise, transforming this type of information into the exact security defect that needs to be corrected can be time consuming science. Instrumentation events often require correlation and other analysis to determine what needs to be changed and how. Finding a way to provide the right balance of information for quick and fast resolution has its advantages. Some choose priority tickets and other choose security grading. Choosing measurement that makes your organization faster is essential. Getting an issue resolved in minutes can be the difference between minor nuance and big breaches depending on your environment. What your organization chooses is entirely part of the culture and tooling being developed to make security part of the ecosystem.
5) Continuous Science
We’ve briefly touched on measurement for reducing blame but there is more to the picture. People who develop amazing technology also need to sleep. Attackers have always used the art of being stealth to advance their efforts but when that fails to work, they turn to techniques to create and exploit apathy. Imagine for a moment that you have finely tuned your security controls within your software defined environment and your application to let you know if your application is being attacked. An attacker can take advantage of this principle in a variety of ways that make it very difficult for small teams to successfully defend a large application. An evasion attack can be costly and uses the art of creating noise to reduce the fidelity of signaling.
For traditional applications these techniques have been studied and purpose-built appliances have been developed to be included at the network layer. Improving data center and network controls is done transparently and outside of the software development path because security professionals have been inundated with these types of attack strategies for years. Pulling a workload out of an environment that has all of these purpose built capabilities and placing it into a software defined environment also means that new techniques and tooling need to be developed. And yes, this also means moving security closer to the software being developed. The natural evolution of security is from network to host to software. Unfortunately the timing is not well-positioned for the current evolution of software which means that many of the same traditional attacks will once again rise before they are thwarted in software defined environments.
Unconventional attacks are on the rise and take advantage of SaaS and Cloud services because it enables attackers to get inside of the trust relationships being established between companies and third parties where security technology is immature. For example, HammerToss uses Twitter, Github and Cloud storage to enact its kill chain and gather intelligence using a set of commonly trusted tools. Data center security models required payloads to violate and provide access from behind a firewall. This essentially meant that security protective and detective controls could be tuned to look for a narrow set of methods. In today’s environment, cloud provides a means for attackers to utilize trust models and passwords to gain access because boundaries have shifted and environments are largely becoming boundary-less. (Google has a great paper on this topic: BeyondCorp)
Figure 5: Obscure attacks require Continuous Security Science
Being more deliberate about trust boundaries and attack models can significantly aid in reducing these new types of unconventional attack but its unlikely to be a bullet-proof scenario. Understanding and studying how software gets used with the hunting necessary to see the forest through the trees is a complicated science. It requires a different lens and time to decipher the anti-patterns being developed or worse utilized against software. This translates into a Security Program that revolves around anti-pattern events and commonly helps to determine where in the Secure Software Supply Chain that they need to be corrected.
Closing Remarks
In conclusion, these five elements of a Shift-Left Security Program can help you to successfully navigate the security transformation required to support DevOps and Software Defined Environments. And hopefully it provides a bit more information about what some of the true challenges are in an effort to take advantage of secure innovation at speed and scale. The Secure Software Supply Chain requires some very interesting problems to be solved with security becoming largely a design constraint and not simply part of release testing.