DevSecOps: How DevOps and Automation Bolster Security, Compliance

Posted on

DevOps bridges the gap between Development and Operations to accelerate software delivery and increase business agility and time-to-market. With its roots in the Agile movement, DevOps fosters collaboration between teams and streamlines processes, with the goal of breaking silos in order to “go fast”.

Information Security (InfoSec) and compliance are critical to businesses across the globe, especially given past examples of data breaches and looming cybersecurity threats. InfoSec has long been thought of as the group that “slows things down” – the wet towel to your DevOps efforts – often requiring a more conservative approach as a means of mitigating risk. Traditionally, DevOps was viewed as a risk to InfoSec, with the increased velocity of software releases seen as a threat to governance and security/regulatory controls (these, by the way, often require the separation of duties, rather than the breaking of silos.)

Despite some initial pushbacks, enterprises that have taken the “DevOps plunge” have shown – consistently – that DevOps practices actually mitigate potential security problems, discover issues faster and address threats more quickly. This has led  to InfoSec increasingly embracing automation and DevOps practices more and more, as the “security blanket” that enables — and enforces — security, compliance and auditability requirements. This makes DevOps a resource for InfoSec, rather than a threat.

As a philosophy, DevOps focuses on creating a culture and an environment where Dev, QA, Ops, the Business and other stakeholders in the organizations work in collaboration towards a shared goal. We now see DevOps evolving to DevSecOps – with InfoSec aligning with your DevOps initiative, and security requirements made a key tenant of your DevOps practices — and your DevOps benefits.

The Security Opportunity of DevOps

DevOps provides a huge opportunity for better security. Many of the practices that come with DevOps — such as automation, emphasis on testing, fast feedback loops, improved visibility, collaboration, consistent release practices, and more — are fertile ground for integrating security and auditability as a built-in component of your DevOps processes.

DevOps automation spans the entire pipeline- from code development, testing, to infrastructure configuration and deployment. When done right, DevOps enables you to:

  1. Secure from the start: Security can be integrated from the early stages of your DevOps processes, and not as an ‘afterthought’ at the very end of the software delivery pipeline. It becomes a quality requirement – similar to other tests ran as part of your software delivery process. In a similar way to how CI enables “shifting left”, accelerating testing and feedback loops to discover bugs earlier in the process and improve software quality, DevOps processes can incorporate automated security testing and compliance
  2. Secure, automatically: As more and more of your tests and processes are automated – you have less risk of introducing security flaws due to human error, your tests are more efficient and you can cover more ground, and your process is more consistent and predictable- so if something does break, it’s easier to pin-point and fix.
  3. Secure throughout: By using tools that are shared across the different functions, or an end-to-end DevOps Automation platform that spans Development, Testing, Opsand Security – organizations gain visibility and control over the entire SDLC, making the automated pipeline a “closed loop” process for testing, reporting and resolving security concerns.
  4. Get everyone on the same page/pipeline: By integrating security tools and tests as part of the pipeline used by Dev and Ops to deploy their updates, InfoSec becomes a key component of the delivery pipeline, and an enabler of the entire process (rather than pointing fingers at the very end!)
  5. Fix things quickly: Unfortunately, the occasional security breach or vulnerability might come up – requiring you to act quickly to resolve the issue (think Heartbleed, for example.) DevOps accelerates your lead time – so that you can develop, test and deploy your patch/update more quickly. In addition, the meticulous tracking provided by some DevOps platforms into the state of all your applications, environments and pipeline stages greatly simplifies and accelerates your response when you need to release your update. When you can easily know exactly which version of the application, and all its components/stack, is deployed on which environment, you can quickly pin-point the component of the application that requires the update, identify the instances that require attention and quickly roll out your updates in a faster, consistent, and repeatable deployment process by triggering the appropriate workflow.
  6. Enable developers, while ensuring governance: DevOps emphasizes the streamlining of processes across the pipeline to have consistent development, testing and release practices. Your DevOps tools and automation can be configured to enable developers to be self-sufficient and “get things done”, while automatically ensuring access controls and compliance. For example, as a resolution to the growing “shadow IT” phenomena, we see a lot of organizations establishing an internal DevOps service for a dev/test cloud – with shared repositories, workflows, deployment processes etc. This allows engineers on-demand access to infrastructure (including Production), while automatically enforcing access control, security measures, approval gates and configuration parameters – to avoid configuration drift or inconsistent processes. In addition, it ensures all instances across all environment – no matter whether in Development, QA or production – are identified, tracked, operating within pre-set guidelines, and can be monitored and managed by IT.
  7. Secure both the code, and the environments: By creating manageable systems that are consistent, traceable and repeatable you ensure that your environment is reproducible, traceable and that you know who accessed it and when.
  8. Enable 1-click compliance reporting: Automated processes come with the extra benefits of being consistent, repeatable, with predictable outcomes for similar actions/tests, and they can be automatically logged and documented. Since DevOps spans your entire pipeline, it can provides traceability into traceability from code change to release. If you have a DevOps system system you can rely on, auditing becomes much easier. As you’re automating things – from your build, test cycles, integration cycles, deployment and release processes – your DevOps automation platform has access to a ton of information that is automatically logged in great detail. That, in effect, becomes your audit-trail, your security log, and your compliance report – all produced automatically, with no manual intervention or you having to spend hours backtracking your processes or actions in order to produce the report.

DevSecOps is enables organizations to achieving speed without risking stability and governance. Security and compliance controls should be baked-in as an integral part of your DevOps processes that manage the code being developed all the way through to Production. By implementing DevOps processes that incorporate security practices from the start you create an effective and viable security layer for your applications and environments that will serve as a solid foundation to ensure security and compliance in the long run, in a more streamlined, efficient and proactive way.

Can We Stop Talking About DevOps?

Posted on

By Tom Hall

I first heard the term DevOps from a friend in the software business when I was working at Dell HQ in Round Rock, TX. He recommended that I read The Phoenix Project, by Gene Kim, Kevin Behr and George Spafford. I did read it and I found it a fascinating story, but as a long-time network administrator (to oversimplify as much as possible) and having never been involved in developing software, I wasn’t quite sure how it applied to me. It was enough to pique my interest though and drove me to volunteer at a DevOpsDays event in Austin. Within weeks I left Dell and joined a small software company, where I could have a better chance of getting my arms around this slippery pig called DevOps and more hands-on practice with the related tools and techniques.

So imagine my surprise when a year later, the same old friend who introduced me to DevOps suggested that the end-goal of a DevOps transformation should be that we stop talking about DevOps. Uh-oh, I thought, he’s lost the plot. But he made his case by pointing out that adding testers to dev teams–a typical step in any Agile transformation–doesn’t result in new devtest or agiledev teams, they remain simply “dev teams”. Once a business/group has fully internalized the Agile or DevOps values and behaviors, he argued, ‘working in an Agile or DevOps way’ becomes simply ‘working’. This is why many DevOps promoters, myself included, have a negative reaction to seeing ‘DevOps’ in job titles, job descriptions, or department names. While it simplifies the job of identifying which job candidates have a DevOps mindset or skillset, it effectively excuses everyone else in the organization from taking responsibility for change. “Doing DevOps” becomes the job of a particular individual, team or department instead of a philosophy internalized by the organization.

Of course the horse is already out of the barn on that one. Each year we see an increase in the number of businesses advertising for DevOps Engineers, forming DevOps departments or listing DevOps among the required skills for a position. It doesn’t matter whether we think this is a good thing or bad thing, evolution will take its course and there’s little we can do to stop it. What we can do is observe, acknowledge and adapt. Sure, the dangers are real. A DevOps department can easily become just a third silo or the DevOps Engineer might become the scapegoat for everything that goes wrong between development and operations. But if a DevOps Engineer, VP of DevOps or DevOps department can form, contribute to and/or nurture an environment of increased empathy, faster feedback and continual growth, that’s awesome; put the focus there. Don’t spin your wheels arguing that they just “don’t get it” or “they’re doing it wrong”.

I’ve spent the past couple years researching, promoting and attempting to practice this thing we call DevOps. While it’s widely believed that DevOps is impossible to define, I am firmly in the camp of those who believe that empathy is, to quote Jeff Sussna, “the essence of DevOps”. Just as the sales team needs to have empathy for its customers, everyone in the value chain needs to have empathy for each other. I’m aware that many people find this term too “touchy-feely”, but the business benefit of more harmonious interaction between one’s teams and between the organization and its customers is clear. The faster and more effective one is at converting solutions into real value for customers, the more successful one will be in the market. Thus being empathetic isn’t just being a good citizen, it’s being a good businessperson.

Taking it a step further, someone on Twitter recently suggested that “DevOps has nothing to do with technology”. The statement seemed ridiculous on its face. After all, the term ‘DevOps’– literally a portmanteau formed from the words ‘developers’ and ‘operations’–was coined by Patrick Dubois, a technologist intent on improving the process of building and deploying software. But as I thought about it I started to understand. As the DevOps movement has evolved, the most fundamental components its promoters have identified, from Gene Kim’s “Three Ways” (Systems Thinking, Amplify Feedback Loops, Culture of Continual Experimentation and Learning), to John Willis and Damon Edwards’ CAMS (Culture, Automation, Measurement, Sharing), to Dave Zweiback’s ICE (Inclusivity, Complex Systems, Empathy), are primarily about how people work. It doesn’t matter if one is building software or running a school, DevOps principles are sound.

I recently tweeted the observation that “anyone who attempts to define DevOps in a paragraph is trying to sell you something”, adding “not necessarily a bad thing, just buyer beware”. This wasn’t a criticism of tool providers; many tools, from build automation to log monitoring, are essential to an effective DevOps transformation. It was a reminder that the problems and solutions DevOps is concerned with are complex and nuanced. There is no prescribed toolset or checklist of actions one can take to be successful; an effective DevOps transformation requires thoughtful, committed action across the whole range of people, practices and products one uses in their work. It might take months or years of practice, but maybe one day we can all stop talking about DevOps.

DevOps Needs Great Managers

Posted on

by David Fredricks

There has been much discussion about DevOps and the benefits it offers to organizations. A lot of what I’ve read discusses defining what DevOps is and how it will improve the bottom line. Much of the information is general in nature, more broad stroke, theoretical type content. According to Wikipedia, “Getting Developers and Operations talking to one another”, “Tearing down the wall of confusion”, “Collaboration between different departments”, “Automating your systems”, etc, etc. This really makes it difficult for one to take specific actions.

From my experience, defining DevOps is personal. The direct challenges that one deals with on a day-to-day basis really shape the concepts of DevOps and the benefits one hopes to accomplish. This makes it difficult to plug and play the process. What works for one organization is not necessarily the right direction for another.  The best case scenario is starting from scratch and building your people, process, and policies around core best practices. (This is why most of the success stories you read about come from greenfield startups). Unfortunately most do not have this luxury.  Reality is companies have layers upon layers of legacy systems, process and people interconnected with technology running in production. Herein lies the challenge.

Achieving a successful DevOps transformation means complete transparency, disclosing all of the cultural bias, loyalties, motivations and scars that internally exist. This is a huge undertaking. Many of the failures in transformation happen because companies are not transparent with themselves.  Having a deep understanding of systems, process, people and customers is vital to being successful. What? Where? When? Why? These questions must be addressed, understood, and accepted before you can move to the “How”.

This is why managers are the key to success. No one else understands the complexities of the business more than the managers. They know where the “skeletons” are buried, who is in bed with who, what systems work, what systems do not. Why some process is done one way, and in other cases done another. Managers are the bridge between what is being told and what is actually getting done. Great managers define the realities for their team, and communicate that back up to leadership. They understand information sharing is vital for success in transformation and growth within the organization. Great managers provide work environments focused on continuous learning for their employees. They create clear and focused workflows for their teams. Shielding them from naysayers or negative process and time sucks. They influence executive direction and gain acceptance. Great managers are able to keep their teams unified and committed. Great engineers stay with and work for Great Managers.

DevOps need More Great Managers!

Key Characteristics of Great Managers:

1. Emotional Intelligence/Empathy: Great managers understand their team. They have personal connections with each member. They know how and when to engage teammates. Great managers accept their role as coaches, mentors, therapist, and sometimes friends. They are able to build  mutual trust and respect from their teams. Great managers lift team members up to achieve more than even they, themselves, believed they were capable of.

2. Teaching and Learning: Great managers create an environment of “Teaching and Learning”. They understand the distribution of information is a strength not a weakness. Dustin Collins wrote a great blog highlighting the risk when information is not shared in “From Zero to Hero, Or There and Back Again”. Great Managers believe in continuous learning, learning with the intent to teach others. This single methodology builds an expectation for everyone on the team to learn with the  intent to teach. Every member of the team is always learning and teaching.

3. Strong Communication/ Listening skills: Great managers create environments of transparency and sharing simply by listening. They understand that all information is important. Being a great communicator also means understanding what questions to ask and why. QBQ by John Miller is a great reference. Uncovering true motivations behind one’s dialog is essential to being a great manager. They deflect white noise and focus the team on the main message. They define clear goals and expectations. This is especially important when organizations are shifting culturally. Change is only scary when information, expectations and direction are not fully and properly communicated and understood. Leadership voids cause fear and fear kills innovation.

4. Protectors/Blockers: Great managers understand their role to block and shield their teams from issues outside of their influence. They take accountability for the actions of their team. If something goes wrong they take the blame. When things go well, the praise goes to the team. Great managers care more for their team’s success and well being than their own. They create a “No Fault Environment”, allowing employees to try new things without the fear of failing.

Great managers are honest with their team, leadership, and especially themselves. They are constantly defining realities for the people around them. Great Managers have the respect of executive leadership. They are able to influence real change at the very top of an organization. They create stable work environments for their teams. PuppetLabs State of DevOps “Engineer do not leave companies, they leave managers.” Great Managers build and retain cohesive teams. Employee churn is one of the most disruptive elements to successful DevOps transformations. DevOps is hard! Having great leadership in place is essential for guidance and reassurance when something doesn’t go as planned, and it WILL happen. Great managers know how to respond in these situations without sounding the alarms. Great Managers keep the ship calm in rough waters to ensure smooth sailing and success for all.

How to build a microservice?

Posted on

By Sonu Meena

Microservice architecture is the advancement over existing architecture which is more commonly found based around monolithic architecture.

In monolithic architecture all the functionalities and features are offered through single piece of software. Generally, this software is built around three tiers: Data, Application (business layer) and View tier.

This has been the de facto standard for starting any software development at least in startups now. It has its benefits that makes it the most sought architecture to begin development with.

  • Faster to build
    It shorten time to market and hence every new tech company start with this architecture.
  • Work best in constraint environment
    In limited number of developers and budget this is proven to work best for both: Marketing and Devs.
    Marketing guys Or Business leaders want feature out very fast and with few hands working on one piece of software they are able to meet this deadlines pretty well.

So, starting with boring monolithic is good.

But…

Beings said, these benefits don’t last forever. With surge in traffic comesscalability issues. Team put more number of resource. They add more CPUs, more RAM, and more disk to cope up with the growth. This is called vertical scaling where existing servers are upgraded to support growing number of users.
Vertical scaling also has its limitation. Beyond a point you cannot scale and then your application response time again started increasing thereby affecting user experience.

Before it go worse and to further scale we look into horizontal scalabilityoption. This involve running multiple instances of the application behind fault tolerant, highly available load-balancer. And to answer spikes in traffic you simply add more number of instances. This is easy? but wait…

Scalability issue isn’t confined to internet traffic only that you can answer every time with increasing numbers alone. Its effect is more profound internally at development side. With more and more feature keep adding you end up knitting a giant ball that’s getting unwieldy to carry around and issues like following pops up:

  1. Codebase gets bigger and bigger 

    Developer keep pushing more features, thereby adding more number of dependencies and more number of lines of code. Now here comes the problem. Bigger code now takes more space and bandwidth to pull or push. It means code will now take more time to deploy. Since it’s lots of dependencies releasing new features may produce downtime. Downtime is bad for any business. It literally mean loss in revenue.Read: Amazon.com Goes Down, Loses $66,240 Per Minute

  2. More time to build and test
    Individual developer working on particular feature now have to download the entire codebase with all the dependencies to make it work on his little workstation. If he has to test even only his changes, he has to run tests against entire software to ensure the change doesn’t break anything.
  3. on-boarding get slower 

    Now here comes another challenge. Your company is growing and hired some more developers. Codebase is one and to teach new developer how it works will now take considerably lot amount of time. This makes new developer boarding process more slower to make him familiar with all the feature is another challenge.

  4. With more features comes more bugs 


    Since many developers are now working together on same codebase, they all are pushing features in the same repository. It creates favorable opportunity for bugs to propagate unnoticed and create havoc in production.
    No matter how hard you try to avoid them they will be missed anyhow. Earlier codebase was small and so it was easy to track and kill them. This time with many releases going per day they are getting more and more harder to trace. Unless you have team of perfectionist bugs are unavoidable.

& the list continue to grow.

Solution: Componentization

 

So, horizontal scalability answer your scalability problem partially. Rest you have to answer by splitting along y-axis as per scalecube model.
Y-Axis split of scalecube model says split your big suit into multiple smaller components i.e componentize it

Componentization should be in a way that each component is individuallyupgradable and replaceable. They should talk to each other on published interfaces built with well known  industry standard technologies like REST and JSON so that it don’t sound esoteric to end users. They can either communicate through synchronous protocol like HTTP or more preferably asynchronous protocol like AMQP.

Componentization brings following benefits to the team:

  • Freedom to choose technology



Team is divided with cross-functional domain. They can choose the right tool for the right job as long as they are in constraint.
These constraints are:

 

  1. tools should not be esoteric to team and should not incur extra cost while doing linear scalability
  2. Technology should be standard so that even end user don’t need to take pain of learning it explicitly.

 

  • C.A.L.M.S


It brings devOps culture into practice inadvertently. There would be more communication among teams now as team are cross-functional. Using devops life-cycle they get the best out of this.

 

Principles Of Chaos Engineering

Posted on

Last Update: 2015 September

Chaos Engineering is the discipline of experimenting on a distributed system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.

Advances in large-scale, distributed software systems are changing the game for software engineering.  As an industry, we are quick to adopt practices that increase flexibility of development and velocity of deployment.  An urgent question follows on the heels of these benefits: How much confidence we can have in the complex systems that we put into production?

Even when all of the individual services in a distributed system are functioning properly, the interactions between those services can cause unpredictable outcomes.  Unpredictable outcomes, compounded by rare but disruptive real-world events that affect production environments, make these distributed systems inherently chaotic.

We need to identify weaknesses before they manifest in system-wide, aberrant behaviors.  Systemic weaknesses could take the form of: improper fallback settings when a service is unavailable; retry storms from improperly tuned timeouts; outages when a downstream dependency receives too much traffic; cascading failures when a single point of failure crashes; etc.  We must address the most significant weaknesses proactively, before they affect our customers in production.  We need a way to manage the chaos inherent in these systems, take advantage of increasing flexibility and velocity, and have confidence in our production deployments despite the complexity that they represent.

An empirical, systems-based approach addresses the chaos in distributed systems at scale and builds confidence in the ability of those systems to withstand realistic conditions.  We learn about the behavior of a distributed system by observing it during a controlled experiment.  We call this Chaos Engineering.

CHAOS IN PRACTICE

To specifically address the uncertainty of distributed systems at scale, Chaos Engineering can be thought of as the facilitation of experiments to uncover systemic weaknesses.  These experiments follow four steps:

  1. Start by defining ‘steady state’ as some measurable output of a system that indicates normal behavior.
  2. Hypothesize that this steady state will continue in both the control group and the experimental group.
  3. Introduce variables that reflect real world events like servers that crash, hard drives that malfunction, network connections that are severed, etc.
  4. Try to disprove the hypothesis by looking for a difference in steady state between the control group and the experimental group.

The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system.  If a weakness is uncovered, we now have a target for improvement before that behavior manifests in the system at large.

ADVANCED PRINCIPLES

The following principles describe an ideal application of Chaos Engineering, applied to the processes of experimentation described above.  The degree to which these principles are pursued strongly correlates to the confidence we can have in a distributed system at scale.

Build a Hypothesis around Steady State Behavior

Focus on the measurable output of a system, rather than internal attributes of the system.  Measurements of that output over a short period of time constitute a proxy for the system’s steady state.  The overall system’s throughput, error rates, latency percentiles, etc. could all be metrics of interest representing steady state behavior.  By focusing on systemic behavior patterns during experiments, Chaos verifies that the system does work, rather than trying to validate how it works.

Vary Real-world Events

Chaos variables reflect real-world events.  Prioritize events either by potential impact or estimated frequency.  Consider events that correspond to hardware failures like servers dying, software failures like malformed responses, and non-failure events like a spike in traffic or a scaling event.  Any event capable of disrupting steady state is a potential variable in a Chaos experiment.

Run Experiments in Production

Systems behave differently depending on environment and traffic patterns.  Since the behavior of utilization can change at any time, sampling real traffic is the only way to reliably capture the request path.  To guarantee both authenticity of the way in which the system is exercised and relevance to the current deployed system, Chaos strongly prefers to experiment directly on production traffic.

Automate Experiments to Run Continuously

Running experiments manually is labor-intensive and ultimately unsustainable.  Automate experiments and run them continuously.  Chaos Engineering builds automation into the system to drive both orchestration and analysis.

Chaos Engineering is a powerful practice that is already changing how software is designed and engineered at some of the largest-scale operations in the world.  Where other practices address velocity and flexibility, Chaos specifically tackles systemic uncertainty in these distributed systems.  The Principles of Chaos provide confidence to innovate quickly at massive scales and give customers the high quality experiences they deserve.