DevOps Needs Great Managers

Posted on

by David Fredricks

There has been much discussion about DevOps and the benefits it offers to organizations. A lot of what I’ve read discusses defining what DevOps is and how it will improve the bottom line. Much of the information is general in nature, more broad stroke, theoretical type content. According to Wikipedia, “Getting Developers and Operations talking to one another”, “Tearing down the wall of confusion”, “Collaboration between different departments”, “Automating your systems”, etc, etc. This really makes it difficult for one to take specific actions.

From my experience, defining DevOps is personal. The direct challenges that one deals with on a day-to-day basis really shape the concepts of DevOps and the benefits one hopes to accomplish. This makes it difficult to plug and play the process. What works for one organization is not necessarily the right direction for another.  The best case scenario is starting from scratch and building your people, process, and policies around core best practices. (This is why most of the success stories you read about come from greenfield startups). Unfortunately most do not have this luxury.  Reality is companies have layers upon layers of legacy systems, process and people interconnected with technology running in production. Herein lies the challenge.

Achieving a successful DevOps transformation means complete transparency, disclosing all of the cultural bias, loyalties, motivations and scars that internally exist. This is a huge undertaking. Many of the failures in transformation happen because companies are not transparent with themselves.  Having a deep understanding of systems, process, people and customers is vital to being successful. What? Where? When? Why? These questions must be addressed, understood, and accepted before you can move to the “How”.

This is why managers are the key to success. No one else understands the complexities of the business more than the managers. They know where the “skeletons” are buried, who is in bed with who, what systems work, what systems do not. Why some process is done one way, and in other cases done another. Managers are the bridge between what is being told and what is actually getting done. Great managers define the realities for their team, and communicate that back up to leadership. They understand information sharing is vital for success in transformation and growth within the organization. Great managers provide work environments focused on continuous learning for their employees. They create clear and focused workflows for their teams. Shielding them from naysayers or negative process and time sucks. They influence executive direction and gain acceptance. Great managers are able to keep their teams unified and committed. Great engineers stay with and work for Great Managers.

DevOps need More Great Managers!

Key Characteristics of Great Managers:

1. Emotional Intelligence/Empathy: Great managers understand their team. They have personal connections with each member. They know how and when to engage teammates. Great managers accept their role as coaches, mentors, therapist, and sometimes friends. They are able to build  mutual trust and respect from their teams. Great managers lift team members up to achieve more than even they, themselves, believed they were capable of.

2. Teaching and Learning: Great managers create an environment of “Teaching and Learning”. They understand the distribution of information is a strength not a weakness. Dustin Collins wrote a great blog highlighting the risk when information is not shared in “From Zero to Hero, Or There and Back Again”. Great Managers believe in continuous learning, learning with the intent to teach others. This single methodology builds an expectation for everyone on the team to learn with the  intent to teach. Every member of the team is always learning and teaching.

3. Strong Communication/ Listening skills: Great managers create environments of transparency and sharing simply by listening. They understand that all information is important. Being a great communicator also means understanding what questions to ask and why. QBQ by John Miller is a great reference. Uncovering true motivations behind one’s dialog is essential to being a great manager. They deflect white noise and focus the team on the main message. They define clear goals and expectations. This is especially important when organizations are shifting culturally. Change is only scary when information, expectations and direction are not fully and properly communicated and understood. Leadership voids cause fear and fear kills innovation.

4. Protectors/Blockers: Great managers understand their role to block and shield their teams from issues outside of their influence. They take accountability for the actions of their team. If something goes wrong they take the blame. When things go well, the praise goes to the team. Great managers care more for their team’s success and well being than their own. They create a “No Fault Environment”, allowing employees to try new things without the fear of failing.

Great managers are honest with their team, leadership, and especially themselves. They are constantly defining realities for the people around them. Great Managers have the respect of executive leadership. They are able to influence real change at the very top of an organization. They create stable work environments for their teams. PuppetLabs State of DevOps “Engineer do not leave companies, they leave managers.” Great Managers build and retain cohesive teams. Employee churn is one of the most disruptive elements to successful DevOps transformations. DevOps is hard! Having great leadership in place is essential for guidance and reassurance when something doesn’t go as planned, and it WILL happen. Great managers know how to respond in these situations without sounding the alarms. Great Managers keep the ship calm in rough waters to ensure smooth sailing and success for all.

How to build a microservice?

Posted on

By Sonu Meena

Microservice architecture is the advancement over existing architecture which is more commonly found based around monolithic architecture.

In monolithic architecture all the functionalities and features are offered through single piece of software. Generally, this software is built around three tiers: Data, Application (business layer) and View tier.

This has been the de facto standard for starting any software development at least in startups now. It has its benefits that makes it the most sought architecture to begin development with.

  • Faster to build
    It shorten time to market and hence every new tech company start with this architecture.
  • Work best in constraint environment
    In limited number of developers and budget this is proven to work best for both: Marketing and Devs.
    Marketing guys Or Business leaders want feature out very fast and with few hands working on one piece of software they are able to meet this deadlines pretty well.

So, starting with boring monolithic is good.


Beings said, these benefits don’t last forever. With surge in traffic comesscalability issues. Team put more number of resource. They add more CPUs, more RAM, and more disk to cope up with the growth. This is called vertical scaling where existing servers are upgraded to support growing number of users.
Vertical scaling also has its limitation. Beyond a point you cannot scale and then your application response time again started increasing thereby affecting user experience.

Before it go worse and to further scale we look into horizontal scalabilityoption. This involve running multiple instances of the application behind fault tolerant, highly available load-balancer. And to answer spikes in traffic you simply add more number of instances. This is easy? but wait…

Scalability issue isn’t confined to internet traffic only that you can answer every time with increasing numbers alone. Its effect is more profound internally at development side. With more and more feature keep adding you end up knitting a giant ball that’s getting unwieldy to carry around and issues like following pops up:

  1. Codebase gets bigger and bigger 

    Developer keep pushing more features, thereby adding more number of dependencies and more number of lines of code. Now here comes the problem. Bigger code now takes more space and bandwidth to pull or push. It means code will now take more time to deploy. Since it’s lots of dependencies releasing new features may produce downtime. Downtime is bad for any business. It literally mean loss in revenue.Read: Goes Down, Loses $66,240 Per Minute

  2. More time to build and test
    Individual developer working on particular feature now have to download the entire codebase with all the dependencies to make it work on his little workstation. If he has to test even only his changes, he has to run tests against entire software to ensure the change doesn’t break anything.
  3. on-boarding get slower 

    Now here comes another challenge. Your company is growing and hired some more developers. Codebase is one and to teach new developer how it works will now take considerably lot amount of time. This makes new developer boarding process more slower to make him familiar with all the feature is another challenge.

  4. With more features comes more bugs 

    Since many developers are now working together on same codebase, they all are pushing features in the same repository. It creates favorable opportunity for bugs to propagate unnoticed and create havoc in production.
    No matter how hard you try to avoid them they will be missed anyhow. Earlier codebase was small and so it was easy to track and kill them. This time with many releases going per day they are getting more and more harder to trace. Unless you have team of perfectionist bugs are unavoidable.

& the list continue to grow.

Solution: Componentization


So, horizontal scalability answer your scalability problem partially. Rest you have to answer by splitting along y-axis as per scalecube model.
Y-Axis split of scalecube model says split your big suit into multiple smaller components i.e componentize it

Componentization should be in a way that each component is individuallyupgradable and replaceable. They should talk to each other on published interfaces built with well known  industry standard technologies like REST and JSON so that it don’t sound esoteric to end users. They can either communicate through synchronous protocol like HTTP or more preferably asynchronous protocol like AMQP.

Componentization brings following benefits to the team:

  • Freedom to choose technology

Team is divided with cross-functional domain. They can choose the right tool for the right job as long as they are in constraint.
These constraints are:


  1. tools should not be esoteric to team and should not incur extra cost while doing linear scalability
  2. Technology should be standard so that even end user don’t need to take pain of learning it explicitly.


  • C.A.L.M.S

It brings devOps culture into practice inadvertently. There would be more communication among teams now as team are cross-functional. Using devops life-cycle they get the best out of this.


Principles Of Chaos Engineering

Posted on

Last Update: 2015 September

Chaos Engineering is the discipline of experimenting on a distributed system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.

Advances in large-scale, distributed software systems are changing the game for software engineering.  As an industry, we are quick to adopt practices that increase flexibility of development and velocity of deployment.  An urgent question follows on the heels of these benefits: How much confidence we can have in the complex systems that we put into production?

Even when all of the individual services in a distributed system are functioning properly, the interactions between those services can cause unpredictable outcomes.  Unpredictable outcomes, compounded by rare but disruptive real-world events that affect production environments, make these distributed systems inherently chaotic.

We need to identify weaknesses before they manifest in system-wide, aberrant behaviors.  Systemic weaknesses could take the form of: improper fallback settings when a service is unavailable; retry storms from improperly tuned timeouts; outages when a downstream dependency receives too much traffic; cascading failures when a single point of failure crashes; etc.  We must address the most significant weaknesses proactively, before they affect our customers in production.  We need a way to manage the chaos inherent in these systems, take advantage of increasing flexibility and velocity, and have confidence in our production deployments despite the complexity that they represent.

An empirical, systems-based approach addresses the chaos in distributed systems at scale and builds confidence in the ability of those systems to withstand realistic conditions.  We learn about the behavior of a distributed system by observing it during a controlled experiment.  We call this Chaos Engineering.


To specifically address the uncertainty of distributed systems at scale, Chaos Engineering can be thought of as the facilitation of experiments to uncover systemic weaknesses.  These experiments follow four steps:

  1. Start by defining ‘steady state’ as some measurable output of a system that indicates normal behavior.
  2. Hypothesize that this steady state will continue in both the control group and the experimental group.
  3. Introduce variables that reflect real world events like servers that crash, hard drives that malfunction, network connections that are severed, etc.
  4. Try to disprove the hypothesis by looking for a difference in steady state between the control group and the experimental group.

The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system.  If a weakness is uncovered, we now have a target for improvement before that behavior manifests in the system at large.


The following principles describe an ideal application of Chaos Engineering, applied to the processes of experimentation described above.  The degree to which these principles are pursued strongly correlates to the confidence we can have in a distributed system at scale.

Build a Hypothesis around Steady State Behavior

Focus on the measurable output of a system, rather than internal attributes of the system.  Measurements of that output over a short period of time constitute a proxy for the system’s steady state.  The overall system’s throughput, error rates, latency percentiles, etc. could all be metrics of interest representing steady state behavior.  By focusing on systemic behavior patterns during experiments, Chaos verifies that the system does work, rather than trying to validate how it works.

Vary Real-world Events

Chaos variables reflect real-world events.  Prioritize events either by potential impact or estimated frequency.  Consider events that correspond to hardware failures like servers dying, software failures like malformed responses, and non-failure events like a spike in traffic or a scaling event.  Any event capable of disrupting steady state is a potential variable in a Chaos experiment.

Run Experiments in Production

Systems behave differently depending on environment and traffic patterns.  Since the behavior of utilization can change at any time, sampling real traffic is the only way to reliably capture the request path.  To guarantee both authenticity of the way in which the system is exercised and relevance to the current deployed system, Chaos strongly prefers to experiment directly on production traffic.

Automate Experiments to Run Continuously

Running experiments manually is labor-intensive and ultimately unsustainable.  Automate experiments and run them continuously.  Chaos Engineering builds automation into the system to drive both orchestration and analysis.

Chaos Engineering is a powerful practice that is already changing how software is designed and engineered at some of the largest-scale operations in the world.  Where other practices address velocity and flexibility, Chaos specifically tackles systemic uncertainty in these distributed systems.  The Principles of Chaos provide confidence to innovate quickly at massive scales and give customers the high quality experiences they deserve.

DevOps Growing in Popularity Despite Unclear Terminology

Posted on

By David Ramel

DevOps in the enterprise is growing, but interpretations of what the term actually means varies widely, a recent study found.

Delphix Corp., a Data-as-a-Service (DaaS) company, partnered with Gleanster Research to survey more than 2,000 DevOps leaders and practitioners, concluding that “DevOps continues to gain momentum, but data issues — including data security — are major challenges, while the definition of DevOps lacks consistency and success metrics among its leaders and practitioners within IT organizations.”

That lack of consistency is shown by the various definitions of the term DevOps proffered by respondents, such as:

  • Developers and system administrators collaborating to ease the transition between development and production (listed by 84 percent of respondents).
  • Using infrastructure automation to facilitate self-service provisioning of infrastructure by development teams (69 percent).
  • Evolving operations to meet the demands of agile software development teams (60 percent).
  • Developers taking full responsibility for all operations tasks (42 percent).
  • Increasing frequency of deployments to uncover defects earlier in the development lifecycle (35 percent).

For another take, Wikipedia defines DevOps as “a software development method that stresses communication, collaboration, integration, automation and measurement of cooperation between software developers and other [IT] professionals.”

Top Four Pressures Causing Organizations To Invest in DevOps

[Click on image for larger view.] Top Four Pressures Causing Organizations To Invest in DevOps (source: Delphix Corp.)

The survey, titled “2015 Annual State of DevOps,” said that while DevOps is one of the hottest trends in the industry, it’s also one of the most ill-defined. “For some people, embracing DevOps is about managing IT resources with Chef, Puppet or CFEngine,” the survey said, “and for others it is about using tools like Jenkins to automate deployments to cloudbased infrastructure. For several of the organizations we surveyed, DevOps was simply about making sure that developers and operations professionals were communicating efficiently.”

Whatever it is, DevOps is gaining traction, according to the report, which stated that nearly every responding organization was practicing the technique or planned to do so within 24 months. The top three reasons for that were: to deliver software faster (66 percent); identify bugs earlier (44 percent); and deliver software more frequently (43 percent).

As for impediments to successful DevOps implementations, a question asking respondents to identify the top two challenges to organizational DevOps initiatives yielded these responses: application teams move faster, the rest of IT struggles to keep up (92 percent); testing environments are limited due to data management challenges (90 percent); several DevOps groups compete for limited resources and budgets (82 percent).

If successful, the primary reported benefits of DevOps were divided into key performance indicators (KPIs) related to software releases and software quality. “It’s abundantly clear that the economic impact of time is a theme that ripples through all facets of DevOps,” the survey stated. “This comes as no surprise given the fact that survey respondents indicated that on average 40 percent of their day is spent re-coding due to bugs. Respondents also indicated it takes an average of 2 hours to reset an environment after a test cycle; reducing this time could have a real and substantial impact on efficiency and effectiveness.”

And yes, the report did propose its own global definition of DevOps:

DevOps is more than just the close collaboration of two departments (development and operations) within IT, it is more than just managing infrastructure with Chef or Puppet, and DevOps is much more than a specific collection of tools and techniques used to automate deployments and manage infrastructure.The term “DevOps” refers to the transformation IT experiences when cross-functional teams develop and deliver software across the full spectrum of IT systems. From software architecture and design to system administration and production support, the term “DevOps” refers to a style of IT management and implementation that places an emphasis on automation and iterative delivery of software, while also empowering developers to manage portions of the software delivery process that were previously inaccessible due to specialization within IT.

DevOps tools and practices have one thing in common: they focus on reducing time to market and making it possible to extend the frequent iterations of Agile into infrastructure and data environments. Overall DevOps is inseparable from both agile software development and cloud computing. As a term, “DevOps” stands for “our infrastructure moves as quickly as our developers need it to.”


Top 10 important DevOps controls

Posted on

by James Henderson

DevOps, the integration of development and operations teams to eliminate conflicts and barriers, often leads to more features in business applications, developed in a faster time and with greater efficiencies.

But the very features that make DevOps attractive to organisations can cause concern for assurance, security and governance practitioners.

A new guide from global IT association ISACA outlines 10 key controls companies need to consider as they embrace DevOps to achieve reduced costs and increased agility.

“DevOps can introduce new risk but, done right, it also mitigates other risk,” says Bhavesh Bhagat, CEO EnCrisp.

According to DevOps Practitioner Considerations, those controls are:

1. Automated software scanning

3. Web application firewall

4. Developer application security training

5. Software dependency management

6. Access and activity logging

7. Documented policies and procedures

8. Application performance management

9. Asset management and inventorying

10. Continuous auditing and/or monitoring

“Because DevOps adoption changes the environment and often impacts a company’s carefully crafted control environment and accepted level of risk, governance, security and assurance professionals need to play a key role,” Bhagat adds.

For Bhagat, the governance decisions relating to risk, including decisions made in the past, may require rethinking, and performance metrics on which business decisions are based may need to be adjusted.

“Furthermore, many security controls that are intertwined with the development process may be compromised,” Bhagat adds.