Docker, Red Hat & Linux: How containers can boost business and save time for developers

Posted on

image

Analysis: A lot of work is being put into making container technology vital to your business.

Container technology is in a period of explosive growth, with usage numbers nearly doubling in the last few months and vendors increasingly making it available in their portfolios.

Docker, which has popularised the technology, reported that it has now issued two billion ‘pulls’ of images, up from 1.2 billion in November 2015.

But containerisation technology isn’t new, back in 1979 Unix V7 developed chroot system call, which provided process isolation, but it took untill 2000 for early container technology to be developed. In 2000, FreeBSD Jails developer Derrick Woolworth introduced jails an early container technology.

Google is perhaps the biggest name user of container style technology, it developed Process Containers in 2006. Google’s creation of Borg, a large-scale cluster management software system is attributed with being one of the key factors behind the company’s rapid evolution.

Borg works by parcelling work across Google’s fleet of computer servers, it provides a centrail brain for controlling tasks across the company’s data centres. So the company hasn’t had to build separate clusters of servers for each software system, e.g.: Google Search, Gmail, Google Maps etc.

The work is divided into smaller tasks with Borg sending tasks when it finds free computing resources.

At their most basic this is essentially what containers do, it offers deployable bits of code that are independent so that applications can be built.

Unix and Linux have been using containers for a number of years but the technology’s recent growth can really be attributed to Docker, whose partners include the likes of IBM, Amazon Web Services and Microsoft.

This isn’t to say that Docker is the only company playing in the field, nor that it has perfected the technology.

As mentioned earlier, Google is a major player with technologies, it has contributed to numerous container-related open source projects such as Kubernetes. Microsoft is another big player having added container support with Windows Servers Containers and Azure Container Service for its cloud platform.

AWS has developed a cluster management and scheduling engine for Docker called EC2 Container Services off the back of the popularity of container deployment on the EC2 platform.

Despite these deployments there are areas that vendors have been looking at with the technology that needs to be improving.

Security is one of the areas.

Gunnar Hellekson, director, product management, Red Hat Enterprise Linux and Enterprise Virtualization, and Josh Bressers, senior product manager, security, wrote in a blog post how containers are being fixed in the wake of the glibc vulnerability.

The flaw means that attackers could remotely crash or force the execution of malicious code on machines without the knowledge of the end user.

The blog post points out that the development of container scanners is just a “paper tiger” that may help find a flaw but doesn’t really help to fix it.

Red Hat says: “We provide a certified container registry, the tools for container scanning, and an enterprise-grade, supported platform with security features for container deployments.”

This has been a common message that I have come across when reading about container deployments, vendors integrating with Docker and other container technology – typically there is a focus on improving security.

Although there are security improvements to be made, containers hold significant promise for the enterprise, here are three examples of where containers can help.

 

Containers can help with the rise of bring your own devices

The problem is that as BYOD increases it has become more important to separate work from play. This can be done through a secure container that provides authenticated, encrypted areas of a user’s mobile device.

The point of doing this is so that the personal side of a mobile device can be insulated from sensitive corporation information.

More distributed apps

Docker can distribute the application development lifecycle by enabling distributed apps to be portable and more dynamic. This means that those distributed applications are composed from multiple functional, interoperable components.

Basically this is the next generation of application architectures and processes that are designed to support business innovation.

 

More innovation

Continuing on the innovation front, because the developer no longer has to focus on the labour-intensive work of typing the application into hardware and software variables, the developer is free to develop new things.

This is similar to one of the benefits of using a cloud vendor for Infrastructure-as-a-Service. Rather than dealing with the time consuming tasks that don’t add a great deal of value, the developer can do other things.

 

The top three Docker application areas have been seen in test and quality assurance, web facing apps and big data enterprise apps.

Its rising popularity can be linked to the rise of cloud, partly because web-scale players have used the containerisation technology to such good effect.

However, unlike cloud, Docker adoption has been driven primarily from the middle out; nearly 47% of Docker decisions have been made by middle management and 24% by a grassroots effort. Cloud meanwhile started primarily as a top down decision being pushed by CIOs.

Given its growing popularity and the benefits it can deliver to the organisation and developers, the technology is likely to continue on its rise to the top, particularly when it is taken into account the numerous big name vendors that are working on perfecting it.

DevSecOps: How DevOps and Automation Bolster Security, Compliance

Posted on

DevOps bridges the gap between Development and Operations to accelerate software delivery and increase business agility and time-to-market. With its roots in the Agile movement, DevOps fosters collaboration between teams and streamlines processes, with the goal of breaking silos in order to “go fast”.

Information Security (InfoSec) and compliance are critical to businesses across the globe, especially given past examples of data breaches and looming cybersecurity threats. InfoSec has long been thought of as the group that “slows things down” – the wet towel to your DevOps efforts – often requiring a more conservative approach as a means of mitigating risk. Traditionally, DevOps was viewed as a risk to InfoSec, with the increased velocity of software releases seen as a threat to governance and security/regulatory controls (these, by the way, often require the separation of duties, rather than the breaking of silos.)

Despite some initial pushbacks, enterprises that have taken the “DevOps plunge” have shown – consistently – that DevOps practices actually mitigate potential security problems, discover issues faster and address threats more quickly. This has led  to InfoSec increasingly embracing automation and DevOps practices more and more, as the “security blanket” that enables — and enforces — security, compliance and auditability requirements. This makes DevOps a resource for InfoSec, rather than a threat.

As a philosophy, DevOps focuses on creating a culture and an environment where Dev, QA, Ops, the Business and other stakeholders in the organizations work in collaboration towards a shared goal. We now see DevOps evolving to DevSecOps – with InfoSec aligning with your DevOps initiative, and security requirements made a key tenant of your DevOps practices — and your DevOps benefits.

The Security Opportunity of DevOps

DevOps provides a huge opportunity for better security. Many of the practices that come with DevOps — such as automation, emphasis on testing, fast feedback loops, improved visibility, collaboration, consistent release practices, and more — are fertile ground for integrating security and auditability as a built-in component of your DevOps processes.

DevOps automation spans the entire pipeline- from code development, testing, to infrastructure configuration and deployment. When done right, DevOps enables you to:

  1. Secure from the start: Security can be integrated from the early stages of your DevOps processes, and not as an ‘afterthought’ at the very end of the software delivery pipeline. It becomes a quality requirement – similar to other tests ran as part of your software delivery process. In a similar way to how CI enables “shifting left”, accelerating testing and feedback loops to discover bugs earlier in the process and improve software quality, DevOps processes can incorporate automated security testing and compliance
  2. Secure, automatically: As more and more of your tests and processes are automated – you have less risk of introducing security flaws due to human error, your tests are more efficient and you can cover more ground, and your process is more consistent and predictable- so if something does break, it’s easier to pin-point and fix.
  3. Secure throughout: By using tools that are shared across the different functions, or an end-to-end DevOps Automation platform that spans Development, Testing, Opsand Security – organizations gain visibility and control over the entire SDLC, making the automated pipeline a “closed loop” process for testing, reporting and resolving security concerns.
  4. Get everyone on the same page/pipeline: By integrating security tools and tests as part of the pipeline used by Dev and Ops to deploy their updates, InfoSec becomes a key component of the delivery pipeline, and an enabler of the entire process (rather than pointing fingers at the very end!)
  5. Fix things quickly: Unfortunately, the occasional security breach or vulnerability might come up – requiring you to act quickly to resolve the issue (think Heartbleed, for example.) DevOps accelerates your lead time – so that you can develop, test and deploy your patch/update more quickly. In addition, the meticulous tracking provided by some DevOps platforms into the state of all your applications, environments and pipeline stages greatly simplifies and accelerates your response when you need to release your update. When you can easily know exactly which version of the application, and all its components/stack, is deployed on which environment, you can quickly pin-point the component of the application that requires the update, identify the instances that require attention and quickly roll out your updates in a faster, consistent, and repeatable deployment process by triggering the appropriate workflow.
  6. Enable developers, while ensuring governance: DevOps emphasizes the streamlining of processes across the pipeline to have consistent development, testing and release practices. Your DevOps tools and automation can be configured to enable developers to be self-sufficient and “get things done”, while automatically ensuring access controls and compliance. For example, as a resolution to the growing “shadow IT” phenomena, we see a lot of organizations establishing an internal DevOps service for a dev/test cloud – with shared repositories, workflows, deployment processes etc. This allows engineers on-demand access to infrastructure (including Production), while automatically enforcing access control, security measures, approval gates and configuration parameters – to avoid configuration drift or inconsistent processes. In addition, it ensures all instances across all environment – no matter whether in Development, QA or production – are identified, tracked, operating within pre-set guidelines, and can be monitored and managed by IT.
  7. Secure both the code, and the environments: By creating manageable systems that are consistent, traceable and repeatable you ensure that your environment is reproducible, traceable and that you know who accessed it and when.
  8. Enable 1-click compliance reporting: Automated processes come with the extra benefits of being consistent, repeatable, with predictable outcomes for similar actions/tests, and they can be automatically logged and documented. Since DevOps spans your entire pipeline, it can provides traceability into traceability from code change to release. If you have a DevOps system system you can rely on, auditing becomes much easier. As you’re automating things – from your build, test cycles, integration cycles, deployment and release processes – your DevOps automation platform has access to a ton of information that is automatically logged in great detail. That, in effect, becomes your audit-trail, your security log, and your compliance report – all produced automatically, with no manual intervention or you having to spend hours backtracking your processes or actions in order to produce the report.

DevSecOps is enables organizations to achieving speed without risking stability and governance. Security and compliance controls should be baked-in as an integral part of your DevOps processes that manage the code being developed all the way through to Production. By implementing DevOps processes that incorporate security practices from the start you create an effective and viable security layer for your applications and environments that will serve as a solid foundation to ensure security and compliance in the long run, in a more streamlined, efficient and proactive way.

Can We Stop Talking About DevOps?

Posted on

By Tom Hall

I first heard the term DevOps from a friend in the software business when I was working at Dell HQ in Round Rock, TX. He recommended that I read The Phoenix Project, by Gene Kim, Kevin Behr and George Spafford. I did read it and I found it a fascinating story, but as a long-time network administrator (to oversimplify as much as possible) and having never been involved in developing software, I wasn’t quite sure how it applied to me. It was enough to pique my interest though and drove me to volunteer at a DevOpsDays event in Austin. Within weeks I left Dell and joined a small software company, where I could have a better chance of getting my arms around this slippery pig called DevOps and more hands-on practice with the related tools and techniques.

So imagine my surprise when a year later, the same old friend who introduced me to DevOps suggested that the end-goal of a DevOps transformation should be that we stop talking about DevOps. Uh-oh, I thought, he’s lost the plot. But he made his case by pointing out that adding testers to dev teams–a typical step in any Agile transformation–doesn’t result in new devtest or agiledev teams, they remain simply “dev teams”. Once a business/group has fully internalized the Agile or DevOps values and behaviors, he argued, ‘working in an Agile or DevOps way’ becomes simply ‘working’. This is why many DevOps promoters, myself included, have a negative reaction to seeing ‘DevOps’ in job titles, job descriptions, or department names. While it simplifies the job of identifying which job candidates have a DevOps mindset or skillset, it effectively excuses everyone else in the organization from taking responsibility for change. “Doing DevOps” becomes the job of a particular individual, team or department instead of a philosophy internalized by the organization.

Of course the horse is already out of the barn on that one. Each year we see an increase in the number of businesses advertising for DevOps Engineers, forming DevOps departments or listing DevOps among the required skills for a position. It doesn’t matter whether we think this is a good thing or bad thing, evolution will take its course and there’s little we can do to stop it. What we can do is observe, acknowledge and adapt. Sure, the dangers are real. A DevOps department can easily become just a third silo or the DevOps Engineer might become the scapegoat for everything that goes wrong between development and operations. But if a DevOps Engineer, VP of DevOps or DevOps department can form, contribute to and/or nurture an environment of increased empathy, faster feedback and continual growth, that’s awesome; put the focus there. Don’t spin your wheels arguing that they just “don’t get it” or “they’re doing it wrong”.

I’ve spent the past couple years researching, promoting and attempting to practice this thing we call DevOps. While it’s widely believed that DevOps is impossible to define, I am firmly in the camp of those who believe that empathy is, to quote Jeff Sussna, “the essence of DevOps”. Just as the sales team needs to have empathy for its customers, everyone in the value chain needs to have empathy for each other. I’m aware that many people find this term too “touchy-feely”, but the business benefit of more harmonious interaction between one’s teams and between the organization and its customers is clear. The faster and more effective one is at converting solutions into real value for customers, the more successful one will be in the market. Thus being empathetic isn’t just being a good citizen, it’s being a good businessperson.

Taking it a step further, someone on Twitter recently suggested that “DevOps has nothing to do with technology”. The statement seemed ridiculous on its face. After all, the term ‘DevOps’– literally a portmanteau formed from the words ‘developers’ and ‘operations’–was coined by Patrick Dubois, a technologist intent on improving the process of building and deploying software. But as I thought about it I started to understand. As the DevOps movement has evolved, the most fundamental components its promoters have identified, from Gene Kim’s “Three Ways” (Systems Thinking, Amplify Feedback Loops, Culture of Continual Experimentation and Learning), to John Willis and Damon Edwards’ CAMS (Culture, Automation, Measurement, Sharing), to Dave Zweiback’s ICE (Inclusivity, Complex Systems, Empathy), are primarily about how people work. It doesn’t matter if one is building software or running a school, DevOps principles are sound.

I recently tweeted the observation that “anyone who attempts to define DevOps in a paragraph is trying to sell you something”, adding “not necessarily a bad thing, just buyer beware”. This wasn’t a criticism of tool providers; many tools, from build automation to log monitoring, are essential to an effective DevOps transformation. It was a reminder that the problems and solutions DevOps is concerned with are complex and nuanced. There is no prescribed toolset or checklist of actions one can take to be successful; an effective DevOps transformation requires thoughtful, committed action across the whole range of people, practices and products one uses in their work. It might take months or years of practice, but maybe one day we can all stop talking about DevOps.

DevOps Needs Great Managers

Posted on

by David Fredricks

There has been much discussion about DevOps and the benefits it offers to organizations. A lot of what I’ve read discusses defining what DevOps is and how it will improve the bottom line. Much of the information is general in nature, more broad stroke, theoretical type content. According to Wikipedia, “Getting Developers and Operations talking to one another”, “Tearing down the wall of confusion”, “Collaboration between different departments”, “Automating your systems”, etc, etc. This really makes it difficult for one to take specific actions.

From my experience, defining DevOps is personal. The direct challenges that one deals with on a day-to-day basis really shape the concepts of DevOps and the benefits one hopes to accomplish. This makes it difficult to plug and play the process. What works for one organization is not necessarily the right direction for another.  The best case scenario is starting from scratch and building your people, process, and policies around core best practices. (This is why most of the success stories you read about come from greenfield startups). Unfortunately most do not have this luxury.  Reality is companies have layers upon layers of legacy systems, process and people interconnected with technology running in production. Herein lies the challenge.

Achieving a successful DevOps transformation means complete transparency, disclosing all of the cultural bias, loyalties, motivations and scars that internally exist. This is a huge undertaking. Many of the failures in transformation happen because companies are not transparent with themselves.  Having a deep understanding of systems, process, people and customers is vital to being successful. What? Where? When? Why? These questions must be addressed, understood, and accepted before you can move to the “How”.

This is why managers are the key to success. No one else understands the complexities of the business more than the managers. They know where the “skeletons” are buried, who is in bed with who, what systems work, what systems do not. Why some process is done one way, and in other cases done another. Managers are the bridge between what is being told and what is actually getting done. Great managers define the realities for their team, and communicate that back up to leadership. They understand information sharing is vital for success in transformation and growth within the organization. Great managers provide work environments focused on continuous learning for their employees. They create clear and focused workflows for their teams. Shielding them from naysayers or negative process and time sucks. They influence executive direction and gain acceptance. Great managers are able to keep their teams unified and committed. Great engineers stay with and work for Great Managers.

DevOps need More Great Managers!

Key Characteristics of Great Managers:

1. Emotional Intelligence/Empathy: Great managers understand their team. They have personal connections with each member. They know how and when to engage teammates. Great managers accept their role as coaches, mentors, therapist, and sometimes friends. They are able to build  mutual trust and respect from their teams. Great managers lift team members up to achieve more than even they, themselves, believed they were capable of.

2. Teaching and Learning: Great managers create an environment of “Teaching and Learning”. They understand the distribution of information is a strength not a weakness. Dustin Collins wrote a great blog highlighting the risk when information is not shared in “From Zero to Hero, Or There and Back Again”. Great Managers believe in continuous learning, learning with the intent to teach others. This single methodology builds an expectation for everyone on the team to learn with the  intent to teach. Every member of the team is always learning and teaching.

3. Strong Communication/ Listening skills: Great managers create environments of transparency and sharing simply by listening. They understand that all information is important. Being a great communicator also means understanding what questions to ask and why. QBQ by John Miller is a great reference. Uncovering true motivations behind one’s dialog is essential to being a great manager. They deflect white noise and focus the team on the main message. They define clear goals and expectations. This is especially important when organizations are shifting culturally. Change is only scary when information, expectations and direction are not fully and properly communicated and understood. Leadership voids cause fear and fear kills innovation.

4. Protectors/Blockers: Great managers understand their role to block and shield their teams from issues outside of their influence. They take accountability for the actions of their team. If something goes wrong they take the blame. When things go well, the praise goes to the team. Great managers care more for their team’s success and well being than their own. They create a “No Fault Environment”, allowing employees to try new things without the fear of failing.

Great managers are honest with their team, leadership, and especially themselves. They are constantly defining realities for the people around them. Great Managers have the respect of executive leadership. They are able to influence real change at the very top of an organization. They create stable work environments for their teams. PuppetLabs State of DevOps “Engineer do not leave companies, they leave managers.” Great Managers build and retain cohesive teams. Employee churn is one of the most disruptive elements to successful DevOps transformations. DevOps is hard! Having great leadership in place is essential for guidance and reassurance when something doesn’t go as planned, and it WILL happen. Great managers know how to respond in these situations without sounding the alarms. Great Managers keep the ship calm in rough waters to ensure smooth sailing and success for all.

How to build a microservice?

Posted on

By Sonu Meena

Microservice architecture is the advancement over existing architecture which is more commonly found based around monolithic architecture.

In monolithic architecture all the functionalities and features are offered through single piece of software. Generally, this software is built around three tiers: Data, Application (business layer) and View tier.

This has been the de facto standard for starting any software development at least in startups now. It has its benefits that makes it the most sought architecture to begin development with.

  • Faster to build
    It shorten time to market and hence every new tech company start with this architecture.
  • Work best in constraint environment
    In limited number of developers and budget this is proven to work best for both: Marketing and Devs.
    Marketing guys Or Business leaders want feature out very fast and with few hands working on one piece of software they are able to meet this deadlines pretty well.

So, starting with boring monolithic is good.

But…

Beings said, these benefits don’t last forever. With surge in traffic comesscalability issues. Team put more number of resource. They add more CPUs, more RAM, and more disk to cope up with the growth. This is called vertical scaling where existing servers are upgraded to support growing number of users.
Vertical scaling also has its limitation. Beyond a point you cannot scale and then your application response time again started increasing thereby affecting user experience.

Before it go worse and to further scale we look into horizontal scalabilityoption. This involve running multiple instances of the application behind fault tolerant, highly available load-balancer. And to answer spikes in traffic you simply add more number of instances. This is easy? but wait…

Scalability issue isn’t confined to internet traffic only that you can answer every time with increasing numbers alone. Its effect is more profound internally at development side. With more and more feature keep adding you end up knitting a giant ball that’s getting unwieldy to carry around and issues like following pops up:

  1. Codebase gets bigger and bigger 

    Developer keep pushing more features, thereby adding more number of dependencies and more number of lines of code. Now here comes the problem. Bigger code now takes more space and bandwidth to pull or push. It means code will now take more time to deploy. Since it’s lots of dependencies releasing new features may produce downtime. Downtime is bad for any business. It literally mean loss in revenue.Read: Amazon.com Goes Down, Loses $66,240 Per Minute

  2. More time to build and test
    Individual developer working on particular feature now have to download the entire codebase with all the dependencies to make it work on his little workstation. If he has to test even only his changes, he has to run tests against entire software to ensure the change doesn’t break anything.
  3. on-boarding get slower 

    Now here comes another challenge. Your company is growing and hired some more developers. Codebase is one and to teach new developer how it works will now take considerably lot amount of time. This makes new developer boarding process more slower to make him familiar with all the feature is another challenge.

  4. With more features comes more bugs 


    Since many developers are now working together on same codebase, they all are pushing features in the same repository. It creates favorable opportunity for bugs to propagate unnoticed and create havoc in production.
    No matter how hard you try to avoid them they will be missed anyhow. Earlier codebase was small and so it was easy to track and kill them. This time with many releases going per day they are getting more and more harder to trace. Unless you have team of perfectionist bugs are unavoidable.

& the list continue to grow.

Solution: Componentization

 

So, horizontal scalability answer your scalability problem partially. Rest you have to answer by splitting along y-axis as per scalecube model.
Y-Axis split of scalecube model says split your big suit into multiple smaller components i.e componentize it

Componentization should be in a way that each component is individuallyupgradable and replaceable. They should talk to each other on published interfaces built with well known  industry standard technologies like REST and JSON so that it don’t sound esoteric to end users. They can either communicate through synchronous protocol like HTTP or more preferably asynchronous protocol like AMQP.

Componentization brings following benefits to the team:

  • Freedom to choose technology



Team is divided with cross-functional domain. They can choose the right tool for the right job as long as they are in constraint.
These constraints are:

 

  1. tools should not be esoteric to team and should not incur extra cost while doing linear scalability
  2. Technology should be standard so that even end user don’t need to take pain of learning it explicitly.

 

  • C.A.L.M.S


It brings devOps culture into practice inadvertently. There would be more communication among teams now as team are cross-functional. Using devops life-cycle they get the best out of this.