DevOps is a philosophy and an approach to software delivery that centers on collaboration for testing, packaging, and deploying software with the aim of enhancing the cadence and reliability of software releases. Growing evidence suggests a well-designed and robust DevOps practice increases the velocity and stability of software deployments while also reducing failure recovery time and software update lead times. DevOps is crucial to the success of software-driven organizations in the cloud era, enhancing IT responsiveness and customer satisfaction. That’s why an increasing number of industry leaders are embracing DevOps by investing in tooling and promoting collaborative mindsets and team dynamics.
What is DevOps?
DevOps represents the broader application and culmination of a movement that began with lean thinking and agile development whose ultimate goal was shipping high-quality software quickly. Agile development generally centers on the role of software engineers and focuses on rapid, incremental development of software. In the cloud era when software is increasingly consumed as a service (rather than installed on premises), it is not considered shipped until it is live in production. In keeping with the agile philosophy, there is increasing emphasis on continual, incremental, and rapid delivery of software functionality. Thus, agile has necessarily broadened to include speed and quality in operations–the set of activities that transform software from code complete to supported in production (e.g., build, test, provisioning, configuration, deployment, and ongoing management). Shipping software quickly like this requires the joint cooperation of both developers (Dev) and IT operations (Ops).
The DevOps movement is an acknowledgment and response to this need. Thus, DevOps is fluid and avoids much of the traditional handoff friction and delays between product development (developers and QA) and IT operations through greater collaboration, communication, and joint responsibility for the success of software delivery.
DevOps stands in stark contrast to the traditional enterprise software delivery mindset. Generally comprised of separate software development and IT operations organizations that work independently, most enterprises encounter a disconnect between developers and IT operations and are poorly positioned for rapid software delivery. For example, most enterprise developers generally cannot easily self-provision configured infrastructure (controlled by IT Ops), and thus cannot spin up repeatable, standardized environments. As a result, they end up developing their portion of code in their private environments with unique configurations that enabled efficient development and testing. They then hand it over to IT operations to take the various software artifacts (developed and tested by multiple developers in heterogeneous environments) and then manifest them into a running application deployment that has desirable enterprise characteristics such as high availability, scalability, and security.
This is generally a complex, manual, slow, and error-prone process for IT Ops to accomplish without support from the development team. Problems encountered in deploying software usually leads to friction and mistrust between the teams. Such friction is exacerbated in a modern, continuous delivery context where developers are incentivized to deliver features rapidly, while IT operations is incentivized to ensure stability, and the only way that is practically possible (without modern automation) is by limiting changes.
DevOps is not prescriptive in how it should be practiced. Nevertheless, successful and high-functioning DevOps practices tend to share several characteristics and impact the culture, process and tooling in an organization.
A culture of valuing collective success, collaboration, and joint accountability permeates successful DevOps practices. Developers and IT professionals are jointly held responsible for successful application delivery. This can be facilitated through organizational changes (e.g., functional silos where personnel are matrixed into projects are jettisoned in favor of co-locating development and operations personnel who build and run their services together).
DevOps also places emphasis on empathy and encourages developers and IT operators to understand each other’s roles better so they can adjust their work to easily accommodate the other party and collaborate more effectively. For example, understanding the production deployment environment would enable developers to have a better understanding of potential operational failures and design around them. Alternatively, IT Ops can gain a better understanding of application design and goals helping them optimize the deployment configuration.
Successful DevOps practices actively eliminate potential sources of friction resulting from misaligned success measures (e.g., developers incentivized to release features quickly while IT Ops is incentivized to minimize changes to production deployments).
Manifesting a DevOps culture in practice requires establishing joint processes such that both developers and IT operations have a forum for communication and sharing of expertise, and a well-defined collaboration framework for rapid and structured software delivery and failure recovery. Such a framework, for example, might designate that developers are responsible for self -provisioning of capacity through APIs, but that IT Ops implements and supports them. Or it may provide for an agreement/contract that the APIs will not distinguish between dev, test, or production environments, thereby eliminating environmental parity issues - a significant source of delays and problems.
Another example of joint process might be agreement and use of a centralized store for software dependencies that is maintained current through jointly developed scripts. Over time, such collaboration yields learning, iterative improvement, and the ability to push out incremental and frequent software changes into production with repeatability and little risk of downtime.
Rapid and reliable software delivery requires consistency and repeatability in processes that are streamlined to eliminate unnecessary manual intervention. For example, enterprises routinely struggle with slowdowns related to capacity provisioning, disparity of development and production environments, and manual complex compile/test phases.
High performing DevOps practices utilize shared tooling to help crystallize and streamline collaborative processes such that there is common understanding of the entire software delivery process. As a result, they are able to drive consistency and automation that helps DevOps practitioners deliver faster and avoid spending time in ad hoc fire-fighting during deployment or failure recovery in production. Many enterprise vendors now offer tooling for continuous integration or infrastructure automation and configuration. However, integrating these disparate point solutions is a complex exercise that adds significant overhead, but little to an enterprise’s bottom line.
Why DevOps Matters
Faster time to market
In a software-driven world, building and releasing software quickly to learn customer needs and ward-off competition is crucial for success. Given the complex interdependencies of distributed components in modern enterprise applications, a mature DevOps practice can avoid miscommunication and delays, and provide streamlined software delivery through the joint expertise of both developers and IT Ops. According to the 2016 State of DevOps Report, organizations practicing DevOps are able to (on average) have 200x more frequent deployments and able to handle 2555x shorter lead times (time between the intent to deploy code and the code being in production). Similar findings are showing up in other industry studies, as the chart below from Gartner shows, which lists even more benefits to shifting to a DevOps approach.
Lower risk and smoother deployments
Orchestrating and managing the development, deployment, scaling, securing, patching, and high availability of modern applications is a complex affair, rife with potential sources of failure. A DevOps practice comprised of a collaborative culture and automation for packaging, deployment, monitoring, and management yields consistency and speed in pushing new code and updates to production. Coupled with continuous learning, DevOps teams can identify and eliminate most sources of problems to establish a more robust, streamlined, repeatable, and mature release process that enables uneventful software deployments consistently.
When code does break in production or cause downtime, a DevOps practice is well prepared to diagnose and recover from it faster. With significant automation and monitoring powering most of the release and management processes, a DevOps team can quickly collaborate to trace and determine the source of the failure, and rollback changes or issue fixes. In fact, the 2016 State of DevOps Report suggests that such quick and structured responses to unexpected production problems yield a 24x faster mean time to recovery in organizations practicing DevOps as well as a 3x lower change failure rate.
Higher customer satisfaction and better product-market fit
Releasing features or bug fixes rapidly and reliably not only leads to higher responsiveness and resultant customer satisfaction, but also leads to rapid feedback and faster convergence to the capabilities customers care about most. A DevOps practice lies at the heart of such continuous delivery and fast cycle times. Without continual and tight collaboration between developers and IT operators to manage the resulting complexity, developing, testing, and delivering modern, distributed production-ready software functionality consistently at a rapid pace would not be possible.
“DevOps is firmly in the mainstream today...DevOps is no longer a nice to have or a differentiator but is now essential to compete effectively in the market.”
[Source: Ovum Research; Changes Within and Driven by DevOps; June 2016]
DevOps advocates a new approach to enhancing the software delivery cycle through a combination of culture, process, and tooling. Getting started can be overwhelming, so here are some best practices that may help in thinking through and planning a DevOps practice.
Be sure to customize to your needs. The implementation of DevOps practices should be structured to meet the unique needs of the organization, taking into account the organizational structure, team incentives, current software lifecycle, sources of delay, and opportunities for automation.
Invest in automation and new processes. “Relentless automation is necessary for modern service delivery.” [Source: Forrester Brief: Good DevOps Requires Collaboration, Automation, And Cultural Change; June 2016]. DevOps is substantially enhanced through tooling that enables automation. In many cases, a completely new take on automation is required for success with DevOps. Investing in training teams on working together, establishing new processes and using new tools will be well worth the effort.
According to the report cited above, by research firm Forrester, 82 percent of I&O professionals surveyed claim they are deploying automation solutions in at least one of the following areas: release management, configuration management, and change management.
Realize team building and culture matter. It’s critical to align incentives and goals among the developer and IT operations team members to establish trust and a spirit of collaboration. For example, aligned and actively collaborating teams staffing a single project and collectively measured on all aspects of its success are more likely to be committed to work effectively together, than one where Dev, Ops, or both have functional responsibilities and incentives (e.g., maintain stability or deliver features quickly) for multiple projects but success ownership for none. Although easy to purchase, tooling alone does not confer the benefits of a DevOps practice. It must be used within the context of a cultural philosophy rooted in collaboration and joint responsibility.
Understand organizational dynamics can become roadblocks. With new processes and tooling in place, disruption in current processes and threats to existing organizational mandates are inevitable, especially with compliance, security, and audit functions. It’s critical that these parties buy into the larger vision and become willing stakeholders. Otherwise, a well-intentioned streamlined DevOps practice may become beset by artificial delays. Top-down executive support and multi-team commitment can help work through these complex organizational dynamics. In fact half of the respondents from a 2015 Gartner research survey listed people issues as the biggest challenge to their organization using DevOps.
Recognize integration overhead and complexity issues. Given the number of vendors offering solutions, it’s easy to purchase point solutions to tackle each aspect of the overall DevOps practice being designed. This can, however, result in significant overhead and complexity as enterprises try to integrate different tools into a seamless experience. An unified approach that provides strong opinions rooted in best practices, integrates with the application runtime platform, and yet provides significant flexibility to tailor the automation to an organization’s unique needs can help sidestep these pitfalls and start on a strong footing.
Start small. Crafting a DevOps practice may require experimentation and refinement. It makes sense to start small. To derive meaningful outcomes, start with a real application, but one that is not too large or mission critical where business outcomes are at risk.
Consider legacy workloads. DevOps isn’t just relevant for new workloads. As teams consider modernizing and migrating existing workloads for cloud, consider how to leverage DevOps principles and automation into their lifecycles. Legacy workloads can also derive significant benefits from DevOps in terms of release cadence and reliability/stability of deployments.
Understand DevOps is a journey. DevOps is a practice philosophy. Once a team and practice is defined, remember that it cannot remain static. As workloads, requirements, and operating contexts shift, the practice needs to grow and evolve to continue meeting its raison d’etre: delivering software quickly and reliably.
DevOps introduces a different mindset and new processes into software organizations. The following are some of the key differences that distinguish organizations practicing DevOps compared to more traditional approaches.
DevOps Practices Are...
Traditional Approaches Are...
|Collaboration Oriented. Successful DevOps relies on the ability of software developers and IT Operations teams to collaborate closely in a successful and ongoing manner to ensure software is developed and delivered quickly and reliably.||Silo Driven. Traditional approaches rely on the “throw-over-the-wall” metaphor for collaboration, where IT Ops is charged with deploying and managing software in production with minimal assistance or insight from the team that developed it.|
|Structured and Substantially (or Wholly) Automated. DevOps practices rely on automation to provide speed, consistency, and repeatability in environment provisioning and configuration such that what works in a development environment is guaranteed to work in a production environment. Structured approaches also make failure recovery faster as repeatable automation makes rollback and recovery easier.||Snowflake Driven and Mostly Manual. Traditional approaches rely on an ad-hoc combination of scripting and manual processes (with servers as unique as “snowflakes”) to provision and configure infrastructure. This makes it hard to get right, repeat reliably, or do quickly. Such approaches tend to encounter many problems that are rooted in the inability to quickly and consistently provision development and production infrastructure with configuration parity.|
|Self-Service Oriented. DevOps-driven organizations establish frameworks for collaboration and automation to empower developers and IT Ops to act independently without getting in each other’s way (e.g., developers can provision dev/test environments quickly through automated means without waiting for manual provisioning by IT Ops).||“IT Ticket” Oriented. Traditional enterprise approaches require IT Ops to manage administrative overhead for IT tickets and perform repetitive, manual, complex provisioning and configuration that can easily be automated. This introduces significant complexity and delays in provisioning, deployment, scaling, and other software delivery and management activities.|
|Business Focused. DevOps organizations are focused on jointly making the business successful. Therefore, they focus on taking joint responsibility for software delivery success.||Function Focused. Traditional approaches require the development and IT operations teams to focus on performing their function, with little connection or responsibility for the overall success. As a result, problems and failures lead to lots of finger pointing and organizational friction.|
|Designed for Change. DevOps practices are designed to be automated, repeatable, and fast. They are built to handle rapid change as well as to recover quickly during a failure. They are built to go fast.||Change Averse. Traditional approaches try to avoid changing the production deployments, for fear of breaking it and not being able to recover quickly. They try to minimize changes and updates, and indirectly encourage the organization to go slow.|
At Pivotal, we help enterprises understand and establish a robust DevOps practice. We offer an integrated and automated experience built into our cloud-native application platform.
|Engage our Pivotal Labs team to understand and learn in a hands-on manner how to build and deliver high-quality software quickly in a DevOps context.|
|Deploy your applications to Pivotal Cloud Foundry and leverage built-in automation for key DevOps workflows (e.g. provisioning, deployment, scaling, high availability) that are consistent across development and production environments on any cloud infrastructure.|
|Learn how to use Concourse, the continuous delivery solution from Pivotal, to orchestrate and automate DevOps practices with Pivotal Cloud Foundry.|
|GE Digital is taking a DevOps approach to application development, and Cloud Foundry is enabling a smooth interface between ops and development. The company is consistently delivering production-ready, scalable applications in a fraction of the time.|
|Insurance company Allstate journeys into continuous integration, continuous deployment, and test-driven development using Pivotal Cloud Foundry.|
|Comcast empowers DevOps with Cloud Foundry.|