Strategic Theme # 3/5: Continuous Delivery/DevOps

[tl;dr This post discusses why state-of-the-art technologies and techniques which brings together agile software delivery with operational considerations are strategically key to business success.]

Continuous Delivery is a subject that makes many operations folks break out in a cold sweat..the concept of being able to deploy change as soon as it is ready rather than on fixed deployment schedules is surely a recipe for disaster.

In most cases, it is: the act of taking a traditional monolithic application build, and moving it from test to UAT to production is often a very painful, labour intensive process, with many gates and controls. Because applications typically aren’t build with an operational perspective in mind, most of the creative coding goes into the business functionality, and not into the so-called ‘non-functional’ requirements such as ‘operability’.

The net result is that it’s a lot of work to deploy change, and hence it has to be planned well in advance.

The solution is automation: automation of building, testing, configuration and deployment.

Such automation requires significant investment, and so is generally only worth the effort when a significant number of teams can benefit from the automation. This in turn depends on standards: there is little point in building a highly automated toolchain for Java when most folks want to do C++, for example.

But the investment pays off, in the ability to respond rapidly to business needs, to quickly make change and be confident that nothing critical breaks, and to provide operations folks with key tools to manage and configure running applications without involving the core development teams.

Continuous delivery may only be an aspiration: it may not be necessary to deploy several times a day, even if you can. But the act of preparing for it makes the whole process much more efficient and massively reduces change-related risk.

The seminal book on this topic, ‘Continuous Delivery: Reliable Software Releases Through Build, Test and Deployment Automation’ by Jez Humble and David Farley covers all the topics very well, and dovetail nicely with related strategic themes of Enterprise Modularity and Lean Enterprise.

The question of Standards always raises its head: in my mind, Standards are in basic conflict with Innovation..although sensible standards can provide the constraints that allow genuine innovation to thrive (as, in general, innovation is a supply-side response to one or more constraints in meeting demand).

When it comes to solution standards (i.e., specific languages, libraries, tools, infrastructure, etc), one of the key drivers for identifying standards should be standards that allow automated toolchains to be built, according to Continuous Delivery principles. This generally means specific languages, specific code repositories, specific package dependency management, specific build repositories and tools, specific test frameworks, configuration engines, deployment tools, etc, etc.

In short, to achieve continuous delivery, firm technical standards need to be in place for the most frequently used architectures. For projects deviating from the standards, it must be understood that additional investment will be required to enable continuous delivery.

As each domain will have its own unique non-functional requirements, the level of toolchain automation appropriate needs to be agreed as part of the domain’s target architecture – and with it, the relevant technical standards the domain wishes to adopt. Short-cuts to deliver change which do not use the automated toolchain process needs to go through an exception process: technical debt in toolchain automation is usually not a sustainable option.

The Learning CTO’s Strategic themes for 2015

Like most folks, I cannot predict the future. I can only relate the themes that most interest me. On that basis, here is what is occupying my mind as we go into 2015.

The Lean Enterprise The cultural and process transformations necessary to innovate and maintain agility in large enterprises – in technology, financial management, and risk, governance and compliance. Includes using real-option theory to manage risk.
Enterprise Modularity The state-of-the-art technologies and techniques which enable agile, cost effective enterprise-scale integration and reuse of capabilities and services. Or SOA Mark II.
Continuous Delivery The state-of-the-art technologies and techniques which brings together agile software delivery with operational considerations. Or DevOps Mark I.
Systems Theory & Systems Thinking The ability to look at the whole as well as the parts of any dynamic system, and understand the consequences/impacts of decisions to the whole or those parts.
Machine Learning Using business intelligence and advanced data semantics to dynamically improve automated or automatable processes.

[Blockchain technologies are another key strategic theme which are covered in a separate blog,]

Why these particular topics?

Specifically, over the next few years, large organisations need to be able to

  1. Out-innovate smaller firms in their field of expertise, and
  2. Embrace and leverage external innovative solutions and services that serve to enhance a firm’s core product offerings.

And firms need to be able to do this while keeping a lid on overall costs, which means managing complexity (or ‘cleaning up’ as you go along, also known as keeping technical debt under control).

It is also interesting to note that for many startups, these themes are generally not an explicit concern, as they tend to be ‘obvious’ and therefore do not need additional management action to address them. However, small firms may fail to scale successfully if they do not explicitly recognise where they are doing these already, and so ensure they maintain focus on these capabilities as they grow.

Also, for many startups, their success may be predicated on larger organisations getting on top of these themes, as otherwise the startups may struggle to get large firms to adopt their innovations on a sustainable basis.

The next few blog posts will explain these themes in a bit more detail, with relevant references.

DevOps & the role of Architecture in Achieving Business Success

[TL;DR] Creating a positive re-inforcing effect between enterprise architecture and DevOps will provide firms with a future digital competitive advantage.

I finished reading an interesting book recently:

The Phoenix Project: A Novel About IT, DevOps and Helping Your Business Win (Gene Kim, Kevin Behr, & George Spafford)

Apart from achieving the improbable outcome of turning a story about day-to-day IT into a page-turner thriller (relatively speaking..), it identified that many challenges in IT are of its own making – irrespective of the underlying technology or architecture. Successfully applying lean techniques, such as Kanban, can often dramatically improve IT throughput and productivity, and should be the first course of action to address a perceived lack of resources/capacity in the IT organisation, ahead of looking at tools or architecture.

This approach is likely to be particularly effective in organisations in which IT teams seem to be constantly reacting to unplanned change-related events.

As Kanban techniques are applied, constraints are better understood, and the benefit of specific enhancements to tools and architecture should become more evident to everyone. This forms part of the feedback loop into the software design/development process.

This whole process can be supported by improving the process by which demand into IT (and specific teams within IT) is identified and prioritised: the feedback loop from DevOps continuous improvement activities form part of the product/architecture roadmap which is shared with the business stakeholders.

The application of these techniques dovetails very nicely with advancements in how technology can be commissioned today: specifically, infrastructure-as-a-service (virtualised servers); elastic anti-fragile cloud environments, the development of specialised DevOps tools such as Puppet, Ansible, Chef, etc; the rise of lightweight VMs/containers such as Docker and Mesos, microservices as a unit of deployment, automated testing tools, open modular architecture standards such as OSGi, orchestration technologies such as Resource Oriented Computing, sophisticated monitoring tools using big data and machine learning techniques such as Splunk, etc, etc.

This means there are many paths to improving DevOps productivity. This presents its own technical challenges, but is mostly down to finding the right talent and deploying them productively.

An equal challenge relates to enteprise architecture and scaled agile planning and delivery techniques, which serves to ensure that the right demand gets to the right teams, and that the scope of each team’s work does not grow beyond an architecturally sustainable boundary – as well as ensuring that teams are conscious of and directly support domains that touch on their boundary.

This implies more sophisticated planning, and a conscious effort to continually be prepared to refactor teams and architectures in order to maintain an architecturally agile environment over time.

The identification and definition of appropriate boundaries must start at the business level (for delivering IT value), then to the programme/project level (which are the agents of strategic change), and finally to the enterprise level (for managing complexity). There is an art to identifying the right boundaries (appropriate for a given context), but the Roger Sessions approach to managing complexity (as described in this book) describes a sensible approach.

The establishment of boundaries (domains) allows information architecture standards and processes to be readily mapped  to systems and processes, and to have these formalised through flows exposed by those domains. This in the end makes data standards compliance much easier, as the (exposed) implementation architecture reflects the desired information architecture.

How does this help the DevOps agenda? In general, Agile and DevOps methods work best when the business context is understood, and the scope of the team(s) work is understood in the context of the whole business, other projects/programs, and for the organisation as a whole. This allows the team to innovate freely within that context, and to know where to go if there is any doubt.

DevOps is principally about people collaborating, and (enterprise) architecture is in the end the same. Success in DevOps can only scale to provide firm-wide benefits with a practical approach to enterprise architecture that is sensitive to people’s needs, and which acts as a positive reinforcement of DevOps activities.

In the end, complexity management through enterprise architecture and change agility through dev-ops go hand-in-hand. However, most folks haven’t figured out exactly how this works in practice, just yet…for now, these disciplines remain unconnected islands of activity, but firms that connect them successfully stand to benefit enormously.