Make changes small and quick
American spiritual leader John Maxwell posited a development concept. He said you should ‘fail early, fail often, but always fail forward’. I would add ‘fail cheaply’. Silicon Valley companies embraced this with gusto. Software or infrastructure systems, doing smaller increments is great cost containment. It can also avoid the lessons born of catastrophe.
The Agile concept of a Minimum Viable Product embraces getting changes in place. Those changes at least do something. In software development, Scrum amplifies the intended quick movement to something visible and viable.
‘What small changes can I make that may accumulate to a spectacular outcome?’ Asking yourself this question is tough. Not a lot of visible glory in it. So how do we get from the today state to this destination? How can we claim what the organization needs quicker, more precisely and at the lowest cost?
- Small Changes Lead to Big Results
- Change Brings Risk
- Authorization, Implementation and Verification for Changes
- Agile: Limiting Changes Size and Number
Small Changes Lead to Big Results
The cumulative impact of small changes apply to more than making an impact with technology. Winston Churchill captured this in his quip ‘To improve is to change; to be perfect is to change often’. Sean Covey, Stephen Covey’s son, observed ‘Small changes can make huge destination differences’. These quotes succinctly point to an underlying truth. Anyone who has built skills for a sport or learned to play an instrument has experienced this. Why wouldn’t it work with organizations?
Turns out that it DOES work with organizations. Jim Collins, in his seminal work ‘Good to Great‘ summarized his study of exceptional businesses. ‘Exceptional’ performance for companies studied included cumulative returns of at least three times the market over a fifteen year period. In his chapter titled ‘The Flywheel and the Doom Loop’, he says:
“Now suppose someone came along and asked, “What was the one big push that caused this thing to go so fast?’ You wouldn’t be able to answer: it’s just a nonsensical question. Was it the first push? The second? The fifth? the hundredth? No! It was all of them added together in an overall accumulation of effort applied in a consistent direction.”
In Collin’s earlier work, ‘Built to Last‘, he related in the chapter ‘Try a Lot of Stuff and Keep What Works’, that the studied companies ‘…best moves [were] not by detailed strategic planning, but rather by experimentation, trial and error, opportunism, and -quite literally-accident.”
In both personal and business life, the principle of small changes holds sway for bringing about substantial and sustainable pay-offs. It’s a safe bet that it applies to applying technology in an organization as well.
Change Brings Risk
With change comes risk. ITIL recognizes this in three practices:
- Change enablement – ensuring that properly assessed, authorized to proceed, and scheduled
- Release management – making new and changed services available for use
- Deployment management – moving new and changed hardware, software, documentation, processes or any other service component, to a live environment
Three practices devoted to managing change. That’s a lot. It speaks to the potential impact of change gone bad.
Kevin Behr, Gene Kim and George Spafford captured the relationship for DevOps practices in ‘The Visible Ops Handbook’. The relationship is best captured in a quote from the mathematician and IT pioneer Nathanial Borenstein:
“The most likely way the world will be destroyed, most experts agree, is by accident. That’s where we come in; we’re computer professionals. We cause accidents.”
Will an organization’s IT changes gone bad end the world? Probably not. Will they have an impact on the survival of an organization? Most certainly.
Managing Change
Yet changes must happen, if only to survive, let alone to prosper. One strategy is to do the change management ITIL practices to perfection, probably in the extreme. The implementation process occurs post-authorization and moves from:
- Development
- Testing
- Beta
- Production
Each stage brings a change closer to production and reduces the risk of surprises and failures. By making the preparation really thorough, the odds of bad things happening will be really low, right? Well, not exactly.
The ‘Practice of System and Network Administration’ summary of ‘Continuous Integration and Deployment’ lists some of the non-optimal and possibly toxic results of strict change approval and release practices.
- It is labor intensive.
- It keeps each team in a silo, optimizing for their own needs rather than the greater good.
- It encourages gaming the system.
- It encourages large batches.
The summary example is:
“As an optimization it is tempting to send a new release to QA less frequently, but this creates a bigger testing burden. If a bug is found, the developers need to isolate it to one of the changes, which creates an even larger list [if changes to test]”
Gagging on Change Aggregation
With each change as a point for breaking, creating an aggregation of changes seems to be a way out. The concept behind creating a release which aggregates many changes is that doing the release once would reduce the likelihood of an outage. Increasing the magnitude of what is implemented does not reduce the number of changes. You have the same number of changes. By creating a release you have made testing and troubleshooting the changes harder. You now have more dependency relying on other changed objects. The chance of compounding errors increases with more dependent changes.
Authorization, Implementation and Verification for Changes
The ITIL practices of Installing change tracking and applying change review reinforces effective implementations. Change tracking records changes and tracks them through authorization, implementation and verification. Change review is the authorization step.
While all changes follow the process, some steps have a different process.
Routine updates ae pre-authorized. These occur when there is no expected outage and a near-zero chance of unintended disruptions. Emergency changes involve significant outages, such as a cyber attack or when the revenue loss mounts quickly. Emergency changes still require at review and approval. The initial review and approval may bandage the current situation. An after action review determines whether to keep the approved change or replace it with a better and more lasting solution.
The tricky part of emergency changes arises in organizations that defy change control. In those organizations, the appearance of change control results in all changes being classified as emergency items.
DevOps: Continuous Integration and Deployment
The DevOps solution for all other changes is Continuous Integration and Deployment. Having applied the ITIL practices reduces the number of changes requiring review. For the remaining changes, the biggest difference from pre-DevOps implementations is reducing the magnitude of changes by making them smaller and more frequent.
Smaller changes makes finding errors easier. Less to review, less sources of errors. Smaller changes can reduce the breadth of what is affected by changes.
Changes, be they hardware or software, flow through a their respective version of development, testing, beta and production steps. Automating as much of each step provides the technology acceleration over in-depth change process control coupled with manual implementations. Even when some steps remain predominantly manual, automating others speeds the movement to the manual actions.
The software development realm lends itself well to change control, continuous integrations and deployment. The body of work in that domain is vast. For organizations with a development effort, even if it focuses on customization of purchased software, can put these changes to work.
Hardware and networking can move to Infrastructure as Code. This opens many of the software tools to claim the same continuous integration and deployment benefits. Powershell, particularly in the form of its testing extension Pester, can be put to use in automating the deployment steps.
Time Required for Change Management
Adding procedures and automation of implementation steps would seem to take a lot longer rather than enabling quicker and more frequent changes? It sounds like a lot of extra steps and work that takes more time. This objection is usually said by team members in organizations resisting introduction of the practices.
It does take more time to do the change steps when compared to the time required for the same changes done manually and on demand. That is true initially. As deployment step automation increases, the time required tips to favor the new direction. More importantly, disruption to the organization diminishes from troubled change implementations. impact on the organization diminishes. This increases productivity in areas outside of IT. As the change process frequency increases, having the changes in production sooner increases end user productivity. Staying with a manual and on-demand change process can only incidentally claim the benefits. Steady and continuous improvement of the implementation process results in exceptional IT service delivery.
Agile: Limiting Changes Size and Number
One means to keep the changes small and in limited number is a frequent implementation cadence. The Agile Sprint concept has a set length effort ended with a review. The review is the gateway to deployment. The change, or set of changes, focus on those requiring effort measured in weeks rather than days. By limiting the effort to a fixed elapsed time, effort is devoted to changes that can pass into deployment. This culls out dispersed efforts over a greater number of changes to bring about small changes that can be completed.
Passing the deployment gate does means beginning the deployment process. The move is usually from development to pre-deployment testing. As described in the ITIL release management practice, it is the the practice of making new and changed services and features available for use. A complete release flow is from development to testing to beta to deployment.
The changes must produce a minimum viable product (MVP) to begin the release process. Eric Ries defined the MVP as ‘…that version of a new product which allows a team to collect the maximum amount of validated learning about customer with the least effort.” When creating a new service, that may mean that the change achieves at least one purpose for which it was created. This may be a very small purpose. It may, for example, demonstrate the solution framework even if it doesn’t do anything substantial. The key is that it clearly points to something that the end user can use to get closer to what they want.
The Agile practices of Sprints and MVP limits the changes size and number from a combination of:
- regular development cadence of Sprints,
- movement to reaching the MVP within a single sprint, or as few sprints as possible
- enhancing the successful MVP with as many new capabilities within a sprint
