"The way to modernize our work is not to use a computer instead of a typewriter and call it innovative." ~~ Heidi Hayes Jacobs, Upgrading the Curriculum
Federal modernization systems have a very high percentage of failure - some report that up to 85% challenged or cancelled. Why? There are many reasons, and probably wide variations between programs. Still, many of the common causes appear to be:
1. Failure to understand the impacts the system has. Many new projects are using Agile methods, but still fail to apply complete knowledge of the systems development world and why the state of the practice has evolved. For example, we imitate elements of the Toyota management system (like kanban boards) but ignore others. When a manager starts a Toyota career, he is given a stopwatch and told to go observe what is happening on the floor before starting to "manage" stuff. Watch, shut up, and measure. But in IT, how often do we send our shiney, new, hot-shot programming rock stars out to observe end-users doing what they do? How often do we measure their productivity, observe how rules, laws, and practices constrain what they do? In the case of the healthcare.gov launch, the developers failed to anticipate the number of hits on the web site. In the case of immigration systems, there may have been a failure to baseline the performance of adjudicators and other people in the system, in order to determine whether the system was impacting their job while trying to move off a creaking infrastructure.
This problem is an old one, agile methods or otherwise. My friend Rusty's term "feature creatures" still applies -- folks so fixated on functionality and new shiny objects that they can't think about performance, user experience, interdependencies, technical debt. Agile methods haven't really fixed this, because some of the predominant ones encourage "navel gazing" within the development team, and abdicating responsibility for understanding client and end user needs to a proxie, such as the Product Owner. While that approach may help prevent interruptions to programmers, it aggravates a bigger problem.
2. Re-automating a bad process. The quote “don’t automate bad business processes” was a mantra in my junior years as a code writer. It was clear to us that our job was not to merely sling the slickest code around, our job was to solve a business or technology need. Somehow that gets lost during most modernization efforts, in which significant effort is spend replicating old functionality in new code/infrastructure, simply assuming the functionality already there is correct. It’s like slapping a new coat of paint on a rusty 1930’s tractor. Scrum being the predominant agile method used in the Federal space doesn’t help, since most product owners have not mastered the role and can be an information bottleneck as much as an information conduit. I have seen very impressive exceptions to this in the form of superb product owners, but they are not common.
3. Undertaking too much at once. The concept of a "strangler pattern" isn't new, but still has not nudged aside old "big bang" habits in system development. System modernization projects tend to discount the fact that what took 20 years to evolve into being in the first place can't be replaced wholesale in two years. Good architecture decisions -- ranging from old-fashioned solid procedural architecture up to the latest fad, microservices -- help systems to be replaced by modifiable components that are more loosely coupled. This loose coupling not only helps with testing and fielding the new system, it also makes it easier to eventually, inevitably replace it. “Retire-ability” is one non-functional requirement that has been rigorously considered in some systems (including at NASA), but is not considered at all in many Federal systems.
The challenge with Federal systems is that a wealth of Byzantine laws handed down by Congress, compounded by executive orders, complicate the quest for loosely coupled systems. And, as suggested by Kent Beck in a Java Magazine interview this month, the quest for ideal architecture can post a risk for system coherence. This matters more for systems embodying federal law than for, say, an e-commerce site that has more business-standard functionality that is easier to break into small, reusable services (many of which can be acquired via open source.)
4. Seeking silver bullets. The PMBOK, the classic SDLC, the CMMI, and now Agile, have failed to save these failing programs. Pieces and parts of such tool sets have helped deal with certain risks, but defaulting to them without understanding raises the risk of complacency and inflicting new methods by rote. Complexity is there, and, as Grady Booch stated in a speech several years ago, software development remains hard and will remain hard for some time (at least for non-trivial systems.) The key to any approach is realizing what it is good for, and the limits to its benefits. Agile approaches may have improved team performance tremendously on programs such as ELIS and the follow-on work for healthcare.gov, but could not solve everything. Nothing solves everything. An Agile method and DevOps will bring benefits, but broader understanding – from the sponsorship level on down -- is necessary to know what additional skills and approaches are needed to make a systems modernization program work within the specific context of the organization.
5. Conservatism. We all have specialties and comfort zones. Some of us have hard-earned skills learned within a particular framework. And like scientists working within a paradigm, it can be hard to accept anomalies that challenge our theory of how things work. If we have seen too many silver bullet initiatives come and go, we can brush off beneficial new approaches just out of change fatigue. But Moore’s Law is with us, technology continues to move at a rapid pace, and some of our old habits (that really were compensating for lack of machine availability or performance limits) can constrain our adaptability. The resulting gap between ultra-conservatives and radical silver-bullet vendors has caused more than one modernization program to founder by dragging the benefits of innovative methods down into the muck of traditional, monolithic “lift and shift” modernization efforts. Instead of seasoned, experienced sponsors and managers embracing innovation and adding great practices gained through personal experience (and not through mere compliance habits), we sometimes see teams performing radically new approaches being cobbled by old cover-your-backside conservatism.
6. Silos. The federal workplace, like many others, can suffer from rice-bowl protection, hero-worship, and power games. Large system modernization requires intense collaboration and communication across groups. Agile at scale attempts to address this, and closing such gaps is a core philosophy underlying DevOps, but it isn't easy to do. With large legacy systems it can be especially difficult to understand who is supposed to talk to whom, about what. Much of the institutional knowledge underlying those systems may have long since walked out of the organization, leaving one or two experts who know who to cajole and tweak the existing systems into running each day, but don't have nearly enough knowledge to replace them. Forming knowledge teams to weed through that legacy knowledge is hard. It also highlights the fact that it may take a long time to perform the modernization, something most sponsors don't want to hear. But sponsors and managers have no choice: identify who the knowledge holders are, understand how complex and convoluted the modernization may be, devise a strategy to implement real value incrementally, suck it up -- and stick together.
7. That one key holdout. One of my pet projects, of which I was very proud, never fully went live because two major sponsors could not agree on the concept of operations definition for about 15% of the end product, and would not approve releasing the other 85% before that last slice was completed. Before they could resolve their debate, a new President and Congress was elected, the funds got diverted to other priorities, and that was that. The end. Modernization programs do run into such "immovable objects" from time to time, and they simply go away without having solved the problems they were initiated to address. This is another reason for "strangler pattern" approaches, to work in new benefits and retire old systems through frequent, high-value releases. It's also another reasons to not take on too much; it's better to get a few things truly complete and shipped rather than risk all investment being considered a waste -- "better incremental improvement than delayed perfection."
In contrast, the first modernization effort I ever witnessed was highly successful. In 1987, my teammates began running tests the mini-computer hosted COBOL system our organization had developed. The tests showed that, as anticipated, a two-digit year field would result in serious problems on January 1, 2000. (Other date-related problems were identified later.) We requested additional hard storage to accommodate a switch over to a four-year date field; there just wasn’t room on our current data stores. The cost would have been about $50,000 for the hardware, plus development effort to revise the data structures and source code. The request was denied; it wasn’t worth investing that much money in hardware that was going to retire out. Instead, four people from the original team were tasked to re-write the system (which at times had over 100 people working on it.) That re-write was undertaken in the early 1990’s, and completed five years before the Year 2000 rollover.
The lessons were multiple. One, a dedicated team not overwhelmed by profit motivations could work quickly and effectively to accomplish a daunting modernization. Two, the team didn’t need to be that large if they were highly skilled and knew the current system well. Throughout my career, I’ve seen the pattern repeated where a dedicated few typically were doing the ‘heavy lifting.’ Challenges with many modernizations are not always about having enough people, but having the right people. Many programs are challenged from having too many people.
********************************************************************
The most recent in a long string of articles about modernization challenges is at http://www.cnsnews.com/news/article/susan-jones/dhs-oig-issues-urgent-recommendation-avoid-using-electronic-immigration.
Shawn Presson - Agile Coach, CSM, ICP, SPC6, PMP, certified CMMI Instructor, Organizational Change Champion, Program Manager, Appraiser