The Democracy of Agile

-- Adapted from my intro at the Keep Austin Agile 2016 Conference.

Fourteen years ago I was promoted to Senior Manager, two short years after joining the company as a Software Engineer. I had managed a small team before but this was a bigger team, a much larger company, with more responsibilities. Over the next 8 years, I would invest countless hours learning how to be a better manager and leader.

Every management book I read, training class I attended, and webinar I listened to had the same goal: to teach me what I needed to do to solve problems for my team. While my developers worked on technical issues, I was the one interacting with Product Managers, Business Analysts, Project Managers, and Executives. When we needed to make a process improvement, evaluate requirements, or come up with a project plan, I was the one responsible for figuring it out. Everything flowed through me and was initiated by me. It was the traditional – and autocratic – way to manage a team. I was in charge!

But something was seriously amiss. This way of operating and thinking assumed that the manager was smarter than the collective wisdom of the team, that the manager had the specialized and general knowledge to address almost every situation, and that the manager alone could devise the best plan for the team every time.

When people talk about Agile, they are often talking about the Industry of Agile – all the practices, ceremonies, rituals and artifacts that make up our collective understanding of agility. But what most people neglect to sufficiently appreciate is an aspect much more important: the Democracy of Agile. Buried in all those rituals, practices and ceremonies is a key tenet: everyone on the team is now in charge. Process improvements? They often come out of team retrospectives. Requirement clarifications and adjustments? This happens in backlog grooming sessions. Even dates and milestones are now projected by the team – on their own.

The empowerment of teams practicing Agile is neither accidental nor discretionary. To quote the Agile Manifesto: “The best architectures, requirements, and designs emerge from self-organizing teams”, and “the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” In its purest form, Agile is a true democracy, where everyone on the team has an equal vote and everyone is involved in making decisions for the team. The manager is no longer the focal point of the team nor the only one initiating and driving change; the manager is now part of the team, a servant-leader who ensures the team members have the environment and support they need, and then trusts them to get the job done.

There is another aspect to Agile Democracy that’s equally important to understand, and that’s the diversity of those practicing Agile. The Agile team is a mix of skills, roles, and experience levels, everyone embracing the Agile values of responding to change, face-to-face communication, technical excellence, empowerment, and intrinsic motivation. The diversity of the Agile team is core to the Agile experience and the success of the Agile team, because it brings all those skills, experiences and opinions into one place with everyone sharing the common goal of delivering value to the customer. Just like diversity makes the communities where we live stronger and more resilient, it also makes every team in our companies stronger and more resilient.

It is the responsibility of every Agile team member to embrace and be the champion of democracy and diversity in their Agile teams. In Agile Democracy, you are empowered – even obligated – to initiate and drive change.

In a future post, I will talk in more detail about the role of the manager in the Agile team. After all, if the team is making decisions and charting improvements on their own, what do we need managers for? How does the team know what decisions they can make? And how does a manager ensure empowerment doesn't run amok? 

The Sterile Cockpit

On September 11, 1974, while flying an instrument approach in dense fog into Charlotte, North Carolina, an Eastern Airlines DC-9 landed just short of the runway and crashed, killing 72 of the 78 on board. After an exhaustive investigation, the NTSB determined that pilot error was the primary cause of this accident, citing “poor cockpit discipline” in their report, specifically, non-essential conversation between the flight crew members during the approach part of the flight. Due to this accident and others like it, the FAA in 1981 instituted a rule called the Sterile Cockpit Rule, which prohibits non-essential activities by the pilots during critical phases of flight, including taxiing, takeoff, approach, and landing – more precisely any activities occurring below 10,000 feet.

So what’s the point of this? In my experience as a software engineer and a manager, we are most productive when we are focused for continuous blocks of time and have few if any interruptions and distractions. Inversely, our productivity takes a non-linear hit with contextual task switching, especially when the tasks are not homogenous. In his book Quality Software Management, Vol. 1, Systems Thinking (Dorset House, 1992), Gerald Weinberg estimates that context switching between projects costs 20% of time for each additional project undertaken. So if a person is switching between 3 projects in a 45-hour week, 18 hours are spent context switching, with 9 hours spent on each project!

Rock climbers often speak of “flow, a state they enter when they’re climbing, in which their entire body is involved in the ascent and the climb up the rocks feels effortless. A recent incredible 60 Minutes report on super star rock climber Alex Honnold demonstrates this state. As Alex approaches the toughest and most dangerous part of his climb, he is totally relaxed, focused and even smiling (advance the video to the 10:45 mark). A book that came out in the late 80’s, Peopleware (Tom DeMarco and Timothy Lister), based on research started in the early 70’s speaks of this magical “flow” state. When software developers enter this state, time stands still and the computer becomes an extension of their mind and body. This is the state when they crank out their most valuable and productive work. According to the book, it takes at least 15 minutes to enter this state, so every interruption costs 15 minutes of productivity. That means four interruptions in a day will cost an hour of productivity.

So where does this takes us? Should we stop answering email, responding to questions from teammates or peers, or even taking a smoke break? That is hardly practical, and ill-serves our customers and stakeholders. However, what this data tells us is that we have to introduce and enforce uninterrupted blocks of time when software engineers can get into the “flow” and remain there for a few hours. Most of us currently experience these magic blocks of time either early in the day before 10:00 am (if you’re an early bird) or late in the day after 5:00 pm (if you’re a night owl). However, during the middle of the day between 1:00 and 4:00, everyone gets interrupted with a meeting at least once a week.

With that in mind, I have frequently implemented for my teams a meeting blackout period – an Engineering Sterile Cockpit – between the hours of 1:00 and 4:00 every day. During those hours, no one would be allowed to schedule a meeting for members of the team unless it is of a critical nature. That includes meetings for design, user story creation, review or sizing, process discussion, demo, planning, etc. Those meetings should be scheduled before noon or after 4:00 pm. While this may create a slight inconvenience for some team members or stakeholders, the benefit will immeasurably outweigh the cost.

Is this a silver bullet that will take care of all interruptions and context switches? Hardly. But it is a concrete action your team will appreciate immensely and one that will pay dividends for the business. A common theme I observed over the years in many team retrospectives is the perception that there are more and more meetings. But whenever I looked at my employees’ calendars, I discovered that for most of them that is not the case. I am now reasonably confident that this perception is borne out of their inability to get into a “flow” state – precisely at the time when they are ready to enter it.

My ELITE Stumble

“Cool! It’s just what we asked for, but not what we want now.”

– Anonymous

Several years back, I was involved in a project to create a custom version of a mainstream stock trading application for a small set of hyperactive traders who executed hundreds – sometimes thousands – of trades every day, and generated quite a bit of commission revenue. We dubbed this version “ELITE”, and put a couple of bright developers to work on it. This was our SWAT team.

The idea went like this: by implementing the advanced features hyperactive traders wanted in a custom version of the application, we can release these features quickly and with little concern for adoption by mainstream traders. What was even more enticing about this approach is that once one of those features was thoroughly vetted by those fastidious traders and had shown to be applicable to the general population, all we had to do was comment out a few #ifdef’s and voila! Every customer had access to this feature.

By any account, the ELITE version was a runaway success! We were responding to those customers quickly and adapting the software to their unique needs. They were trading like crazy and racking up commissions.

In terms of execution, we had little or no documentation to rely on – sometimes just a sketch and some handwritten notes. We sat down together, the business analyst and me, and tinkered with the wireframe until we were happy with it. Then the developers built it and showed to him, and then we would adjust it and repeat the cycle a few times. When we were satisfied we had it right, we made a build, sent it to QA, and a week or two later, the ELITE traders had their precious feature.

What was remarkable about this is the fact that at that time we were still a couple of years away from implementing an Agile approach in any of our projects. We literally stumbled upon a working Agile model, and executed it inside of a Waterfall system. In fact, while these ELITE features were being added to the code base, the mainstream projects were trudging along, business as usual. For every mainstream release, we would release 3-4 ELITE releases. This may not seem like a lot, but for that product, in that timeframe, in that environment, it was indeed a miracle.

Years later, as I reflected on that period, I realized how forward thinking we were, some would say lucky, in adopting a model that was Agile at its core if not in the skin on the outside. True, we didn’t have daily standups or retrospectives, nor a burndown chart or a story board. What we did have is the essence of what Agile is all about: we were sitting down to discuss and design the features, building a working model for the customer to validate, and making incremental changes based on the feedback.

These days, with Agile practices as commonplace as happy hours are, I worry that we might be losing the essence of that Agile experience, namely quickly responding to change and constantly iterating based on customer feedback. There is incessant noise about Agile artifacts and ceremonies cluttering our conversations. Unless we keep the essence of the Agile experience front and center as our guide, and measure every decision we make and every direction we take with that ruler, we risk creating an environment as rigid and unforgiving as Waterfall was. We especially have to be cognizant of traditional boundaries between technology and business getting slowly reintroduced. So stand up, leave your desk, and walk over to the other side of the building. You’d be surprised how a long email chain can be cut short with a 10 minute conversation.

Agile-Waterfall, is that an oximoron?

I was reading a blog post the other day with the provocative title “Agile is a Fad”, and it occurred to me that as Agile practices become widespread and almost ubiquitous, and as some company executives feel the pressure to hastily adopt Agile at any cost, a predictable backlash will ensue, especially when these organizations struggle with Agile introduction and adoption. This backlash will not only come from the leaders of these organizations, but also from Agile purists who foresee the Agile movement being corrupted by companies wanting a quick cure, and a cadre of consultants willing to sell them the magic pill.

So it comes as no surprise that some organizations are getting to a point in their adoption of Agile where they are questioning its usefulness and value. Some are even reverting to Waterfall after "failed" experiments with Agile.

Why is this happening? Agile was supposed to be the answer to all their past woes with timely delivery, customer acceptance, and quality. Yet, Agile becomes a battle of wills between the folks who championed it and those who simply wanted things done. At some point, failure to deliver a product on time leads the non-champions to the conclusion Agile methodology is to blame.

In reality, the problem has deeper roots. It originates with the reason these companies adopted Agile in the first place. Often, the product side of the house wants Agile so that:

a) they don't have to write extensive requirements upfront,
b) they can change their mind up to the point a product ships, or
c) they want to know what's going on inside of the software development black box.

On the other side, the technology side of the house often pushes to adopt Agile so that:

a) they can start designing and coding right away,

b) they don't have to sit through laborious requirements review meetings, or
c) they can show off their handy work early and often.

The fact is, Agile facilitates and supports many of these desires by both sides. Yet, a lack of understanding of the fundamental goal of Agile dooms the experiment to failure in many companies before or soon after it starts. Everyone becomes so focused on the ingredients and steps of the recipe they forget what they were cooking in the first place!

So what is the fundamental goal of Agile? The Agile Manifesto talks about 4 key principles that are highly valued:

-          Individuals and interactions over processes and tools

-          Working software over comprehensive documentation

-          Customer collaboration over contract negotiation

-          Responding to change over following a plan.

Yet, what do these principles help with ultimately? What is the key benefit of using Agile in software development? The answer is:

Facilitating Change that Adds Value at the Lowest Possible Cost.

Think about it... Your work as a software professional starts out with a feature request, enhancement, or bug, i.e. some kind of a change to the product. Everything that follows are steps to facilitate that change, whether you follow a Waterfall or an Agile approach. Yet, where Waterfall comes up short is on the questions of Value and Cost. Every feature that the customer doesn't use is a zero value change. Every delay in delivering to the customer is a higher cost change. A 100% successful Agile implementation is one where every feature added meets a customer need in a timely fashion and is built in the most efficient and cost effective manner.

A fundamental understanding by executive management, key stakeholders, and team members of what Agile is all about is the key to success in implementing Agile in any organization. The artifacts of Scrum, XP or Kanban are there to facilitate change, add value and reduce cost. If your organizations wants to use an incremental Scrum process (some call it Agile Waterfall) that works more seamlessly with organizational boundaries, go for it. If your organization wants to run one team with Scrum and another with Kanban because of the unique qualities of the products and customers serviced by these teams, why not. As long as you can demonstrate "with actual hard metrics" that you are facilitating requested changes that ALWAYS add value at the lowest possible cost, your version of Agile will more than likely be a success.