Why not Scrum?

Scrum is one of the most widely used flavors of Agile software development, a movement that started with the Agile Manifesto in 2001. Over time, Scrum and Agile have spread outside of software development as a general-purpose project management methodologies.

Although some teams love Scrum, others hate it. Why? I now believe this depends mostly on the nature of the team and their work, rather than on the personalities involved. Specifically, Scrum is designed to benefit tightly-coupled, product-focused teams with clear short-term goals and strong links to outside stakeholders. Additionally, Scrum is designed to meet the short-term operational needs of the team itself, and not the long-term strategic needs of management.

Team attributes dictate ceremonies

Scrum revolves around "ceremonies" (meetings) that recur in a regular "sprint" cycle, typically every two weeks. Scrum is only a good bargain if the increase in team productivity is greater than the time cost of these meetings. The meetings target the pain points of a certain kind of team: tightly-coupled, product-focused teams with clear short-term goals and strong links to outside stakeholders.

A daily "standup" meeting is useful for tightly-coupled teams, where team members' work is highly interdependent on an hour-to-hour basis. For those teams, standups provide important tactical coordination.

Backlog grooming is useful for product-focused teams, where there are many discrete tasks ("tickets") that can be clearly enumerated, and anyone on the team can tackle any one of them. "Product focused" can include products or services, and development or maintenance activities.

Sprint planning is useful for teams with clear short-term goals. Story points, estimation, and hard commitments are particularly useful when other teams care about the timing of deliverables over the next 1-2 months. Sprint commitments can also help a team defend their time from frequent distractions or shifts in priorities, or help an individual who works better in the face of deadlines.

Sprint review is useful for teams with strong links to outside stakeholders. Ideally, those stakeholders are highly engaged; want frequent in-depth status updates; and can help set the near-term priorities of the Scrum team.

However, not all teams exhibit these attributes. In fact, the scientific R&D teams I've spent my career with often exhibit the opposite attributes. Such teams are often loosely-coupled, with members working independently for days or weeks at a time. Some consist of specialists who are not interchangable, and cannot contribute much to each others' tasks. Many are discovery-oriented, with loose timelines and few short-term deadlines. Sometimes they cannot foresee the work beyond the next experiment. Finally, some teams have weak links to many stakeholders instead of strong links to a few, or the few are only engaged with the high-level output of the team, and not the details. For teams that work differently than the prototypical Scrum team, the ceremonies may be much less effective.

There is one Scrum ceremony that I feel is useful for most teams: retrospectives. However, many teams will find a quarterly retrospective sufficient. Only in times of rapid change are biweekly retrospectives useful.

Scrum serves the team, not the management

Scrum is intended to help individuals and teams with a short-term, tactical question: What work should be done next? It does not try to look beyond the next month or two, at least not in detail. These methods are called "agile" because they accept that long-term plans are fundamentally uncertain, so either scope (priorities) and/or deadlines will need to evolve.

However, upper management still has to make long-term plans in the face of this uncertainty, and Scrum sings a siren song. Story pointing and velocity calculations seem to promise accurate timelines for future deliverables. And Scrum tools (JIRA, Aha!, Azure DevOps, etc) seem to promise that all work will be cataloged and tracked, with easy overviews and status reports. However, both promises are false, because Agile methods were not designed to serve the needs of upper management.

First, Scrum estimates do not accurately extend to periods of quarters or years. Teams only expect to make reasonable estimates for the coming sprint because the work is imminent; they have a close-up view of it. Far-future work is intrinsically unclear and fuzzy. Scrum isn't worse at these long-term estimates than other methods, but it's not better either.

Second, Scrum tools don't address the needs of management. Tickets written for the team are at the wrong level of abstraction for upper management, with too much operational detail and not enough explicit strategic context. Furthermore, the tools do not surface insight into the key risks or bottlenecks that leaders should devote attention to: Everything is given equal prominence. Last but not least, only highly diligent teams manage to keep their ticketing tool fully up-to-date!

Teams may choose to adopt Scrum because it fits their way of working and offers benefits they need. Even when Scrum doesn't fit the current team dynamic, a team lead might (for example) try introducing standups to encourage more collaboration, or sprint reviews to encourage a tighter connection with stakeholders. However, upper management should understand there is little benefit to themselves from imposing Scrum on a large organization from the top down. The potential tactical benefits of Scrum at the team level do not "scale up" to help management with strategic planning in the large.

My context

I have observed Scrum in data science and data engineering teams in multinational life-sciences R&D companies, where I've seen the same scenarios play out multiple times in different organizations. I've had Scrum training from a variety of Agile coaches -- from AWS, from consulting companies, and internal. I've read extensively about Agile, and spent four years using and modifying Scrum approaches to lead teams. I hope these notes help leaders recognize which Scrum practices are (un)likely to be useful to their teams, so that if they choose to implement part or all of Scrum, they do it in clearly thought-out way.

Many thanks to MWD, DFB, and RGC for thoughtful comments on drafts of this post.