This guide seeks to provide an opinionated, recipe-style approach for how team leaders (which, depending on your organization, may be engineering managers, product managers, engineers, and technical leads) can define their products, teams, backlogs, and plan and execute projects leveraging agile fundamentals. The end goal? You should be able to deliver results predictably, adapt to the continually-changing landscape of software development, and iterate based on customer feedback so you can build the right thing for your customers.
This guide is written as a series of phases, allowing you to start with some foundational activities like defining products, backlogs, and teams, and then progress to building a sustainable project management and execution processes on top of that foundation. Regardless of your project management proficiency, it is recommended that you start at the beginning of this process, and work your way through it phase-by-phase.
You, your team, your customers, your stakeholders, and your leaders should have a development process which is:
- Customer Focused: You and your team should always know who your customers are, and work should be defined in a way that represents its customer value. It doesn't matter if your customers are internal, external, or even your own team.
- Transparent: Anybody - you, your team, your leadership, your customers - should be able to see your team's backlogs (aka roadmap), at any time. There don't need to be any misconceptions about "what you're actually doing" vs. "what you said you would do".
- Empirical: Your process should be data-driven, where your primary decision-making factor is what has happened in the past.
- Predictable: For near-term periods (e.g. 1-to-4 months ahead), you and your team should be able to predict when work will be completed, including compensating for risk factors and ambiguity. Over longer periods of time, you should still be able to predict general timelines, but these estimates should inherently compensate for the uncertainty and unreliability that comes with predicting the future. Over any time period, if things change, you should be able to promptly measure the impact of that change.
- Durable: Your product development should not get put on hold because of minor resourcing changes. Your team should be productive regardless of week-to-week fluctuations of on-call rotations, vacations, or illness. Estimates shouldn't vary wildly depending on who does the work. An individual team member winning the lottery shouldn't cause significant impact to your roadmap.
- Measurable: You and your team should be able to easily measure your team's performance, the cost of projects and operational activities, and understand the cost of disrupting existing plans. You should be able to understand the impact of adding or removing individuals from the team or a project.
- Clear Priorities: You and your team should always have a single list of priorities across all projects, so there's only one place to look to answer questions like "What's next?" and "What's important?" (or "What's more important?").
- Well-Defined: For work occurring in the near-term (e.g. a 2-to-4 month period), you and your team should have a clear definition of requirements, and you should always know what "done" means. For work further in the future, you should still have a reasonable definition of "done" (proportional to how far out the work might be), but it should be higher-level and more flexible to allow for change as you learn more about your customers or products.
- Minimal Bureaucracy: You and your team should have the minimum effective set of processes, meetings, and overhead necessary to produce maximum benefit. That doesn't mean zero overhead - it's crucial that you have mechanisms in place to encourage planning, collaboration, transparency, and stakeholder participation - but it's key that these are impactful and non-redundant.
- Flexible: You and your team should be able to adapt to change, and adjust plans accordingly, without throwing away significant amounts of work. You should be able to embrace change, rather than try to prevent it.
- Autonomous: Your team should be empowered to execute independently, without day-to-day oversight from a manager.
- Continuously Improving: You and your team should be able to experiment with your process, and measure the results of those experiments, in order to improve your it over time.
Empiricism and anecdotes
You might be asking yourself, "What data says this stuff will work?". Spoiler alert: the backbone of this guide is pure, vanilla Scrum, but with more deep dives into how to effectively leverage its fundamentals in an engineering organization. There are many great books and blogs on strengths and criticisms of scrum, so I won't cover them here (I've made reading recommendations below).
The basic value proposition of Scrum states:
Scrum is a framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value.
Sounds like a pretty good fit for most teams, right?
Scrum is founded on empirical process control theory, aka empiricism. Empiricism asserts that knowledge comes from experience and making decisions based on what is known. Scrum employs an iterative, incremental approach to optimize predictability and control risk. Three pillars uphold every implementation of empirical process control: transparency, inspection, and adaptation.
Anecdotally, I've worked on teams and projects using a variety of software development frameworks and methodologies, including waterfall, kanban, and various versions of pure and mutant scrum. They all have their advantages and disadvantages, but through experience, I've settled on plain old vanilla Scrum as one of the most effective processes for software development.
Common pitfalls when getting started
I've consulted for (and worked on) a surprising number of teams that described themselves as being scrum or agile teams, but in actuality, many of them weren't doing anything even close to scrum or agile. In many cases, they were running some form of "scrummerfall", which most often meant using phase-driven waterfall methodologies coupled with a daily stand-up.
If you're looking to go agile, my best advice is: start with vanilla Scrum. I've seen so many teams that want to be more agile with Scrum, but before they start, they make a bunch of changes to the framework, or disregard a bunch of fundamentals, and then blame the framework for not working properly. The Scrum framework is super flexible and can be adapted to fit your needs (which is a great feature!), but don't forget that it's an empirical process. You shouldn't be making changes unless you 1) measure the thing you're trying to change, 2) make the change and measure its impact, 3) adapt to your findings (and keep measuring!).
So, if you you want to simply stop reading right now and get started: Just read the official Scrum guide with your team, do what it says, and run vanilla Scrum to start. After running it for a few months, and building up some muscles you or your team may have never used before, you can start making limited, specific changes to the framework and measure the output.
But it isn't quite that easy, right? I mean, why does this guide even exist? The official Scrum Guide puts it well:
Scrum is simple to understand but difficult to master.
I'm going to lean on you to own the "simple to understand" part, and the focus of this guide is going to be more about the "difficult to master" part.
Prerequisites: Recommended Reading
The foundations of this guide are based on decades of project management experiences and recommendations taken from many books and individuals. While learning-by-doing is one of the most effective ways to learn a skill (especially if you subscribe to the 70/20/10 model, as I do), when it comes to project management, you may not have the flexibility to learn (and fail) on the job. Instead of doing everything the hard way: augment your learning by taking from others' experiences by reading highly-regarded books.
Regardless of your experience level, if you're looking to execute Scrum on your team, I consider the following resources to be required reading.
Essential Scrum: A Practical Guide to the Most Popular Agile Process by Kenneth Rubin
Why it's worth reading: I consider this to be one of the best books on scrum, covering everything you need to know end-to-end, and also includes modern best-practices developed by the community. Whenever I'm coaching teams on scrum, I recommend that they read this book first, regardless of their experience level.
The Official Scrum guide by Ken Schwaber and Jeff Sutherland
Why it's worth reading: This document contains the most important, distilled fundamentals of the Scrum framework. Created by the founders of Scrum, this brief guide will give you all the basics needed to execute Scrum, and will help you get started on the right foot. Plus, it's only 13 pages long (!), which speaks to how simple Scrum really is at its core.
Fifty Quick Ideas to Improve Your User Stories by Gojko Adzic and David Evans
Why it's worth reading: Writing good user stories is really hard. It seems easy at the surface, but once you get into large or complex products or engineering projects, it's easier write bad stories than good stories, and bad stories will harm your team's ability to get work done. Well-written stories are a significant advantage in agile software development, clearly describing customer value while breaking down the work into actionable pieces, so it's important for engineering managers, product managers, and even the development team to know how to write good user stories. It's also critical that the Product Owner knows how to be a good story editor to ensure your team's stories meet a high bar.
If you're looking for some additional great books to read, I recommend:
Agile Estimation and Planning by Mike Cohn
Why it's worth reading: While I don't agree with everything presented in this book, it does a great job of taking you through the longer-term aspects of running projects with Scrum, including how to take an agile approach while still planning out long-term milestones and roadmaps. If you run projects with long-term deliverables, I highly recommend reading this book, and I especially recommend it for engineering managers, product managers, and technical program managers.
Scrum Shortcuts Without Cutting Corners: Agile Tactics, Tools & Tips by Ilan Goldstein
Why it's worth reading: This book is a compilation of short, great tips on how to improve your scrum process. I'd recommend reading this once you've already established a good vanilla scrum process on your team, then you can start tweaking things to fit your team's needs. This is also a great book for individuals on your team who serve as Scrum Master.
Scrum: The Art of Doing Twice the Work in Half the Time by Jeff and J.J. Sutherland
Why it's worth reading: Written by one of the founders of scrum, this book has some interesting anecdotes about how scrum has been successfully used across many industries (not just software development), like manufacturing, medicine, journalism, and or charitable work. I often recommend this as a great book to read if you need to sell your leadership on scrum, without overwhelming them with technical process details.
That's the end of Part I: Introduction! If you found this helpful, check out Part II: Defining Your Products and Teams and subscribe to my newsletter (it's free!) to get notified as additional parts are published, which will be posted under the tag #agile-manifesto.