Three Years of Scrum at Socialtext
Adopting Scrum is simple enough, but what are the long term implications? Matt Heusser took some time to pause, reflect, and discuss the realities of living on a Scrum team every day for three years.
It seems that you can't swing a dead cat without hitting someone who is trying to sell you advice on "Transitioning to Agile with Scrum." That's important work, and I'm pleased someone is doing it, but that's not this article.
Instead, I tell the story of one very specific Scrum transition — at my employer, Socialtext. We switched to an Agile process in the fall of 2007, right before I was hired, and the development team has been evolving it for over three years now.
More than an article about Agile transition, this is about actually doing Scrum, with a focus on the daily standup meeting and how it changed over the years.
Socialtext's main office is in the heart of Silicon Valley. That location functions primarily as the base of operations for Sales and Marketing. While the heads of product management, engineering, and a few developers also work from there, the majority of the technical staff works from satellite and home offices across the country. With social software, we can work from anywhere with Internet access and power. This flexibility to work — and flexibility in hiring — is a huge benefit to the team. It means we can always hire the right person for the job, no matter where they're located.
If you have objections to Scrum because of location, or geography, or time zone, don't worry. We had plenty of concerns, too.
Instead of limiting expensive and time-consuming communication, we tried a different tactic: Make communication cheap.
Enabling Communication and Collaboration
To make communication cheap, our team adopted an open-source tool called astertisk that serves as a voice-over-IP server including conference rooms. Most team members agreed to at least work core hours of 10am-2pm Pacific time; we began with two-week fixed-length "iterations" and a standup ("daily scrum") meeting three times a week at 1pm.
The classic status update for a daily Scrum meeting is three pieces of information: What I worked on yesterday, what I'm doing today, and my blocker issues. Blockers are productivity killers. Because each team member focuses on one feature at time, becoming blocked turns a developer into an extremely expensive paperweight.
But being blocked for a day is still troublesome. Waiting a day for a meeting to tell people — that's no good. Instead, we found different tools, like micro-blogging or internet chat, to make requests.
At this point, the team had a decision to make: If we allow blocked team members to interrupt one another, it would be a direct interruption onto their screens, possibly a beep, breaking the other person out of "the zone."
We decided to allow the interruptions, because being blocked is so wasteful.
We also began using screen-sharing tools, like GNU Screen for text and VNC for GUI debugging, to enable pair programming. As of early 2011, roughly 80% of our programming is done in pairs.
Improving Daily Standup Meetings
Permitting interruptions meant that developers could work out blocking issues immediately, outside of the meetings. This meant when it came time to list our blockers at standup time, well... we didn’t have any. Removing obstacles early and often has a second side-effect, allowing us actually complete our daily commitments. Thus "what I did yesterday" and "What I said I would do yesterday at the last standup" were generally the same, making the second status item redundant.
The result: Three items to discuss a day, reduced to one.
Suddenly, our meetings started to get fast, usually 8-12 minutes. Sometimes less.
To make those meetings go even faster, we recorded the items in advance on a web page. At standup time we could share a screen with the page, and, when the work was fairly routine, people started to say, "Just what you see there," or "Nothing new, pass."
A Scrum Expert might come along and say that something was wrong; that the team wasn't communicating. I hold that the opposite is true: The team is communicating well, just outside the standup. Each day we pull work from the status board, actually a generated webpage showing what pieces of work were where in the process, something like this:
Screen shot of Socialtext’s Status or Kanban board. WIP is Work In Progress. Other elements of the page include In prep, In QA, and Awaiting Signoff.
Getting status from the Kanban and removing obstacles in the moment made the standup seem less relevant, as if it were a holdover from a different era with different challenges.
For us, it was. When Schwaber and Sutherland defined Scrum at EasleCorp, they had a very specific problem: Requirements churn was so high that the technical staff could not get anything done. Literally, they could not ship a product. So you can think of the "sprint" as breathing room from changes, or "Just give us a month to actually make something, then we can adjust." Socialtext didn't have that problem; or, at least after a year or so, we no longer had that problem. We needed to adjust.
At Socialtext, we started with two-week, time restricted “sprints,” and we held to those for two years. In that time we shipped 48 out of 52 two-week "sprints" to production. (If you must know, the breakdown was two long sprints due to Christmas and New Years, one quality issue for which we held back release, and two sprints with incomplete features that didn't make sense to ship.) In that two-year period we had a number of incidents — perhaps a half-dozen times that an upgrade caused an issue that needed development support. Overall quality was high and we have had few customer complaints about defects.
We also improved our skills at slicing our work very thin and in estimating how long features take to complete. It turns out that if you make all the slices roughly the same size, you don't need to estimate the work as much as measure it. When things get very uniform, the team can progress from "We ship every two weeks with what is complete and at high quality" to "We ship to production when this bag of work is done" — and still give management some confidence about deadlines.
So we moved from a classic Scrum approach to a more "bag of work" approach, which some might call Kanban. About that time, we started to slowly evolve our daily scrum meeting, first adding a "engineering topics" segment on Wednesday, then moving the meeting to every day of the week but not expecting 100% attendance. ("Engineering topics" include the kinds of things you might expect on a traditional staff meeting: announcements about recruiting, changes in policy, in-iteration suggestions for change, risks that the testers found that needed attention, and so on.)
Thanks to the Kanban board, the nature of the meeting is no longer "What I'm working on.” Instead we talk about what needs to happen when, and how we can adjust our priorities to help meet the needs of the day. Rather than talking about how to accomplish our goals and tasks, and why we are blocked, we've started to talk about what is blocking the team and what we can do about it.
No process is perfect, but I have to say, I like where this one is headed.
Please don't take this article the wrong way. I'm not saying the traditional daily scrum is bad, nor do I want to generalize and tell people that our way is better, more virtuous, or some kind of best practice.
Instead I am saying that good processes reflect, adapt, and change. Expect some growing pains, and demand improvement: From good to better. From better to awesome sauce. You get the picture.
Getting “better” doesn't mean the old way was wrong. Scrum doesn't work that way. Instead, it is an continually improving process. Like cooking soup, you flavor to taste, and you get better at it with every batch.
Along the way, you're not just shipping software; you are creating culture.
It can be hard, challenging, and painful.
And it's worth it.