Lessons From the Daily Scrum

What is Scrum?

Scrum, as you can see in the image below, and as per Wikipedia, is the method of restarting play in a Rugby game that involves players packing closely together with their heads down and attempting to gain possession of the ball.

I am pretty sure that is not the definition that you wanted to see, at least not in this blog. But, bear with me for a couple of seconds. It was not a failed attempt at a bad joke. I just wanted you to see the imagery and get a feel of what is happening. Also, I wanted to throw some light on the history of that name. How did a Rugby method give its name to the popular Agile methodology that we use almost throughout the software engineering world?

An oversimplified definition of Scrum is that it happens to be a common Agile framework where actions are planned to be completed in timeboxed intervals known as Sprints and progress is tracked on a daily basis in short meetings known as Daily Scrums. This Daily Scrum is a stand-up meeting with team members standing in a rough circle and explaining what they intent to work on for the day and any roadblocks they may be facing. Since this arrangement looks like a civil version of the rugby scrum, both the daily meeting and the process take the name of this particular Rugby activity.

The point of this is that the entire team meets, gets to know the statuses of all the activities being done, finds mechanisms by which anyone who is stuck gets help and anyone should be able to suggest improvements. Because of this, it also allows for course corrections and other changes needed to make sure that what was planned for in that Sprint is indeed completed within the same sprint. Everyone is present to tackle whatever situation is there at hand, both in rugby and in this Agile Methodology. Hence, the name is shared in both the disciplines.

Challenge in Implementation

Scrum is probably the most popular of all the Agile Methodologies in the software engineering world now. It is popular because it gives you as a team manager and as a team member a lot of freedom and expects you to fill only a minimal amount of documentation. It also offers both stakeholders and developers opportunities to reconsider and change their minds about requirements, architecture and other stuff that is usually set in stone in an upfront manner in other older methodologies. If you need to change any of these, you only have to wait until the end of the sprint. It also has periodic feedback built into it which is facilitated by the Sprint Review and Retrospectives, and the availability of a dedicated Product Owner gives you the ability do a lot of experimentation with your feature set before arriving at a final one.

For us, choosing Scrum made a lot of sense. After all, we had seen a lot of benefits of a Scrum in our previous organizations and have been able to make good use of them as well. But it was not smooth sailing, at least not in the beginning. Here were the primary reasons.

Non-Constant Tasks During a Sprint

Our company, pCloudy, is essentially a SaaS offering and in a startup frame of work. So this usually meant that some issues would crop up mid-sprint which were important for a major customer, and the priority, therefore, gets bumped up. This would mean that developers have to strategically pivot and fix those issues, many times leading to the issues that were originally planned getting sidelined. So the usual 2-week sprint got broken all too frequently and this meant we had unfinished tasks at the end of each sprint, which was definitely not desirable.

Sprint Length

The ideal sprint length is considered to be 2 weeks to a month, with the most popular length being 2 weeks. This is ideal because it is short enough for a feature to be completely done from end to end and yet short enough for stakeholders to experiment with features and decide to pivot if needed. However, with many customer-related interruptions coming, we decided to reduce the length of a sprint to one week. The idea was that we would take customer interruptions only in the next sprint, which never would be more than a week away, and with this whatever we planned for in the sprint would be completed. However, what we say here was that a week was usually not enough time for a feature to be completely developed, let alone for it to be tested and deployed. This meant that at the end of the sprint, most of the features were not “done-done.” They would be in various stages of completion. Another issue we faced was the length of the review and retro meeting of one sprint and that of the planning meeting for the next one.

Our Scrum(ish) Principles

So here is what we finally arrived at, which takes the best principles of Scrum and at the same time enables us to be as Agile as possible. Remember, we are not here to be Agile champions, so agility is very highly desirable, but not indispensable. We did some modifications to our process, one small change at a time and kept doing those changes. But as our changes continue, I am pretty sure our methodology will not look like what it looks now even 6 months down the line. Here are the most salient items from our thought process.

Sacrosanct Backlog

Every Feature that we need to pick up is stored in a backlog. We used JIRA for some time and have now migrated to Trello. But the point is that our Agile Backlog is stored in an automated tool. Tomorrow, if our team’s complexity changes, we may choose something else, but the point to note is that team members are able to access the backlog and are able to see all the items they are required to work on and report all changes in statuses. With JIRA we used Kanban like a board that the system provides, and Trello by default comes with this board. This board was one of the most important artifacts used by the team.

Also, we changed our methodology in such a way that while the Backlog itself is sacrosanct, but we allowed ourselves to change the backlog of a sprint to ensure that our customers are not waiting. But unless something earth-shattering happens, we do not interrupt a particular task before it is finished. So strictly speaking, we are not doing Scrum, but we are in fact doing something more in line with Kanban with some trappings of Scrum like Planning, Review, Retrospectives and, of course, the Daily Scrum. While we continue to internally call the daily meeting a Scrum out of habit, we also do not pretend that our process is Scrum. It is now a form of what many people call the “Scrumban,” where the best features of Scrum and Kanban are merged together to form a practice that looks like both. We have found that at this time, this suite our business and development needs best.

One Scrum/Many Scrums

We were once a smaller team and it made a lot of sense for the entire team to have a common Scrum, but as the team became bigger, the Scrums dragged longer and both heads and legs began to ache. We then divided the team based on the specific area that each of the team worked (Portal, Devices, etc.) and had each team have their own Scrum. Then all the Scrum Masters of each team had another “Scrum of Scrums” where the larger team could focus on larger goals. This made each Scrum shorter and each Scrum more relevant for each attendee.

Management View

When the team was small, management found it easy to sync with each team member and follow their pace of tasks with a single tool like JIRA or Trello. However as the team grew, management needed something more robust and flexible to track their day to day tasks. We started with a shared Google document that managed the status of each work item which was always updated and visible to management. The shared Google doc, while easy, seemed a bit flimsy. We then moved to something a bit more formal with Atlassian Confluence. Now broader level tasks are looked at in Confluence, which breaks down to smaller issues in Trello for each developer to work on. The Scrum Masters make sure that Confluence is updated based on the status of the tasks in Trello. Conveniently, the Confluence report has links to Trello issues.

Next Steps

So what is next and what is the way forward? Well, I do not want to answer that question. That question takes the fun out of the journey, the fun of not knowing what you will discover at the next bend, what you will invent in the next time of need. However, as I have kept saying, ours will be a journey where we continue to tweak our processes so that it fits us at every phase of our growth.

We are still young as a company and still growing and we need to be able to continue tweaking, at least till we are in a few hundred in terms of headcount. But that doesn’t mean we are blind about the future, and we have no idea what to expect. In fact, on the contrary, every tweak we make also comes with the question of how long will this stay and what do we need to do when we outgrow this phase? This is a constant question that we ask ourselves and make sure we have answers. Whenever we don’t, we step back and replan.

Appendix

Before I sign off, let me also shed some light on the experiences we had with some of the tools that we used to track the development. Also a small tidbit about compliance.

Excel/Google Sheets

A spreadsheet software is the simplest and most flexible mechanism to track virtually anything. In fact, it is this simplicity that tempts a lot of people to run their software agility through a series of shared Excel sheets. We had the same experience, but the only problem is that if you have to manage complex calculations and deductions, then the number of tasks that you have to perform to keep your sheet in control becomes difficult to manage. This is not a problem if this is the main thing you do or if you have people to do it for you; otherwise, it soon becomes very cumbersome.

JIRA

JIRA is a globally popular Issue Management System which is very helpful in tracking issues. It also deals well with dependencies and broken down tasks. It does what it does very well. But it has one issue that makes it a bit difficult for a fast-paced startup. And that is it is very rigid and inflexible. Maybe it was designed that way and it was intentionally not made too flexible so that there are not too many changes done. I wanted some changes in the workflow that was specific to the work that we were doing and I could not quickly do it. Remember, altering JIRA was not the only or the most important work that we were doing. I tried a lot to understand the complexity of JIRA under the hood, but I could not achieve something as simple as adding as many more columns to the JIRA Kanban board as I wanted. I know it could be done as I have seen something like what I wanted working elsewhere, but I simply did not have the patience to understand the arcane JIRA workflow editor and other such entities in it. I understand, this is probably a job for a JIRA admin, but we were not so big to recruit one to do the customization for us. So, for now, we went to another tool from the Atlassian stable. Probably we will revisit JIRA when we grow bigger. And that brings us to the tool we are using now.

Trello

We have settled on Trello. Trello is another project tracker from Atlassian. It is built around the Kanban board and allows you to do a lot of changes to the Kanban Board, including custom columns(we view them as states). We liked it a lot since I can now track the exact (and many times specific to pCloudy) state each item is in like Backlog, Requirements, Design, Test Planning, Coding, Testing, Code Review, Ready to Deploy, Post-deployment Sanity, and others.

Compliance

One common thing with these tools is the amount of developer (and tester and DevOps engineer) compliance. The success of any of these tools depends on the readiness of the developer to actually update statuses in these or any tools. I have seen this for the past 15 years that many developers, including me when I was a fresher, simply do not want to do this. Somehow they are ready to spend nights and weekends in office running behind an elusive bug, but spending 5 minutes updating the tool is many times. You have to make your developers see that it is imperative to do the necessary status and other updates in these tools. If your developers do not do it, then none of these tools will be useful. In the end these tools as much for the developer as much as it is for a manager, but they will work only if they are used properly.