Setting up the metrics that helps Scrum Team can indeed be a very difficult job and, in this article, I will explore some things to keep in mind while you are setting up your Metrics

• Metrics Are Not About Keeping Things Green – One common anti pattern that I see in many Scrum Teams is where the focus of metrics reporting is to showcase that team is always “within acceptable boundaries” This is usually seen when stringent metrics are defined in agile contracts and are tied with financial penalties. These types of metrics encourage the Scrum Teams to represent the data in way that makes the metrics always seem “green”. In such a case the focus cannot be on what we need to do in order to improve. Where possible, I try and coach the teams not to have these service level agreements in their Agile Contracts – I encourage the teams to look at metrics as data points that will help them to improve rather than a tickmark needed for contractual compliance.
• Focus on End-to-End Metrics – It is usually a better idea to have metrics which measure the value end to end across the value chain rather than measuring value at interim points. To give an example, metrics measure end to end time required for a feature rather than tracking time for a specific task like coding or testing are more useful. Another example can be, number of zero defects features delivered is a better Metrics than number of defects identified during testing phase. When defined in this manners Metrics encourage Scrum Team to work as a unit rather than individual sub functions. It becomes a challenge to define such metrics in a multi-vendor scenario. 
• Measure Only What You Want to Improve – Metrics are not meant to be “frozen forever” – Often, I see exhaustive metric structures that take up a lot of time and energy for the Scrum Team to create and maintain the data. These metrics frameworks often are collecting data without having a plan of how the data will be used to help the Scrum Team to be more effective and efficient. I have seen teams that are lost in this huge data that they maintain. These teams struggle to understand the data and make any useful inferenve from that. In my experience, it is often easier to have a smaller number of metrics that the Scrum Team has to track. This way the team can actually use the data being captured. Some of the most effective organization and Scrum I have had good success with Scrum Teams that decide their improvement goals for the upcoming period of 6 months to an year and then determine the metrics that will help them track their journey towards these these goals. These teams periodically review their goals and adjust the metrics accordingly. This also aligns with the Scrum Value of Focus
• Avoid metrics that Compare one team or one person against another – Agile believes into collaboration. A scrum Team needs to wok in a Self managed collaborative manner to deliver value frequently. If the metrics measure individual(or sub team) performance, it may sometimes lead to people working against each other to protect their own Metrics level. It is a better idea to measure value delivered as the whole team. The same principal can be applied to creating a competition between the teams- for large products requiring many teams it can hamper the collaboration amongst Scrum Teams. It works better if the teams are measured against their own last performance and encourage to have a continuous improvement in their way of working rather than comparing against other teams.
• Trends Need To Be Monitored – NOT individual numbers – Sometimes a particular metric may show a bad value. This can happen because of multiple transient reasons and it may not be worth it to really worry about an isolated value or reading in a metric. It is much more useful to monitor the trends and analyze if the Scrum Team is going in the right direction.

In the previous article, I have described some sample metrics which may be tracked by the developers or the Product Owners. These metrics are designed to measure how well we generate value. The metrics can also track the frequency of delivery as well as the quality.

These metrics do not capture how well the Scrum Master is performing – When I pose these questions to people, they often respond that as a coach it will be difficult to quantify and measure the progress achieved by the Scrum mMster.

I do agree that there no easily apparent metrics prevalent in the industry to help the scrum master to inspect and adapt based on quantified data. However I do believe that the scrum master needs to have some quantifiable tools that can help him or her to track and monitor the progress

In order to identify such metrics, it may help consider the key accountabilities of Scrum Master as explained in the Scrum Guide. As per the latest scrum guide, the scrum master is accountable to

• Help the Scrum Team continuously improves its effectiveness
• Help the Developers are able to create high value increment that meet definition of done
• And the Scrum Team implements scrum the way it is defined.

When we look at these accountabilities and the position of SCrum Master as a “true leader who serves”, it becomes clear that to measure the effectiveness of Scrum Master we need to somehow track how the Scrum Master helps the Developers as well as Product Owner to become more effective and efficient while staying true to Scrum.

Also, the Scrum Master is a coach for the Scrum Team. As a coach the Scrum Master should help the Scrum Team to be Self Managed and able to take their decisions independently without depending too much on the Scrum Master. Scrum Master also coaches the organizations to empower the Scrum team.

Below I have given some sample Metrics that can help the Scrum Master.

1. Team Independence –
   1. Number of times Scrum Master has to be the interface for the developers – with PO /With Organization
   2. Support needed by team to solve impediments
2. Over All team effectiveness – trends in other metrics especially value delivery and quality
3. Cross Functionality – Can be measured by something like “T Index”

Common Misconceptions

People think that just by moving to Scrum all their issues will get resolved and magically all timelines will be met, no budgets will ever get overrun, all people will be happy, all customers will be happy, all changes come free and so on….

What Really is Scrum?

Scrum is a framework for solving Complex Adaptive Problems. Complexity means “Unknown-ness”. The Unknowns can be on requirements or on solutions. Where things are unknown, you cannot make detailed plans and just track the plan to closure. In such situations where things are unknown, what you see in front of you can be the only decision factor. Taking decisions based on what you see and constantly Inspect and Adapt based on what you see is called “Empiricism” or “Empirical process Control”. Scrum is a framework which sits on the “Empirical Process Control Theory”. For known problems (requirements known and solutions known), you will not need the inspect-adapt approach. The approach used for known problems is called “Defined Process Control”. Waterfall framework is based on “Defined Process Control”

Recommendations

One must understand that Scrum does not necessarily reduce cost and timelines. In Empirical process control, there will always be a feedback structure since the decisions are taken based on what you see. That means, we think about short-term in detail and long-term at a high level. When we see the results of short-term, we get feedback and then we inspect and adapt based on what we see. Therefore, there may be repetitions due to the feedback structure. Repetitions may mean that some re-work may happen. One must not think that just by doing Scrum, the cost and time will be lesser. If we use Scrum for problem statements which could have been solved using waterfall (or Defined process control), then unnecessary overheads and unnecessary inspect-adapt cycles may increase the cost and timelines. My recommendation is to choose the right framework for the right purposes. Taking a “one-size-fits-all” is not a great idea.

The teams are not necessarily happy doing Scrum. In the Scrum way of doing things, the timeframes are short. Delivery cycles shrink to a few week cycles. There is a constant pressure on teams. Many times, you will notice comments in the team like “Waterfall was better”. Scrum is actually Pervasive. That means people don’t like this drastic change in their lives. This change touches people’s lives. My recommendation is to ensure that every organization first implements a culture which sustains this pressure. The earlier model, where there was “push culture” should change first. Management should not create unnecessary pressure on the teams. Agile principle number 8 should be implemented first “Agile processes promote sustainable development. Sponsors, Developers and Users should be able to maintain a constant pace indefinitely”

The business teams are not necessarily happy doing Scrum. The Business teams were used to pushing accountability conveniently on the technical team’s side under the umbrella of a few terms like “Managed Services” or “Managed Outcomes”. The IT companies also marketed these terms like “Managed Services” to create differentiators for themselves. The Business teams suddenly start feeling the heat when they are held accountable for ensuring that the delivery happens properly and making sure that they have to work with the technical teams on a day-to-day basis. In Scrum, we define the accountability “Product Owner” where participation from a Business perspective is extremely important. My recommendation that the Business Teams must be made to understand that Accountability cannot be outsourced and one cannot toss the accountability on the technical side. The business should be brought in sync with the Agile principle “Business People and Developers must work together daily throughout the project”. Scrum emphasizes that the right people take the right accountability and accountability cannot be outsourced.

The Business teams have conveniently understood that “Changes are free in Agile and Scrum”.  One has to understand that no changes are ever free. There is always a cost associated with a change. It is just differently looked at in Scrum. In Scrum, we tend to give importance to more valuable features. That means, the less valuable features go down the product backlog and may never get implemented. So, pushing the teams to do all requirements may not be a great idea, else the cost will obviously go up. My recommendation is to follow the Agile principle number 10 which says “Simplicity – The art of maximizing the work NOT done is essential”. The business has to know that nothing in this world is free and we need to learn to compromise less-value features for the more valuable ones

Conclusion

Scrum is not a magic-wand solution to solving all problems. In fact, if you do things incorrectly then the problems will multiply. It is better for organizations to understand Scrum and Agile correctly and apply only where required. “One-size-fits-all” is not a great approach.

Myth about every Sprint Output resulting in a go-live to production

Common Misconceptions

Most people feel that every Sprint MUST make an Increment which HAS to be sent to production and given to the end-users for using. This is actually not true. Sometimes, it is not possible to produce Incremental “production-ready” outcome every Sprint

What really is the outcome of the Sprint?

Scrum defines the “OUTCOME” of the Sprint as an Increment which is “USABLE”. Usability is associated with getting feedback and not necessarily “Go-Live”

Recommendations

• Sometimes, in a short sprint it may not be possible to create an “Go-Live-Ready” product. However, every Sprint must result in a deliverable where feedback can be received. The feedback may happen from internal stakeholders and not necessarily the end-users
• Sprint should be considered separate from “Release”. While “Release” is not a Scrum term, most teams use the word “Release” to describe “Go-Live-Ready” outcome. Generally, it is observed that multiple Sprint outcomes can be combined together to create a “Release”. However, this is also not a necessary condition. Sometimes, teams may create multiple “Go-Lives” within the Sprint itself. It is important to understand that feedback is necessary every Sprint so that we know as a Scrum Team that we are going to achieve the Product Goal (Long term Goal). The intent of Sprint should be to optimize the probability of achieving the long-term objective and the feedback received should be used to optimize this probability. Now, the feedback may come from internal stakeholders or end-users. Important part is that feedback should come every Sprint.
• While this article says that it is not necessary to create a “Go-Live-Ready” outcome every Sprint, that does not mean that we create “Documentation” as output. We CANNOT be doing a “Requirements Sprint” which creates a “Requirement Document” as an output. We CANNOT be doing a “Coding Sprint” to create a “Code” as output. If we create outputs such as Documents then what we end up doing is “Outputs” (which are of no value) instead of “Outcomes”. “Outputs” do not necessarily result in “Outcomes” which are of business value or feedback-able value. “Outputs” do not necessarily result in feedback where we can check progress towards the Product Goal.

Conclusions

Sprint creates an Incremental outcome which is called Increment. The Increment should get the Scrum Team feedback, which will optimize the probability of meeting the Product Goal. The Incremental outcome may be a “Go-Live-Ready” outcome many times within a Sprint or sometimes over multiple Sprints. The Increment should never be an output which is not a valuable outcome.

Common Misconceptions

Most people consider Daily Scrum is presumed to be a Status Meeting where Scrum Master takes status from the teams. One of the most common reasons for this is because people have used Project Manager as an entity for years and assume that Scrum Master is a renaming of Project Manager role and Daily Scrum to be a new name for “Status Meeting”.

What is Daily Scrum Really?

• Daily Scrum is not a Status Meeting. Daily Scrum is a inspect-adapt forum for the Developers to make a plan for next 24-48 hours by checking if the Sprint Goal is being met or not
• Developers sync up daily so that the plan for next 24-48 hours is effectively done
• Updating the Sprint Backlog should be the main goal of the Daily Scrum. This optimizes the probability of meeting the Sprint Goal
• Daily Scrum is a effective forum for improving communication and collaborations between Developers

Recommendation about Scrum Master’s Role in Daily Scrum

• Scrum Master may facilitate the Daily Scrum to ensure it gets over in 15 minutes and people do not make it a detailed discussion forum. However, Scrum Master is not considered the compulsory participant of Daily Scrum.
• Scrum Master’s role is to make the team independent and help them becoming self-managed team. Therefore, Scrum Master should teach the Developers to run the Daily Scrum effectively by doing a quick sync up and updating the Sprint Backlog
• Once the team becomes self-managed and know how to get the Daily Scrum completed within 15 minutes, Scrum Master should change his/her stance to an observer stance. That means the team has to talk to each other without Scrum Master’s intervention
• Scrum Master should teach the Developers to keep the status transparent through various tools and techniques such as Burn Up Charts, Burn Down Charts, Scrum Boards. This avoids a lot of wasteful status discussions in the Daily Scrum
• Once the Scrum Master feels that the Developers are managing on their own, the Scrum Master should purposely skip a couple of Daily Scrum and find out if the Developers still keeps doing the Daily Scrum on their own. If the Developers also skip the Daily Scrum, just because the Scrum Master was not there, then Scrum Master should treat this as an opportunity to coach the team again
• If Scrum Master notices that Product Owner is conducting reviews of Product during the Daily Scrum, then, Scrum Master should coach the PO not to do it and create another forum where the review of the product could be done
• Similarly, if Scrum Master notices that the line managers are wasting the time of Developers by taking status, then Scrum Master must intervene and prevent the line managers from doing this
• Sometimes, one of the Developers wastes a lot of time of others during the Daily Scrum trying to dominate the forum by showing off on his/her skills. Scrum Master’s job to facilitate and ensure that everyone gets equal chance during Daily Scrum. There are many facilitation techniques which could be used. For example, tossing the ball every 60 seconds to the next person to speak, introducing friendly penalties (buying a drink for everyone, doing 10 push-ups) for exceeding the time allocated to speak etc.

Conclusion

Thus, Scrum Master is not a mandatory participant of Daily Scrum. “Not Mandatory” does not mean that Scrum Master is always Absent in Daily Scrum. Scrum Master’s role is to make the team independent so that they can manage the Daily Scrum to inspect and adapt and optimize the probability of meeting the Sprint Goal. Once the team becomes independent, it is better for the Scrum Master to take a step back and take observer position.

Some Sample Agile Metrics

Each team needs to define Metrics that help them inspect and adapt. Below I have given some metrics PO as well as Developers can use

Measuring Value –

Scrum is a Value Framework. The focus is on “Value delivered to customer” rather than “scope delivered as per plan”. It is then clear that we need to find a way to measure “value”.  How a PO will measure value will be different than how developers will measure Value. The table  below shows some sample Metrics that can help the PO and developers do the same

Perspective	Some Metrics
PO	Revenue Feature Usage Reduction in Complaints Increase in CSI….. NPS
Developers	Defects Leaked Prod/UAT First Time Pass Rate  Tech debt

Measuring Frequency-

Agile manifesto states the importance of delivering value early and in a continuous or frequent manner. Measuring the frequency of delivery will help  the PO and Developers to

Perspective	Some Metrics
PO	Lead Time Release Frequency
Developers	Velocity Sprint Duration

Measuring Day to Day Work

While measuring the Value delivered and the frequency of the said duration gives PO and Developers

Perspective	Some Metrics
PO	# Of Times PBI (Story)need to be re-negotiated or acceptance criterion updated How many times PO has seen a feature before Review
Developers	Work prediction– burnup, burn down , WIP , CFD Planning Quality – Estimations, sprint planning value (# of tasks identified Vs Time spent) Operational Excellence – Broken Builds , automation %, test coverage , code quality Self Organization Index – how many tasks they needed outside help Cross Functionality – “T Index”

Do we need Metrics in Agile Projects?

We saw in earlier article that traditional metrics do not help when working in Agile manner. The question then comes to mind is do you really need metrics while working in Agile?

The answer is a definite yes – When done well, Metrics provide us with transparency by giving quantified information that does not depend upon individual perception. This quantified information helps us to inspect the work we are doing and value we are delivering. The inspection then allows us to adapt our way of working to improve the value we deliver

In other words, metrics help us to work in an empirical manner.

What should we measure while working in Scrum?

In My experience there are 3 major dimensions where Metrics helps us to inspect and adapt

1. Value – Are we delivering Value?
2. Frequency – Are we delivering value frequently enough?
3. Day to day Metrics

Perspectives in Metrics

For each of the metrics above, developers as well as Product Owner  will need different information to inspect and adapt their way of working and metrics need to be defined accordingly

Traditional Metrics are often centered around measuring planned versus actual. They measure efficiency of scope delivery.

The focus is often around measuring the efficiency of the delivery be it the schedule or cost adherence, Scope delivered or quality of the delivery.

Some of the usual metrics companies track are

1. Schedule and Cost Variance
2. Volume Of work delivered (KLOC)
3. Productivity
4. Requirement adherence /Rework

The above metrics help us to see if the teams are delivering as per plan – But in Agile we believe that while planning is indeed quite valuable – plans need to be updated often. Then these metrics do not seem relevant when plans are being updated very often

Another key point missing in the above metrics is the focus on value – These metrics measure only “what work the team did” they do not measure “what Value was generated”

Common Misconceptions and Negative Implications

·        It is a common practice for a Product Owner to come up with a set of requirements in the Sprint Planning and disappear during the Sprint. Then appear directly at the Sprint Review forum and conduct the review of the Developers work during the Sprint Review.

·        By doing this, an important opportunity of giving feedback continuously is lost. The feedback becomes a big-bang-end-of-the-sprint activity

·        Since Product Owner is seeing the product first time during the Sprint Review, it can become difficult to take feedback from stakeholders

·        If Product Owner brings along the stakeholders to do a Sprint Review, the Product Owner ends up acting like the “Other party” and does not build confidence in the team doing the work

• When the Product Owner sees the newly increment only at review time, Sprint Review becomes a gate that holds back value from reaching the users. When the Product Owner sees the increment during Sprint an opportunity is created to release value earlier

Recommendations

·        The Product Owner owns the product. Therefore, it is important for the Product Owner to participate in the product development continuously and constantly provide feedback.

·        The Product Owner has to build confidence with the Developers. Developers being left alone is not a great idea. The Product Owner should not behave like an outsider and should not sit on the other side of the table. Once the Product Owner wears the “Product Owner Hat”, he/she is first part of the Scrum Team and then part of the Customer or Client or Stakeholder team

·        Developers must not fear the Product Owner. Feedback is not a cause for worry. Feedback is good. The earlier you receive the feedback, the better it is for the Product

·        Typical steps in the Sprint Review that are seen useful by many teams could be as below

o   The Product Owner should invite the relevant stakeholders for a Sprint Review

o   During the Sprint Review forum, the Product Owner should sit with the Developers

o   The Product Owner should thank the Developers in front of the stakeholders for the work done

o   The Product Owner should take ownership of the product during the Sprint Review and collaborate with stakeholders to make it a working session

o   During the Sprint Review, the feedback should be from Stakeholders and not from the PO. The PO feedback is assumed to have been taken already during the Sprint

o   The stakeholders and Product Owner should think about the next steps based on market place conditions and make adjustments to the Product Backlog

Conclusions

·        Sprint Review should be a working session between entire Scrum Team and Stakeholders

·        Product Owner should be reviewing the product with stakeholders with help of the Developers

·        Product Owner should have provided feedback to the developers continuously throughout the Sprint and should not be seeing the product first time during the Sprint Review

This post is republished under my name @ Myth – Sprint Review being the review of the Developers done by Product Owner (worldofagile.com)

Common Misconception

A Product Owner is considered a “Client” or “Customer”. So, most people feel, how can you invite a “Client” for an internal forum to discuss “what went well, what did not?”

Recommendations

• Most Product Owners indeed behave like a “Client” or a “Customer” instead of a Scrum Team Member. Therefore, the Developers do not consider the Product Owner a part of them and consider him/her as an outsider. The Product Owner should be first part of the Scrum Team and then be a “Customer” or “Client”
• Product Owners often ridicule the teams or shout at their teams for not understanding the requirements properly. Product Owners often escalate against the teams to the Line Managers of the Developers. This creates a divide between the Product Owner and the Developers. The Developers then don’t feel comfortable about opening up in front of the Product Owner. The Product Owner’s job should be to get the Developers to feel comfortable and put the “fear-factor” to rest. This means, the Product Owner should be empathetic, calm, show patience and be looked at as a Leader instead of a manager.
• Sprint Retrospective is a forum to see what went well and what did not go well. This increases the effectiveness of the next Sprint. If the Product Owner is not there in the Sprint Retrospective, then effectiveness improvement will be looked at only from the perspective of the Developers. The Developers are doing the technical work for the Product itself. So if the Product Owner is absent in a Sprint Retrospective, the effectiveness improvement cannot be done from the Product perspective
• Sprint Retrospective is a good forum to discuss the DoD (Quality Measure for next Sprint). If the Product Owner does not attend, the Sprint Planning of the next Sprint may be an issue since DoD is not finalized.

Conclusions

As a part of the Scrum Team, the Product Owner is a mandatory participant of the Sprint Retrospective. Sprint Retrospective is an excellent forum to thrash out any opinion differences and plan for improvements and everyone from the Scrum Team must participate.