How to know if your data team is successful?
The dark art of measuring ourselves.
After asking data leaders what the purpose of a data team was, I asked one follow-up question: ‘How do you know if you’re successful?’
This was almost universally followed by a heavy sigh. This is not a simple question.
We in data have never been great at measuring ourselves. Despite our role in helping others measure their success, we lack a standard set of metrics, or even a framework by which to measure our own.
“You’d think since we’re in the business of measuring things, we’d be great at measuring ourselves, but we’re not. We’re probably worse at measuring oursleves than we are anything else.”
- Conor O’Kane, Head of Analytics at Cleo AI
Articles on the topic generally gravitate towards laundry lists of operational metrics like dashboard utilization and time to insight. But do those really tell us if we’re doing our jobs well? (They don’t).
You won’t be surprised to hear then that this article was particularly challenging to write, but one I thought was well worth the effort.
The following is everything I learned from these leaders about the different approaches to measurement, the patterns I found, and the advice they offered for those on the daring mission to define and measure their own success. Here it goes…
Thanks to
Anders Kring - Director of Data at Templafy
Alan Cruickshank - Insights Director at tails.com
Carl Johan Rising - Director of Data at Too Good To Go
Conor O’Kane - Head of Analytics at Cleo AI
George Spiteri - Head of BI & Analytics at Chilliz & Socios.com
Jimmy Pang - Business Intelligence Lead at Vestiaire Collective
Margot Monges - Data Analytics and Data Engineering Manager at Dott
Triston Cosette - Principal Analyst at GetYourGuide
William Mahmood - Head of Data Science at MUBI
for your help navigating this heavy topic!
Do we even need to measure ourselves?
Of all the data teams I’ve known and worked with only about 10% of them had formalized metrics for themselves. Many of these teams without any kind of measurement were still objectively healthy - they were valued in their organizations, and they seemed to have satisfied team members. They just didn’t have any way of quantifying all that, which begs the question, do we even need to measure ourselves?
The simple answer? No.
In fact, measuring for measuring sake can be harmful:
“If you’re in an environment where you need to really, really distinctly show the impact of a data team or an analytics team, your data and analytics team will end up working differently. That’s because they have to work in a siloed way so that you can draw a dotted line around them and isolate the effect they’re having. In trying to make our impact easier to measure - we in turn lessen the impact we can have.”
- Alan Cruickshank, Insights Director at tails.com
But would any of us even be in data if we didn’t understand the potential value in measurement, especially when it is done well?
Measurement done right can be an invaluable activity - one that helps us understand ourselves and our wider context better, and that can help us deliver more value to the wider organization.
Teams that did have some kind of formalized measurement framework did so for the following reasons:
aligning the team around a common mission or goal
knowing if you’re getting closer to achieving it
staying true to your team’s values
We’ll dive into these different motivations for measurement below. But first, a brief digression.
A brief digression: What measurement won’t help with
One mistake many people highlighted was using metrics to help prove your team should exist.
When leadership asks you for metrics to quantify the value of the data team, the question behind the question is usually ‘Do I really need a data team?’ or ‘Do I need one of this size?’. Those concerns won’t be quelled with your utilization numbers or your average time to insight.
And in case it wasn’t obvious, the costs of not addressing their concerns can be high:
“Over the course of my career I have seen multiple data science teams who were doing fantastic work in building and sharing models getting dismantled overnight. The simple reason was that the business wasn't seeing their value. The value was there. I could see it. But the business didn’t.”
- George Spiteri, Head of BI & Analytics at Chilliz & Socios.com
So what should you do if you’re asked to ‘prove your value?’ Here are a few tips from the leaders I spoke to:
1. Go back to purpose
As a very frustrated HR colleague once told me “Frustrations are often just a failure to communicate expectations.”
Meaning the reason you’re being asked to demonstrate your team’s value is likely that no one knows what to expect of you. Or they have expectations of you that they haven’t told you about, so they perceive you to be failing.
For example, if you believe your data team’s role is to advise the business on good decisions, then someone can be very frustrated that you aren’t just serving them reports and numbers as quickly as possible. You believe you’re partners, and they believe you’re a service team.
So, one of the most powerful ways to address the existential question is to clarify who you are, and what others should expect from you and your team. This practice of consistently and proactively communicating who you are, how you can help people, and what people should expect of you is one of the things I’ve noticed can distinguish good from great data teams.
If you’re looking to reset the expectations of your data team, try the following:
Speak at other department meetings to introduce yourself, clarify what the role of the data team is, and how people can start engaging with the team
Meet with department leaders, and explain how your team can help them achieve their goals
Make sure all new hires are introduced to the data team and expectations are clear on how and when to engage with them
And if you haven’t check out our earlier article on finding your team’s purpose here.
2. Focus on how people show that they value you
If you still find yourself trying to justify how valuable you are, it can be helpful to focus on how other people in the business demonstrate that they value you.
💡 For example, one team I spoke to has relinquished their data budget and given it back to the larger company. They then ask for business units to pay for their own analytical resource. This is perhaps the best sense of how much they value the role of the data team since they are putting a dollar value on it, and some of the best advice I received on this topic.
Some other examples of demonstrated value are:
Satisfaction scores on stakeholder surveys (akin to NPS scores)
Anecdotal evidence and quotes from stakeholders
3. Get closer to the business
In my conversations, I observed that data teams with embedded analysts were asked to defend their existence far less often than centralized teams. The organizational structure creates a shared sense of value that is so often felt that it is rarely called into question:
“What I like to do is make us more visible and integrated into the business teams to show the value we bring. I try to have every analyst a part of the projects they work on, not just supporting with numbers, but participaing in the decision-making and project management. It’s not quantitative but does create a valued image of Data Analysts among the business teams.”
- Margot Monges, Data Analytics and Data Engineering Manager at Dott
In short, the question that seems to be all about metrics is probably not about metrics at all. But now, back to the regularly scheduled programming.
Measuring to move the needle
Setting targets and goals for the team is an incredibly useful exercise. These goals help unite the team around a common mission, and ensure that everyone is working towards that outcome. This can take a few different shapes though.
1. Knowing we’re having impact
As we mentioned above, most teams know if they’re successful if the business is also successful. However, it’s helpful to break that down further. What does it look like inside the data team when we’ve achieved that outcome?
These more internal objectives help focus the team on what’s important:
“[To know if we’re successful] I look at the strategy docs that are coming out of each department and seeing how aligned those are with the insights coming from our work. Sometimes I’ve had the analyst actually writing those memos. That means we have a seat in the room and are making a difference.”
- Triston Cosette, Principal Analyst at GetYourGuide
Other data team goals I’ve heard are:
% of questions from the business that are answered by the business (not the data team)
% of stakeholders that say they trust the data team and the assets it provides (via a survey)
how frequently we are asked to step in and help on strategic decisions
2. Changing habits and behaviors
Many times these goals can help influence behavioral changes within your team. For example, Carl Johan Rising, the Director of Data at Too Good To Go, is planning to experiment with this idea in the New Year by introducing a new key metric for his team: the percentage of work tied to decision-making.
“They need to ensure that they always ask that question - what decision are you trying to make, and by what criteria? This goal will help encourage that behavior.”
- Carl Johan Rising, Director of Data at Too Good To Go
Carl Johan’s plan underscores the purpose of setting good goals in the first place: to focus a group of people around a shared ideal outcome. With that idea, your team goal can be as broad or as narrow as you like. The power comes in simply choosing one.
Measuring operations and efficiency
Ah, yes, everyone’s favorite thing to measure. This group includes metrics of efficiency, downtime, coverage, utilization, satisfaction, and speed. Ideally, they help us see whether we’re on track to hit the goals we’ve set above, so they should reveal two things:
what can we learn from past mistakes and inefficiencies?
can we take any immediate action to avoid issues right now?
Most data teams are good at capturing these kinds of metrics, which can be a weakness. Many teams I spoke to had at least one experience of tracking too many operational metrics or simply focusing on the wrong ones.
Therefore the key with these metrics is to be selective. For example, don’t worry about freshness until you find out that it is causing trust issues, which is diminishing your ability to influence the larger strategy.
Other examples of operational metrics include:
speed to understanding or speed to decision
be wary of speed to insight as it is ambiguous (what exactly is an insight?) and can encourage bad habits.
dispersion: the typical return on a single piece of work
utilization of assets
cost of tooling and support
Data quality metrics like:
freshness
downtime
test failures
test coverage
model coverage
It’s also important to balance operational metrics with qualitative feedback from stakeholders, such as:
% of people who say they trust the data provided to them
% of people who feel confident analyzing and making decisions off the data (and tools) provided to them
% of people who are satisfied with the speed and quality of the data provided to them
Measuring to stay true to our values
Having a set of goals for your team that are about job satisfaction can be a powerful way to build morale and keep the team motivated.
Anders Kring, Head of Data at Templafy, has the following things that he wants his team to check for themselves every day:
Did I create value today for myself, my team, or, preferably, the company?
Did I feel successful today? Did I achieve what I set out to do?
Did I learn something or teach something to somebody else?
Most importantly, did I ensure I did not take it all too seriously and broke my neck while wanting to achieve the former three?
He reviews this with his team regularly as a group and individually so they can calibrate their priorities and ensure they’re having a good time while delivering value for everyone else.
Where to begin?
Looking at all of this can be overwhelming, especially if you are one of the majority of teams with minimal measurement in place today. To get started, here are a few questions to ask yourself:
Are you being asked for metrics from leadership? If so, then maybe it's best to start by defining and resetting expectations with the wider organization before measuring anything.
Are you a lab or a factory? Are you focused on optimization or transforming your team? There is always a balance of both required, but you should have a sense of where your team is and what is most important. If you are optimizing, then starting with operational metrics may be best. If you’re trying to transform and grow, then start with a very focused OKR for the next quarter to inspire even a small change you want to see.
Does your team feel like they’re successful? If you have a culture or motivation problem, then starting with ways to measure how true to your values you’re being can be a good place to start.
Here’s a summary of the above information:
What’s next?
As always, if you have any topics you want us to cover or any folks you’d love to have us interview, drop me an email: taylor@count.co. Otherwise, we’ll be bringing you more content in the New Year on topics like hiring, data team structures, and what we do about analysts. There will also be more practical playbooks from teams at Trainual, Substack, Gelato, and more!