Digging for gold with stakeholder surveys
What they can do, and how to run them well.
[Quick recap] Two weeks ago we chose a nice light topic to cover for our first article: What is the purpose of a data team? We assembled advice and approaches from many data leaders to create not a single over-arching purpose, but a framework by which we can see and understand the spectrum of data teams we see today. These three data team roles were: enablers, advisors, and connectors.
While I believe the framework is helpful, it’s important that this newsletter shares immediately applicable tips and tricks in addition to theory - things that you and your team can implement tomorrow and start seeing results right away. We will call these playbooks - and this week is our first, and it’s all about stakeholder surveys.
Big thank you to the many folks who contributed to this article, but an extra big thanks to the following for being kind enough to share their stories and surveys!
Lina Mikolajczyk - Director of Advanced Analytics at Bumble
Daniel McNamara - Head of Analytics Engineering at TapTapSend
Phillip Van Blerk - Director of Data and Analytics at Omnipresent
The problem with measurement
Measuring a data team is difficult, for reasons I will discuss in depth in an upcoming article, but the biggest challenge is that so much of what we do is not quantifiable. For example, the utilization of a dashboard doesn’t tell you if someone trusted the numbers they saw, made a decision based on that, and if that decision was a good one or not.
“I’ve come to a realization that the data team, even though we’re driving things forward and we’re trying to get deeper insights and drive deeper strategic impact, the reality is we’re still a service organization. And the best way to assess a service organization is to have some sort of NPS figure. But the NPS doesn’t have to be all promotors and detractors. It can really focus on the quality and availablity of the data and how its used by stakeholders.” - Lina Mikolajczyk, Director of Advanced Analytics at Bumble
Enter stakeholder surveys.
Stakeholder surveys are designed to measure all of those intangibles that reveal how we are really doing. How is the business really using data? Do they feel confident? Do they really understand their metrics or just get tired of asking you what everything means? What do they wish you were doing more or less of?
And with this information, we can make sure we’re working on the right things and doing our best to support our teams.
What can they do?
Case study
In another role before Bumble, Lina was Head of Data Science and Analytics for another company. When she arrived, she joined a team in flux. Based on some company-wide internal surveys her new data team:
was the least trusted and least trusting team in the organization,
had low happiness rates,
and had the highest rate of attrition
It was clear there were some serious problems, and to get to the bottom of it, they had to go out and talk to people. In addition to one-on-one and small focus groups, they also set out to do a company-wide survey to find out what people thought:
From this simple survey, they learned:
the majority of the company had a neutral or unfavorable view of the data team
many people didn’t understand how to work with the data team, or with their BI tool
many people didn’t think the data team was aligned with the larger business objectives
there were low levels of trust in the data across the organization
the data team’s perception of themselves was very different from how the rest of the company saw them
With the realization that neither the organization nor her team were very happy, Lina took action. She worked with department leaders, managers, and her team to process the survey results and start making a plan to correct the major issues it uncovered. The changes included:
changing how the data team was structured
updating how data and business teams communicated and how often they met
sharing the data team strategy and roadmap with all parts of the business so they could be held accountable
Lina and her team were able to dramatically reduce attrition on the data team, and the data team's trust scores shot up to the company average.
Furthermore, when Lina started at Bumble, one of the first things she did was implement another company-wide survey to take stock and make her team’s roadmap.
Tips for Starting Your Stakeholder Survey
1. Be specific about what you want to understand
Don’t just put out a survey for the sake of it. What do you want to get from this? Some examples are:
get a general sense of how your team is perceived and valued
know how well people trust and understand the data you provide
see if there is a tooling problem
identify data literacy gaps and find possible data champions
2. Promote the survey, and get others to help
For the survey to be helpful you need people to use it. To do that, you need to get the word out any way you can.
“I went into the organization’s all hands and said this is who we are, this is what we do, this is where we play. We’re asking you do do this survey because we want to assess whether the services we provide you are what you need and whether we’re doing the most impactful things because we want to build a proper roadmap for next year based on your results.” - Lina Mikolajczyk
Some other suggestions from Lina and Daniel are:
talk to team leaders and get them to get their teams to fill it out
tell people what it will be used for and how that will impact them
3. Keep it short
No more than 4-5 questions. You shouldn’t need more than that if you are being razor-focused.
4. Make sure you ask 2-3 questions to help profile respondents
While surveys should be anonymous, you can ask questions to help work out different populations of respondents, such as:
Which department do you work in?
How long have you been at the company?
How confident are they using data? (e.g. is this a Finance spreadsheet whiz, or a content designer who may be new to data)
4. Let people speak their minds
Include open-text questions where people can leave their thoughts and explain previous responses.
“I’m always surprised at how much people are willing to share when you ask. I didn’t expect anyone to use the open text field but nearly everyone did, and it became the richest source for feedback we had.” - Lina Mikolajczyk
Daniel takes this one step further by allowing people to chat more directly in the survey. His final question asks if they would be willing to talk about any of their responses and if so, to offer their name so he can message them directly.
Example questions
We talked to Lina, Dan, Phillip, and others about questions they ask in their stakeholder surveys and assembled a list. This is a good starting point if you’re looking for inspiration:
What do you do with the results?
1. Report findings to leadership
Let your leaders know what you’ve found, and have a discussion about what you plan to do about it. Often the survey may reveal hypotheses you’ve already had about the problems facing your team and the larger organization around data, but the survey can help turn those into quantifiable facts. With their support, you and your team are more likely to be successful.
2. Talk through the results with your team and make a plan
Working through the results with your team can be a painful process. Many data teams don’t struggle for lack of effort. We have all poured our hearts out on data teams trying to be as helpful and effective as possible but never seeing the fruits of that labor.
Be sensitive to how poor survey results may demoralize your team, and be sure to focus on what we all stand to gain by changing as a result of these findings. This isn’t a punishment, it's an opportunity for us all to have a better time.
3. And share back with the business along with your plan
“It’s important that you follow up with the organization so they know it was worth their time to do that survey. We immediately made the results available in Slack as well as what we were planning on doing with it. When we run data trainings for new hires now, this is the dataset we use as well. It’s become a shared artefact, not just something for the data team.” - Daniel McNamara
Being transparent with the organization about the results and plans to correct any issues you’ve found is key to establishing trust and holding yourself accountable for better results in the future.
4. Make this a regular thing
Continue to run surveys regularly and use them as part of your roadmap, OKRs, and other measurement frameworks.
What’s next?
Next week we’re diving into the abyss of measuring a data team. The topic has been a rollercoaster to dive into and I’m looking forward to sharing the journey.
Want to contribute?
Many of the folks we’ve spoken to for this series have been suggested to us by friends (“Hey I met so and so at a dinner and they had some really interesting ideas around X”). This has been an amazing way to hear from a variety of people and to capture stories like Lina’s featured in this article. So if you know someone who you think has some wisdom to share with us, let us know! We’d love to meet them 🙂