How to achieve consistency over your metrics?

‍

Introduction

Inconsistency over metrics is a painful issue. Metrics and KPI's are vital to help stakeholders track performance and align towards goals. In most organizations, metric definitions are spread across spreadsheets, internal dashboards, and data tools. Reporting supposedly simple metrics such as revenue can become very tricky due to multiple definitions and lack of trust in the data. This fragmented system leads to team disagreements about definitions, ownership, and accuracy of metrics. A lot of manual work is required to settle disagreements and align stakeholders.

This article summarises insights gathered during a virtual discussion involving

Allegra Holland, GTM strategist @Transform. Transform is a metrics store amazing at creating companywide alignment around key metrics through its tool. The organization’s founding team met at Airbnb. They decided to launch Transform after witnessing how Airbnb’s internal metrics store transformed the way teams made decisions and leveraged metrics.

Zdenek, product manager @Gooddata. GoodData is an embedded BI and analytics platform that seeks to help organizations build analytics for their end-users. GoodData provides self-service analytics, low-code/no-code interfaces, and embeddable data visualization.

Burak, founder @MetriQL.  MetriQL is an open-source project that lets you define your metrics as code in a central metric store. MetriQL was acquired by Liveramp, a platform for data connectivity that seeks to enable companies to better connect, control, and activate data to improve customer experience.

Together with these three experts, we discussed the origin of the metrics inconsistency issue, what a successful solution should look like, and the role of metrics stores in solving the problem.

‍

I - Framing the metrics consistency issue

Why is consistency over metrics so hard to maintain?

                 The roots of inconsistency over metrics

• Metrics definition is challenging: It’s hard to define KPIs that matter to the business and come to an agreement on how they should be quantified.  The process of defining metrics as a group with both technical and non-technical people in the room is challenging and most of the time doesn't happen. Both groups have a hard time conveying the right context, with a lot of things getting lost in translation.
• Data pipelines are complex: There are lots of steps in the data pipelines and a lot of people bear responsibility for ingestion, transformation, and visualization of data. Data people mostly fail to understand how their work will be used downstream for business purposes. If you’re lucky you end up with a nicely prepared dataset handed over to your sales team. The issue is, this team doesn't understand what they’re getting because they don’t operate in the world of datasets and data structures at large. These datasets end up not getting used at all.
• The modern data stack is fragmented: There are different ways to define metrics in different tools. There’s no consistent way of defining and accessing metrics. Tableau and other BI tools provide interfaces for metric definition, but these tools don’t resonate with technical profiles. Conversely, technical profiles define metrics in their own tools, often not enriching definitions with the right context for domain experts to understand them.

Is the issue of consistency over metrics becoming more salient?

Cloud data warehouses provide a unique source of truth for data, making the metrics consistency issue less worrisome to some extent, as people are all working with the same data. However, the data democratization trend means that data is consumed by many more people now, and we’ve also witnessed an explosion of tooling in the modern data stack. There are too many tools, using too many languages to define the metrics. This creates complete chaos. This is only going to get worse in the future as tools continue to multiply, and more people start using data.

‍

II- Identifying a solution to the inconsistency issue

People, processes or tools? what matters the most?

• Burak: people. We need to build something that will give business people the confidence to analyze the data. Domain experts want to rely on metrics to be correct and reliable, but they should be patient enough to conduct some analysis themselves. We need to have the right tooling and workflows to allow for this to happen.
• Allegra: We need to extend the confidence of business people to feel they’re comfortable enough to define metrics for themselves. The solution might lie in treating metrics like data products. Metrics are what both data teams and business teams care about. Essentially, we need to decouple the metrics abstraction from the downstream business component. Context needs to be shared functionally for any company to deliver useful metrics. This needs to be done in an iterative process, in an ongoing way. People on the technical and business side must have an ongoing relationship. Once these people AND institutional processes are set in stone, then the tooling can come in to provide the context that everyone requires (version control for example). Most importantly, its people and processes, and the tooling provides the appropriate structure for all this to ensure the right processes are in place to deliver actionable insights.
• Zdenek: It’s better to look at things in terms of what we can change more easily in organizations. It’s ultimately hard to change the way that people behave. It’s impossible to impose new processes and change the mindset of people in organizations. The most important aspect is the tooling and how we expose metrics. We need the right tooling to ensure everyone in the company can understand and access business definitions. Metrics need to be defined in multiple layers, and you need to have some kind of lineage to understand how they change over time. We need to allow people to consume these metrics from their tool, without having to download, export, and analyze data. When they have to do so, it is the beginning of inconsistency in your metrics. In short, having the right tooling is much easier than defining normalized processes and changing people’s behavior.

III - How does the metrics store help?

‍

‍Which technological changes brought about the need for metrics stores?

Zdenek: The main issue is consistency over metrics. In businesses, where you try to track the performance of a metric, such as revenue, you get a lot of different answers from different departments. Metrics stores were born out of the need to align all the customers to see the same numbers and be aligned on business decisions.

Burak: Metrics stores emerged following the growing complexity of the data infrastructure. Let’s take the example of revenue. Historically, things were easier because the revenue stream was coming from a single channel. Right now, businesses contact customers through a lot of different channels. So to be able to calculate revenue, we need to gather data from many different applications. It’s not easy to build up metrics in this context. You need to take into account the multiple downstream data sources and store all these data in a single place to be able to calculate it. That’s why we have the data warehouse. The second reason is: you want to be consuming metrics from multiple tools. That’s the whole idea of headless BI. Now, in a single organization, there are different persona who need to interact with these metrics. All these departments prefer different tools. We’re going in this decentralization era in the B2B tools as well because we want to use the right tools for the right job. All these tools should have access to the metrics in a convenient way. That’s why we have the metrics stores, where you can define everything at once and then talk to the downstream tools for different use cases.

‍

‍What are the main use cases of metrics stores?

Zdenek: The main use case for the metrics store is metrics consumption. People in the organization need to be able to consume metrics from a consistent place and rely on the fact that the metric is defined consistently itself. By, this, we mean the expression computing the metric is the same everywhere. Another use case is discovery. A metrics store makes it easier to find the metrics and the documentation around them. It makes it possible for people to drill down how these metrics were computed exactly. Finally, metrics stores are useful for change management. If something changes in the way a metric is computed, people need to be notified because it will affect their work.

Allegra: Key KPIs drive organizational alignment across the board. An important use case for the MS is that it allows specific departments to share the metrics that other departments may care about and that everyone should align on. There’s also an exploratory use case. Metrics stores help people understand the business by understanding the metrics which the business cares about. When someone recently joined a company, it’s useful for them to understand how their company thinks about themselves. To this end, it’s interesting to do some exploratory understanding of what metrics are and how they extend to different operational workflows. It’s the one-stop-shop to see how the business thinks about itself and how performance changes over time.

‍

‍What are the main benefits of a metrics store?

‍Consistency: metrics stores bring about a consensus on how metrics are defined and thus more alignment across the board.

Productivity: When a metric is properly quantified, it can simply be filtered out by different criteria so that analysts don’t have to write another SQL query but just switch a “where” clause to pull in one other field. This allows analysts to offload the work to business people who care about specific business use cases. Now the analyst has time freed up to spend time on more value-generating analysis rather than just re-write SQL queries.

‍

How do metrics store fit in the modern data stack?

The metrics store is essentially the layer between the database and the data consumption tools. For the moment, we mostly used dbt to define our metrics and use the downstream tools (BI tools) to analyze them. But the downstream tools just see database tables. The metrics store is the layer between the transformation tool and the downstream tools, allowing us to expose all these metrics to downstream tools in a different form than just tables and columns. It allows presenting metrics in terms of numbers, which are much more consumable than queries. That’s because queries are still foreign to quite a lot of people, while numbers are much more familiar.

‍

‍How do metrics store and data catalogs interact?

Burak: Data catalogs are aimed at people that know how databases/ columns work. The metrics stores allow the data catalog to be elevated one level up by making metrics consumable by regular people, who don’t know about data structures.

Allegra: Data catalogs are sort of the raw data assets in the development environment for data where technical persons can go and understand the lineage of data, where a piece of data comes from, etc.. Exposing that lineage and passing that through to the metrics store allows end-users to feel more comfortable about how these metrics are defined. The metrics store is somehow the productionalization of the data that gets served to the business so that the business can make decisions based on that data.

‍

How do data teams use metrics stores in practice?

‍A lot of different personas use the metrics store. Technical folks define metrics, providing the necessary context around them to ensure trust in how those metrics are going to be used downstream. The work is then offloaded to business users, who can go to the metrics store and search the metrics catalog to find particular metrics and filter them for their particular business use cases. Metrics stores reconcile business and technical users, with technical users defining and giving context around metrics while business users consume them. they consume without altering the definition for their business reporting needs.

‍

Subscribe to the Castor Blog

About us

We write about all the processes involved when leveraging data assets: from the modern data stack to data teams composition, to data governance. Our blog covers the technical and the less technical aspects of creating tangible value from data.

At Castor, we are building a data documentation tool for the Notion, Figma, Slack generation. We designed our catalog software to be easy to use, delightful and friendly.

Want to check it out? Give the CastorDoc software a test drive with our free 14-day demo.

‍