Data Catalog and Reverse ETL: Enabling Data Activation at Scale

I. Data Activation: What is it and why should we care?

Data Activation has become an essential priority for companies that rely on data to drive their business. It is the best way for data-driven companies to maximize the ROI of their data.

Data Activation is an approach promoted by Hightouch, consisting of making data accessible to business teams, for operational use cases such as sales, customer success, finance, or marketing.

The concept of Data Activation is very close to the idea of data democratization. It is an initiative in which business teams are given access to data, regardless of their technical background, and are educated on how to leverage it. This initiative is achieved through building a shared understanding between teams based on trusted, accessible data from your warehouse.

Data activation is an alternative to the more classical approach of only centralized data teams using data stored in the warehouse for reporting and Business Intelligence purposes.

Instead of using data to influence just the longer-term strategy, Data Activation informs strategy for the day-to-day operations of the business. To put it simply, it’s putting the company’s data to work in real-time so everyone in the organization can make smarter decisions more efficiently.

This approach creates more value than reporting alone - it can now help with active marketing campaigns, lead generation, customer support, finance, and sales efforts.

Enabling Data Activation requires business teams to access meaningful, trusted data. Reverse ETL and Data Catalogs both play a role in bringing this about, and their interplay unlocks compounding levels of data democratization.

This guide will introduce you to the concept of Data Activation and explain how it can be implemented through Reverse ETL and Data Catalogs.

II. Overcoming Obstacles to Unlock the Power of Data Activation

There are four barriers that block business teams from accessing data:

1. The data is stored in the warehouse. Non-technical teams can't access or query the warehouse because they aren't familiar with SQL.
2. The data warehouse is complex, ever-changing, and it’s often difficult to understand what’s going on in it, especially for non-technical employees. This complexity is driven by:
3. Table proliferation: The data warehouse is constantly updated with fresh data from various sources. Thanks to dbt, data transformation and modeling have become much more accessible. Tools like dbt allow data engineers to easily create new tables for ad-hoc use cases; however, this explosion in table creation can increase the load on the warehouse and may lead to additional chaos.
4. Poor documentation: In many organizations, the data sitting in the warehouse is not well-documented. This makes it near impossible for non-technical people to accurately locate or understand the tables they need for their use case.

Yet, Data Activation requires that people in the organization can both access and use the data.

This is where Reverse ETL and Data Catalogs come into play. Reverse ETL provides access to the data while a Data Catalog provides the trust and usage part. Both approaches focus on resolving the challenge of leveraging untapped data sitting in your warehouse. We’ll explain how these tools can work in tandem to enable data democratization.

Reverse ETL

Companies have been trying to activate their data for years, but in the past, moving data out of the warehouse required you to either manually download/upload CSV files or build and maintain custom pipelines to every single one of your SaaS applications and end systems. Neither option is scalable.

Reverse ETL is the process of copying data from your central data warehouse to your operational tools, including but not limited to SaaS tools used for growth, marketing, sales, and support.

Instead of reacting to your data as it's persisted into a dashboard, Reverse ETL allows you to take a proactive approach and put it in the hands of your business users to take action.

Reverse ETL creates a hub-and-spoke approach, where the warehouse is your central source of truth, completely eliminating the complex web of pipelines and workflows that come with conventional point-to-point solutions.

Data Cataloging

Modern Data Catalogs, such as Castor, are designed to make it easy for anyone, regardless of their technical expertise, to understand and work with data. By facilitating data discovery within organizations, these catalogs enable people to quickly and accurately locate, comprehend, and utilize data. In this way, Data Catalogs help unlock the full potential of data discovery within an organization.

The features associated with Data Discovery are the following:

Data lineage: Understand, record, and visualize data as it flows from data sources to consumption. Data lineage provides an understanding of the upstream and downstream dependencies of data.

Search: Find data through a powerful search engine-like feature. Locate data assets using the metadata, glossary terms, classifications, and more.

Context: Enrich your data assets with the right context. Allow everyone in the company to understand assets right away. Find information on the table name, owner, purpose, last updates, frequent users, and tags.

Popularity: Automatically assign a popularity score to your data assets. Identify immediately the most popular tables.

Query: Make it easy for everyone to query the data, with or without code. Re-use queries from more experienced data people on your team.

Together, these features enable organizations to quickly and accurately locate, comprehend, and utilize data, regardless of users’ technical expertise.

III - The Data Catalog and Reverse ETL in Tandem: Creating the Virtuous Cycle of Data Activation

Your Data Catalog helps to create context around the data, while Reverse ETL brings the data to the right places. These two processes are complementary.

The relationship between Reverse ETL and Data Cataloging is based on the idea that Reverse ETL excels at moving transformed data to a myriad of destinations, without needing to explore a data warehouse to discover which datasets should be moved. By contrast, Data Cataloging helps you explore a data warehouse and discover which data might be worth moving. This motion is known as Data Discovery.

‍

The interplay of Data Discovery and Reverse ETL creates a virtuous cycle:

1. Create workflow: Data discovery supercharges your ETL tool when it comes to workflow creation. Data discovery helps you find the right data assets to send into downstream SaaS tools. Before sending data from the data warehouse to operational tools, you first need to understand what’s sitting in your data warehouse. For example, let’s say you have a Lead Scoring table in your warehouse, and you would like to send it to Salesforce to make it accessible to your sales teams. Your sales operations team can rely on a Data Discovery tool to find the right lead-scoring table in the data warehouse. This ensures the correct metric is then synced into Salesforce (via Reverse ETL).

2. Explore new workflows: Your Data Catalog also helps uncover other potential Reverse ETL use cases. Let’s say - sales ops is enabling a workflow to move a specific business metric into a downstream tool (e.g syncing lead scores into Salesforce). While exploring the Data Catalog to find the lead score metric, sales ops might unearth a new data point worth activating to business users, for example exporting customer segments calculated on top of the warehouse to Intercom. By assisting in data discovery, your Data Catalog can help you more fully leverage your Reverse ETL tool.

3. Enrich your data discovery tool: The more Reverse ETL workflows you want to deploy, the more warehouse exploration you must undertake. As you do so, your Data Catalog usage will rise and become more important in your company. This will contribute to making data documentation a priority in your company as you'll need to make sure that your Data Catalog can keep up with all of your business-critical Reverse ETL workflows to make sure your business continues operating smoothly.

The combination of Reverse ETL tools and Data Catalogs is like a superpower duo. The more Reverse ETL workflows you create, the more Data Catalog usage increases. The more Data Catalog usage increases, the more Reverse ETL use cases you uncover and the more powerful your organization becomes. This is the virtuous cycle of data activation.

Get Started

Activating your data means putting trustworthy, quality data in the hands of your business teams. Data Catalogs like Castor help with the “trustworthy” part, while Reverse ETL tools like Hightouch makes it easy to sync that data into the right places.

The interplay of both tools is extremely powerful. Your Data Catalog helps you ensure you are migrating the right data from the data warehouse to business tools. In return, Reverse ETL improves your Data Catalog’s usage and the quality of the documentation.

If you want to learn more about how to enable this virtuous cycle for your teams,  get in touch with Hightouch and Castor.

‍

Subscribe to the Castor Blog

‍