How to Create Data Lineage With the Tableau GraphQL Metadata API
Tableau has a rich metadata API exposed through a GraphQL interface that you can use to extract your data lineage.
Join the DZone community and get the full member experience.Join For Free
I love data. The ways it can be used to curate value and express relationships never ceases to amaze me. To this extent, visualizing data is often one of the most powerful ways to share insights and Tableau certainly is one of - if not the - most popular data visualization tools on the market. It's extremely simple for non-technical users to develop rich and meaningful graphs with a pretty intuitive UI and there are some really nice features under the hood that are used to speed up query performance when extracts are stored within Tableau.
My absolute favorite Tableau feature is that you can query your metadata using the same GraphQL API that Tableau itself uses. A portion of the metadata exposed includes the lineage for the fields, sheets, tables and data stores that exist within your Tableau Site. Exposing the metadata via an extensive API like this is a really forward thinking idea from the team behind Tableau.
How to Use the Tableau Metadata API
The Tableau Metadata API is exposed via GraphQL and Tableau is wrapped in a python library, the Tableau Server Client. This library is one of the easiest APIs to use - Tableau has simplified all authentication and serialization to allow users to just focus on the query they want to execute.
- The graph enables many different entities and data assets within Tableau to be queried
- The API performance is really good, even when requesting a large number of multidimensional relationships
- The Python client is extremely simple and intuitive to use, handling the authentication and serialization for the user
- The documentation is sparse - it's not clear when to expect upstream or downstream data lineage assets to be provided or when they will be null
- A "full" lineage for each data asset is not available, you can only extract lineage from one step upstream or one step downstream (at least from what I could tell from using the API)
- Tableau releases a new API version every quarter or so but the docs do not depict which features are available in which version
Let's look at some code that can be used to query your Tableau metadata.
You can use the Tableau API by authenticating with your username and password but the more secure and suggested approach is to use a client token. I've also created a simple helper function below to authenticate and execute queries.
Define the Query
The Tableau Metadata API is a fantastic way to start learning GraphQL since Tableau handles all of the serialization for you and their Graph follows a consistent and easy to understand set of conventions.
The function below executes a query that will return all calculated fields that exist within your Site. The beautiful thing here with GraphQL is that we can simultaneously ask Tableau to return all of the fields that reference each of the calculated fields and we can go even deeper to request all of the sheets for each field that is referencing a calculated field.
Create Your Data Lineage
Now that you have your metadata from Tableau, how you structure and use the output is completely up to you. This example will define edges and nodes. These are the fundamental building blocks for network relationships and data lineage.
View the Edges and Nodes
Running all of the functions above will now result in creating the objects required to visualize your data lineage. The nodes and edges can be plugged into just about any network visualization tool, such as NetworkX, to view the output.
I applaud Tableau for enabling this form of data access even though I believe this is a highly underutilized and under-leveraged benefit. Many companies do not fully make use of this metadata from Tableau to the full extent. Understanding how data moves and dependencies between data is such a critical feature especially as organizations try to maintain well-managed practices and controls around how their data is used. As you look to leverage Tableau metadata and data lineage within your company, make sure that you're taking the extra step to connect that data lineage with the upstream processes to give a complete and comprehensive perspective of your lineage.
Published at DZone with permission of Grant Seward. See the original article here.
Opinions expressed by DZone contributors are their own.