Exploring Databricks Genie: Conversational Analytics with Unity Catalog

Databricks Genie introduces a new model for data platforms by enabling conversational analytics directly within the Databricks ecosystem. Instead of starting with SQL, users can start with a question in natural language format. Genie interprets natural language prompts, generates SQL queries, executes them against governed datasets, and returns results instantly.

One important point to note is, conversational analytics works best when it operates on well-governed and well-documented datasets. This is where Unity Catalog becomes critical in Databricks. If you are new to Unity Catalog, I recommend reviewing the earlier DZone article on Databricks governance and Unity Catalog. 

In this article, we will build on that foundation catalog and demonstrate how Databricks Genie can leverage datasets registered in Unity Catalog to enable conversational analytics. We will walk through how Genie works and explore practical scenarios where conversational agents can help users query data, generate SQL, explore features of Databricks Genie.

How Genie Works?

Users such as analysts, engineers, and data scientists can ask questions in plain English using the Genie interface. Genie reads the question and understands what the user is asking. It then creates the correct SQL query automatically. To make sure the results are accurate and secure, Genie uses information stored in Unity Catalog. Unity Catalog keeps important metadata such as table definitions, column descriptions, data lineage, and access permissions.

Using this information, Genie knows which datasets to query in the Databricks Lakehouse. These datasets may include tables such as sales, inventory, or product catalogs.The generated SQL query runs on Databricks SQL and compute. The results are then returned to the user as tables, charts, or insights.This architecture allows users to interact with trusted data using simple questions. They do not need to understand complex schemas or write SQL queries themselves.

Getting Started With Databricks Genie

After understanding how Genie works, the next step is to create a Genie space and connect it to datasets governed by Unity Catalog.A Genie space is the environment where users interact with data through natural language. In this space, you define which datasets Genie can access and provide the context it needs to generate accurate queries.

Open Genie in Databricks

From the Databricks workspace, navigate to the Genie section in the left navigation panel.Click Create Genie Space to create a new conversational workspace.This space will act as the interface where users can ask questions and explore data.Here you can connect your Genie to the datasets/untiy catalog available in your organization.

Exploring Genie using Bakehouse Unity Catalog

For this example, we will use the bakehouse datasets are already registered and governed in Unity Catalog, we can directly use them inside Databricks Genie. Since this metadata is available through Unity Catalog, Genie can better understand how the datasets relate to each other and generate more accurate queries.

In this Genie space, users can now start interacting with the Bakehouse data using natural language. Instead of manually exploring schemas or writing SQL queries, they can simply ask questions about bakery sales, inventory levels, or product performance.

Let’s ask a question like “What is the reason for revenue decline”

Genie responds with an explanation indicating that the observed revenue decline is likely due to incomplete data rather than an actual drop in sales. It also highlights the affected time window where the missing records occur.

This type of conversational analysis helps teams quickly identify whether a business problem is truly operational or simply a data quality issue. Instead of spending hours running queries or investigating dashboards, users can immediately uncover potential data gaps and escalate the issue to the data engineering team.

To go a level further Genie, breaks down the data by products cohorts to confirm the stable growth and gives a chart to compare the missing week data to the earliest available data to confirm its findings. The charts help visualize this pattern clearly. Revenue trends remain consistent before and after the affected period, indicating that the issue is related to data completeness rather than business performance.

Databricks Features, Suggested Questions:

Another helpful feature of Genie is that it automatically suggests questions for users to explore. Genie analyzes the datasets and metadata available in Unity Catalog and generates relevant questions that users might want to ask. For example, in the Bakehouse Genie space, suggested questions might include things like

Show me all products

What recent temporal patterns emerge in this dataset?

Identify interesting outlier entities in the dataset (and potential causes)?

Which product sold the most units?

What was our revenue in May 2024?

I asked one of the question, What recent temporal patterns emerge in this dataset?, it analyzed the dataset and gave insights about the highest sales found in mid-week especially between Wednesday and Thursday.

Configure Feature

In the Configure section of the Genie space, users can manage which datasets Genie is allowed to access. This allows teams to add or remove tables depending on the analysis they want Genie to perform. 

Instructions Feature

In the Instructions section of the Genie space, teams can provide additional context about how data should be interpreted within the organization. This helps Genie generate more accurate queries and explanations. For example, organizations can specify that the financial year may differ from the standard calendar year, define important business terms, or explain how certain metrics are calculated. Providing this type of context ensures that Genie understands the organization’s data conventions and aligns its responses with how the business actually operates

Benchmarks Feature

In the Benchmark section, Genie automatically generates suggested questions and the corresponding SQL queries by analyzing the datasets available in the Genie space. These suggestions can be reviewed by the data team before being used more broadly. Engineers or data stewards can approve or reject the generated queries to ensure they are accurate and aligned with the organization’s data definitions. Once approved, these queries effectively become a trusted query bank that users can reference when asking questions.

Monitoring Feature

Genie also provides monitoring capabilities that allow teams to track the questions being asked in the Genie space. This helps data teams understand how users are interacting with the data and what types of insights they are looking for. By reviewing these questions, engineers and data stewards can identify common queries, improve dataset documentation, and refine the context provided to Genie.

Sharing the Genie Space

Genie also provides options to share the Genie space with users across the organization. Data teams can control who has access and allow analysts, engineers, or business users to interact with the datasets through the conversational interface. In addition, Genie spaces can be marked with statuses such as Certified or Deprecated. A Certified space indicates that the datasets, queries, and context have been reviewed and approved for production use. A Deprecated space signals that the content is outdated or should no longer be used. These controls help maintain trust in the data while ensuring that users interact with the most reliable and up-to-date Genie environments.

Summary

In this article, we explored how Databricks Genie lets people ask questions about their data in simple, natural language. Using the Bakehouse datasets managed in Unity Catalog, we saw how Genie understands a question, creates the SQL query behind the scenes, and shows the results as tables or charts.

We also looked at how to set up and publish a Genie space. This included choosing which datasets Genie can use, adding instructions that explain business context, reviewing suggested queries, monitoring the questions users ask, and sharing the Genie space with teams.

Once the Genie space is published, users across the organization can ask questions and quickly explore the data. Instead of writing SQL or searching through complex tables, they can simply ask a question and get insights to help solve business problems.