Using a Databricks Connection

Once you have set up a Databricks connection, you can use it to bring Databricks data into Nexadata (Datasets) or send transformed data back to Databricks (Outputs). This article walks through each option step by step.

Here are the key capabilities:

• Create datasets from tables and views in an OpenSharing share, chosen from lists of what the share contains

• Create datasets from a SQL statement that runs on a SQL warehouse you choose, so you can filter, join, and aggregate before the data moves

• Write Workflow outputs to a Databricks table with Append or Upsert

• Optionally create the target table and allow new columns as your outputs evolve

• Every written row is stamped with the Nexadata run ID in a nexadata_run_id column, so you can trace any row back to the run that wrote it

---

Part 1: Creating a Dataset from a Databricks Share

Use this option with a connection that uses an OpenSharing Profile.

Step 1: Create a New Dataset

1. Navigate to the Datasets section in Nexadata.

2. Click Create Dataset.

3. Enter a unique Name for the dataset.

4. Under Data Connection, select your Databricks OpenSharing connection.

5. Under Data Format, leave Tabular selected.

Step 2: Select the Share, Schema, and Table

1. Select the Share. Start typing to search the shares the profile can access.

2. Select the Schema within that share.

3. Select the Table. Shared tables and shared views both appear here, and both work the same way.

Step 3: Submit and Define Columns

1. Click Submit. Nexadata reads the table from the share.

2. On Define Columns, review the columns and save, as for any dataset. See How to Create a New Nexadata Dataset. The dataset is ready to use in Workflows, and a full run reads every record.

Note

Databricks computes a shared view on its side before returning it, so the whole view is materialized each time Nexadata reads it. Reads of views take longer than reads of tables of the same size.

---

Part 2: Creating a Dataset from a Databricks SQL Statement

Use this option with a connection that uses Service Principal or App Connection (User Sign-In).

Step 1: Create a New Dataset

1. Navigate to the Datasets section in Nexadata.

2. Click Create Dataset.

3. Enter a unique Name for the dataset.

4. Under Data Connection, select your Databricks connection.

5. Under Data Format, leave Tabular selected.

Step 2: Choose a SQL Warehouse

Under SQL Warehouse, start typing to search, then select the warehouse the statement should run on. The list shows the warehouses the connection has Can use on.

Step 3: Write Your SQL Statement

Enter the statement that defines the dataset in SQL Statement. Use fully qualified table names in the form catalog.schema.table, since no catalog or schema is preselected. For example:

SELECT region, product_line, fiscal_month, SUM(net_revenue) AS net_revenue
FROM finance.actuals.gl_revenue
WHERE fiscal_year = 2026
GROUP BY region, product_line, fiscal_month

If a catalog, schema, or table name contains a hyphen, wrap it in backticks, for example `nexadata-demo-data`.default.revenue.

The connection needs SELECT on every table or view the statement reads. See Setting up a Databricks Connection.

Note

Planning models need flat columns. If a table has ARRAY, MAP, or STRUCT columns, select the individual fields you need, or convert the column to text in the statement, for example to_json(attributes) AS attributes.

Step 4: Submit and Define Columns

1. Click Submit. Nexadata runs the statement on the warehouse. If the warehouse is stopped, Databricks starts it first, so this can take a minute or two.

2. On Define Columns, review the columns the statement returned and save, as for any dataset. See How to Create a New Nexadata Dataset.

---

Part 3: Sending Data Back to Databricks

Use this option with a connection that uses Service Principal or App Connection (User Sign-In). Nexadata stages the Workflow output as a CSV file in a Unity Catalog volume, loads it into the target table, then deletes the staged file.

Step 1: Add an Output

1. Open the Workflow that produces the data you want to write, and go to its Outputs tab. See Workflow Outputs.

2. Add an output and enter a Name for it.

3. Click Customize Outputs, then under Data Connection to use, select your Databricks connection. The Databricks settings appear below.

Step 2: Choose the Warehouse, Target Table, and Staging Volume

• SQL Warehouse: The warehouse that runs the load.

• Target Table: The table to write to, as catalog.schema.table, for example finance.planning.revenue_allocations.

• Staging Volume (optional): The Unity Catalog volume to stage files in. Leave it blank and Nexadata uses a volume named nexadata_staging in the target table’s schema, creating it if needed.

Step 3: Choose a Write Mode

• Append (add the rows to the table): Adds every row to the table on each run.

• Upsert (update rows matching the business key, insert the rest): Under Business Key Columns, select the columns that together identify a row, for example region, product_line, and fiscal_month. Rows that match the table on all of these columns are updated; the rest are inserted.

Note

Choose business key columns that identify each row uniquely, and leave out amounts and other values that change. If an amount is part of the key, a changed amount no longer matches its row, so Upsert inserts a second row instead of updating the first. If two rows in the Workflow output share the same key, the run fails, because Databricks cannot tell which of them should update the table.

Step 4: Set Optional Behaviors

Both settings are off by default.

• Create the table if it doesn’t exist: Nexadata creates the target table with the output’s column types before the first load. Enter a Read Access Group: this Databricks group is granted SELECT on the new table. The group must be an account-level group; enter account users to let every user in your Databricks account read the table. Without this grant, only the service principal can read the table.

• Allow new columns: If the Workflow output gains a column, Nexadata adds it to the table instead of failing the run.

A completed output, upserting into finance.planning.beverage_sales and creating the table on the first run, looks like this:

Step 5: Save and Run

Save the output and run the Workflow. Nexadata writes the results to the table according to your settings.

Note

Nexadata does not check the target table or your privileges when you save the output. Permission or table problems surface when the Workflow runs, with an error that names the missing permission and, where it can, the exact SQL statement your data team can run to fix it. See Troubleshooting the Databricks Connector.

---

How Nexadata Writes to Databricks

• Explicit column types. Every column is written at the type defined in the Nexadata output rather than one Databricks guesses from the file. Money is written as DECIMAL, so totals in Databricks tie to the source to the cent.

• Run traceability. Nexadata adds a nexadata_run_id column to the target table if it is not already there, and stamps every row it writes with the ID of the Workflow run. With Upsert, the run ID is updated on the rows that change.

  

• Safe reruns. Rerunning a failed or partly completed write does not create duplicate rows. Each run stages a uniquely named file, and Databricks skips files it has already loaded.

• Upsert staging. An upsert loads the run into a staging table named <table>_nexadata_staging in the target schema, replaced on every run, then merges it into the target on the business key. The staging table stays in the schema between runs; it is not a copy of the target, so do not query or build on it.

---

Tips and Best Practices

• Use OpenSharing for read-only feeds. If a team only needs to read curated tables, a share is the narrowest access to approve.

• Push work to the warehouse. With a SQL statement, filter and aggregate in the statement so only the rows you need move into Nexadata.

• Use Upsert for recurring loads. Plan versions and allocations that are restated each cycle should upsert on a business key rather than append.

• Run once against a test schema. Since table and permission problems surface at run time, point a new output at a sandbox schema first.

Next: Try a round trip between Databricks and Nexadata