Skip to content
Nexadata Nexadata Docs

Troubleshooting the Databricks Connector

If you encounter issues with your Databricks connection in Nexadata, this guide will help you resolve the most common ones. Databricks returns structured error messages, such as [TABLE_OR_VIEW_NOT_FOUND], and Nexadata shows them as they are. For write-back permission errors, Nexadata names the missing permission and shows the SQL statement your data team can run to grant it. Most fixes are a privilege your Databricks admin can grant.

This guide is organized into four parts:

  1. Connecting to Databricks

  2. Creating a Dataset from a Share

  3. Creating a Dataset from a SQL Statement

  4. Sending Data Back to Databricks

Part 1: Issues with Connecting to Databricks

Section titled “Part 1: Issues with Connecting to Databricks”

  • Verify Credentials fails for a Service Principal

    • Cause: The Base URL, Client ID, or Client Secret is wrong, or the OAuth secret has expired. Secrets have a fixed lifetime of 1 to 730 days.

    • Fix: Confirm the Base URL is your workspace URL, for example https://dbc-a1b2c3d4-e5f6.cloud.databricks.com. If the secret has expired, generate a new one on the service principal and update the connection.

  • Connect to Databricks is unavailable or sign-in fails for an App Connection

    • Cause: The Base URL or Client ID is missing, or the App connection’s redirect URL does not match.

    • Fix: Enter both fields, and check that the App connection in the Databricks account console uses the redirect URL https://app.nexadata.com/databricks/oauth/callback.

  • App Connection asks you to sign in again

    • Cause: Your sign-in has expired. With the default settings on the App connection, this happens after the connection goes unused for 7 days (Refresh token TTL), or 90 days after you signed in, however often it is used (Max session TTL). If single-use refresh tokens is not enabled on the App connection, the sign-in lasts only 7 days even with regular use.

    • Fix: Open the connection in Nexadata and click Connect to Databricks to sign in again. If it happens every week despite regular use, ask your account admin to enable single-use refresh tokens on the App connection.

  • Error: Unable to parse OpenSharing profile

    • Cause: The pasted text is not the complete contents of the config.share file.

    • Fix: Open the file in a text editor, copy everything including the opening and closing braces, and paste it again.

  • Verify Credentials fails for a valid OpenSharing profile

    • Cause: The profile is valid, but its recipient has no access to any share. Verification checks that the profile can list at least one share. This happens when the share wizard was closed before clicking Share data, so the share or its grant was never saved.

    • Fix: In Databricks, open OpenSharing → Recipients, click the Nexadata recipient, and check that your share is listed under Shares. If it is not, click Grant share and choose it. If the share is not offered, create it again. You do not need a new credential file.

  • OpenSharing connection stops working

    • Cause: The recipient token in the profile has expired or been rotated.

    • Fix: Ask your Databricks admin to rotate the recipient’s token, download the new config.share from the new activation link, and paste its contents into the connection.

Part 2: Issues with Creating a Dataset from a Share

Section titled “Part 2: Issues with Creating a Dataset from a Share”

  • Share, schema, or table not in the list

    • Cause: The recipient has not been granted access to the share, or the table has not been added to it.

    • Fix: Ask your Databricks admin to add the table to the share and confirm the Nexadata recipient is granted access to it.

  • Reading a shared view is slow or fails

    • Cause: Databricks computes the whole view each time Nexadata reads it. A very large view can fail on the Databricks side.

    • Fix: Ask the share owner to share a narrower view, or a table that holds the view’s results.

Part 3: Issues with Creating a Dataset from a SQL Statement

Section titled “Part 3: Issues with Creating a Dataset from a SQL Statement”

  • No SQL warehouses to choose from

    • Cause: The service principal, or the signed-in user, does not have Can use on any SQL warehouse.

    • Fix: In Databricks, open the warehouse’s permissions and grant Can use. Workspace entitlements alone are not enough.

  • Error: [TABLE_OR_VIEW_NOT_FOUND]

    • Cause: The table name is wrong, is not fully qualified, or the connection lacks access to it.

    • Fix: Use the full catalog.schema.table name, and confirm the connection has USE CATALOG, USE SCHEMA, and SELECT on the table.

  • Error: [UNRESOLVED_COLUMN.WITH_SUGGESTION]

    • Cause: The statement names a column that does not exist.

    • Fix: The error lists the closest matching column names. Correct the column name in your statement.

  • Creating a dataset or the first run takes longer than usual

    • Cause: The SQL warehouse was stopped, and Databricks is starting it.

    • Fix: No action is needed. Later runs are faster while the warehouse is running.

Part 4: Issues with Sending Data Back to Databricks

Section titled “Part 4: Issues with Sending Data Back to Databricks”

  • Error: Target table must be in catalog.schema.table format

    • Cause: The Target Table is missing its catalog or schema.

    • Fix: Enter the fully qualified name, for example finance.planning.revenue_allocations.

  • Error: Table ... does not exist or is not accessible

    • Cause: Create the table if it doesn’t exist is off, and the table does not exist or the connection cannot see it.

    • Fix: Correct the table name, grant the connection access to the table, or turn on Create the table if it doesn’t exist.

  • Error: missing the CREATE TABLE permission on the target schema

    • Cause: Create the table if it doesn’t exist is on, and the connection cannot create tables in the schema. Nexadata stops before uploading anything.

    • Fix: Grant CREATE TABLE on the schema, or have your data team run the CREATE TABLE statement shown in the error.

  • Upsert fails with a Databricks permission error on <table>_nexadata_staging

    • Cause: Upsert loads each run into a staging table next to the target, and the connection cannot create tables in the schema.

    • Fix: Grant CREATE TABLE on the target schema.

  • Error: missing permission to grant read access

    • Cause: Nexadata created the table but could not grant SELECT to the Read Access Group.

    • Fix: Have your data team run the GRANT SELECT statement shown in the error. The group must be an account-level group, such as account users.

  • Error: missing MODIFY permission, needed to add the nexadata_run_id column

    • Cause: The connection does not have MODIFY on the existing target table.

    • Fix: Grant MODIFY on the table, or have your data team run the ALTER TABLE statement shown in the error.

  • Error: could not write the staged file

    • Cause: The connection cannot write to the staging volume, or the service principal’s OAuth secret was generated without the files scope.

    • Fix: Grant READ VOLUME and WRITE VOLUME on the volume you named, or CREATE VOLUME on the schema so Nexadata can create nexadata_staging. If the privileges are in place, generate a new secret that includes the sql and files scopes.

  • Upsert fails with [DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE]

    • Cause: Two or more rows in the Workflow output have the same values in the Business Key Columns, so Databricks cannot tell which one should update the table.

    • Fix: Add columns to the business key until it identifies each row uniquely, or remove the duplicate rows in the Workflow, for example with a Group By.

  • Upsert adds rows instead of updating them

    • Cause: The business key includes a column whose value changes between runs, such as an amount or a generated ID, so rows no longer match.

    • Fix: Remove that column from Business Key Columns, keeping only the columns that identify a row. To clean up existing duplicates, filter on nexadata_run_id to find the rows each run inserted.

  • Error: [DELTA_FAILED_TO_MERGE_FIELDS]

    • Cause: A column in the Workflow output has a different type from the same column in the table. The error names the column.

    • Fix: Change the column’s type in the Nexadata output to match the table, or change the table.

  • Run fails after the Workflow output gained a column

    • Cause: The table does not have the new column and Allow new columns is off.

    • Fix: Turn on Allow new columns, or add the column to the table in Databricks.

  • Nobody can see the table Nexadata created

    • Cause: Tables created by Nexadata belong to the service principal, and Unity Catalog gives nobody else access, admins included, until SELECT is granted.

    • Fix: Check the Read Access Group on the output, or have the table owner grant SELECT in Databricks.

  • Worried about duplicates after a failed run?

    • Rerunning is safe. Databricks skips any staged file it has already loaded, so a rerun does not insert the same rows twice.

General Best Practices

Section titled “General Best Practices”

  • Start with a test schema. Point your first outputs at a sandbox schema before writing to production tables.

  • Use one service principal per purpose. A dedicated service principal for Nexadata makes its grants easy to review and revoke.

  • Track secret and token expiry. Note when the OAuth secret or the OpenSharing token expires and replace it before scheduled runs depend on it.

  • Use the run ID. Filter on nexadata_run_id to find exactly which rows a given run inserted or updated.

Manage Cookies