Configure Databricks Data Destination

This article explains how to configure a Databricks Data Destination for Databricks running on AWS or Azure. One Model exports your data as files to cloud storage you own (Amazon S3 or Azure Data Lake Storage Gen2), and Databricks then loads those files into tables.

Before you start

You will need:

• A storage location you control: an Amazon S3 bucket (AWS) or an Azure Data Lake Storage Gen2 container (Azure). The storage Databricks creates for its own workspace (the DBFS root or default storage) can't be used. Create a new bucket or container, or use a dedicated folder in an existing landing location.
• Credentials that let One Model write to that location. The S3 and Azure articles linked below explain what's required.
• A Databricks admin who can create a storage credential and external location in Unity Catalog (requires the CREATE EXTERNAL LOCATION privilege), so Databricks can read the exported files.

Create the destination

Go to Data Destinations › + and choose Databricks AWS or Databricks Azure, depending on which cloud platform your Databricks workspace runs on.

Databricks on AWS

For Databricks running on AWS, One Model writes the exported files to an Amazon S3 bucket.

For details on how to configure the S3 destination, refer to Configure a Data Destination - S3 and Redshift

Databricks on Azure

For Databricks running on Azure, One Model writes the exported files to an Azure Data Lake Storage Gen2 container.

For details on how to configure the Data Lake Gen 2 destination, refer to Configure Data Destination - Azure

Load the data into Databricks

One Model writes the exported files to your storage. To get them into Databricks tables, a Databricks admin needs to:

1. Create a storage credential in Unity Catalog for the bucket or container. On AWS this uses an IAM role; on Azure it uses an Azure Databricks access connector (managed identity).
2. Create an external location over the path One Model writes to, using that storage credential.
3. Load the files into tables using COPY INTO, Auto Loader, or an external table.

For example, to load a table with COPY INTO (replace the placeholder names and path):

CREATE TABLE IF NOT EXISTS my_catalog.my_schema.my_table;

COPY INTO my_catalog.my_schema.my_table
FROM 's3://my-bucket/onemodel/my_table/'
FILEFORMAT = PARQUET
COPY_OPTIONS ('mergeSchema' = 'true');

On Azure, use a path in the form abfss://<container>@<storage-account>.dfs.core.windows.net/<path>.

For more detail, see the Databricks documentation on external locations and COPY INTO.

Limitations

Databricks running on Google Cloud (GCP) is not currently supported.