This guide explains how to set up the Databricks connector so that One Model can ingest tables from your Databricks workspace. It is intended for the Databricks administrator setting up the export, working alongside your One Model team. By the end you will have a scheduled Databricks job exporting your selected tables to a One Model storage location, ready for One Model to configure the data source and load the data.
Before you start setting up the connector, ensure:
- Your Databricks workspace runs on AWS and has Unity Catalog enabled.
- You have Databricks permissions to create a Storage Credential and an External Location in Unity Catalog (typically a metastore admin, or a user granted CREATE STORAGE CREDENTIAL and CREATE EXTERNAL LOCATION).
- You can create and schedule jobs in the workspace, and have a user or service principal to run the job. We recommend a dedicated service principal rather than a personal login.
- You can run jobs on Serverless compute, or on a cluster with Unity Catalog access.
- You know which tables you want to send to One Model, and the key column(s) that uniquely identify a row in each table.
- Row tracking or change data feed can be turned on for each source table: see Step 5 (Create the sync job). This may need the table owner's agreement.
Additional Resources
- For more detail, see Databricks' guide to creating a storage credential that accesses an AWS S3 bucket.
Securing Sensitive Data
Only the tables you list in the sync job are exported, and every column in those tables is sent. One Model does not typically use or require highly sensitive information such as social security numbers, national identification numbers, passport numbers or bank account details. Before adding a table to the job, check that it doesn't hold columns like these. If it does, export a table that excludes them instead, or discuss the options with your Customer Success team.
How the Databricks Connector Works
The Databricks connector works by export rather than by One Model querying your workspace. You run a One Model sync script as a scheduled job in your own Databricks workspace. Each run writes your selected tables as Parquet files to a dedicated S3 bucket that One Model provides for your company, and One Model loads the data from there.
Databricks reaches the bucket through a Unity Catalog Storage Credential linked to an IAM role that One Model creates for you. The script itself holds no credentials, and no passwords or keys are exchanged between you and One Model.
Setup Overview
Setting up the Databricks connector is shared between you and One Model, and passes back and forth in a chain of handoffs. At two points you'll need to wait for One Model to complete a step and send you the information you need to carry on. One Model's steps are handled entirely on our side and don't need any action from you.
| # | Who | What happens | Handoff |
|---|---|---|---|
| 1 | You | Ask your One Model team to start a Databricks connector setup. See Step 1 (Request the Databricks setup from One Model). | You → One Model |
| 2 | One Model | Creates a dedicated S3 bucket and IAM role for your company. | One Model → You: IAM role ARN, S3 URL and sync script |
| 3 | You | Create a Storage Credential using the IAM role ARN. See Step 2 (Create the storage credential). | You → One Model: External ID |
| 4 | One Model | Adds your External ID to the IAM role. | One Model → You: confirmation to continue |
| 5 | You | Create the External Location, grant permissions, then create, schedule and test the sync job. See Steps 3 (Create the external location) to 6 (Set run as and save the job). | |
| 6 | You | Send One Model your table list, keys and confirmation of the first run. See Step 7 (Send One Model the final details). | You → One Model: tables, keys |
| 7 | One Model | Configures the Databricks data source and begins loading your data. | One Model → You: confirmation to run the job |
| 8 | You | Run the job, turn on the schedule and let One Model know the first run succeeded. See Step 8 (Run the job and turn on the schedule). | You → One Model: first run complete |
Step 1: Request the Databricks Setup from One Model
Contact your One Model Customer Success team to let them know you'd like to set up the Databricks connector. One Model will create a dedicated S3 bucket and IAM role for your company, then send you:
| Value | Example format | Used in |
|---|---|---|
| IAM role ARN | arn:aws:iam::{accountId}:role/om-{yourCompany}-ingestion-bucket-dedicated |
Step 2: Create the Storage Credential |
| S3 URL | s3://{bucketName}/ |
Step 3: Create the External Location |
| Sync script | databricks-source-connector-client-script.py |
Step 5: Create the Sync Job |
Keep these values to hand. You'll need all three to complete the setup.
Step 2: Create the Storage Credential
The Storage Credential lets Unity Catalog use the IAM role One Model created for you.
-
In Databricks, go to Catalog > Connect > Credentials > Create credential:
- Fill in:
- Credential type: AWS IAM Role
- Name: a meaningful label, eg: onemodel
-
IAM role (ARN): the IAM role ARN sent by One Model in Step 1 (Request the Databricks setup)
-
Click Create, then copy the External ID shown in the dialog:
- Send the External ID to your One Model team.
One Model will add your External ID to the IAM role and confirm once it's done. Wait for this confirmation before continuing to Step 3 (Create the external location), as the External Location can't be created until then.
Step 3: Create the External Location
The External Location points Unity Catalog at your One Model S3 bucket, using the Storage Credential from Step 2 (Create the storage credential).
-
Go to Catalog > Connect > External Locations > Create external location:
-
Choose Manual and click Next:
- Fill in:
- Name: a meaningful label, eg: onemodel_export. Make a note of it, as the sync job uses this name.
- Storage type: S3
- URL: the S3 URL sent by One Model in Step 1 (Request the Databricks setup), exactly as sent.
-
Storage credential: the Storage Credential created in Step 2 (Create the storage credential)
- Click Create.
Don't add a folder to the end of the S3 URL. The sync job adds its own folder when it writes the export files.
Step 4: Grant Permissions
Grant the following permissions to the user or service principal that will run the sync job:
| On | Grant |
|---|---|
| The External Location from Step 3 | READ FILES, WRITE FILES |
| The source tables to be exported | USE CATALOG, USE SCHEMA, SELECT |
The schema that will hold the --state-table (Step 5 (Create the sync job))
|
USE CATALOG, USE SCHEMA, CREATE TABLE, MODIFY, SELECT |
The script creates the state table but not its schema. If the schema doesn't exist yet, create it first, for example: CREATE SCHEMA IF NOT EXISTS main.onemodel;
Go to Catalog, External Locations, select the user/service principal, go to the Permissions tab and click Grant. Select the job's user or service principal, then tick READ FILES and WRITE FILES:
Step 5: Create the Sync Job
The sync job runs the One Model script to export your tables. Set up the script first, then create the job using either a single table or several tables.
Prepare the source tables
The script sends only what changed since its last run. For that, it needs to read each source table's change data feed. Set this on each table it syncs:
| Databricks Runtime | Table property |
|---|---|
| 19 or later, with Unity Catalog | delta.enableRowTracking = true |
| Earlier | delta.enableChangeDataFeed = true |
Example:
ALTER TABLE main.hr.employees SET TBLPROPERTIES ('delta.enableRowTracking' =
'true');
We also recommend turning on column mapping ('delta.columnMapping.mode' = 'name'). Without it, any run that spans a schema change, even an added column, sends a full snapshot instead of just the changes. The run after that goes back to sending changes.
- Neither setting applies to changes made before it was turned on, so the first run after enabling one still sends a full snapshot.
- If the change data feed can't be read, the table still syncs, but every run sends a full snapshot and logs a warning saying how to fix it.
- Enabling these properties upgrades the table's Delta protocol. Check with the table's owner before changing it.
Add the Script to the Workspace
- Go to Workspace and open the folder the script will be kept in. We recommend a shared folder rather than a personal one.
- Click Create > File, give it any name ending in
.py(eg:onemodel_sync.py) and paste in the sync script sent by One Model. The file saves automatically. Alternatively, click Import on the folder and upload the.pyfile. - Share the file with the user or service principal the job will run as (Step 4 (Grant permissions)): click Share, add them with Can run, and click Add. Without this, the job fails with "Unable to access…" even when the permissions in Step 4 are correct. If the folder is shared with them instead, the file inherits that access.
Create the Job for a Single Table
-
Go to Jobs & Pipelines > Create > Job and give the job a meaningful name, eg: OneModel sync:
- Fill in the first task:
- Task name: eg: sync_table
- Type: Python script
- Source: Workspace
- Path: the script file added above
-
Compute: Serverless, or a cluster with Unity Catalog access
-
Under Parameters, enter the parameters for the table:
Parameter Value --catalogThe source table's catalog --schemaThe source table's schema --tableThe source table's name --external-locationThe External Location name from Step 3 (Create the external location) --state-tableFull name ( catalog.schema.table) of a table the script uses to keep track of what it has already sent, e.g.main.onemodel.sync_state. The script creates it on its first run, but its schema must already exist (Step 4 (Grant permissions)). It stays in your workspace and is never sent to One Model. Use the same one for every table.--force-snapshotOptional, and takes no value. Sends a full snapshot even if the table has synced before. Later runs go back to sending only changes. Leave it out normally. - Click Create task.
Example:
["--catalog", "main", "--schema", "hr", "--table", "employees", "--external-location",
"onemodel_export", "--state-table", "main.onemodel.sync_state"]
Create the Job for Several Tables
To export more than one table, use a For each task. It runs the script once for each table in a list.
- Go to Jobs & Pipelines > Create > Job and give the job a meaningful name:
-
For the first task, set Type to For each and give it a Task name, eg: sync_tables:
-
Under Inputs, list one entry per table. For example:
[{"catalog": "main", "schema": "hr", "table": "employees"},
{"catalog": "main", "schema": "hr", "table": "jobs"}]
-
Click Add a task to loop over and fill in the task that runs once per table:
- Task name: eg: sync_table
- Type: Python script
- Source: Workspace
- Path: the script file added above
- Compute: Serverless, or a cluster with Unity Catalog access
- Parameters: the same as for a single table, but taking the catalog, schema and table from the current input:
["--catalog", "{{input.catalog}}", "--schema", "{{input.schema}}", "--table",
"{{input.table}}", "--external-location", "onemodel_export",
"--state-table", "main.onemodel.sync_state"]
Tips:
- Export every table in one job. A One Model data load waits until every configured table has arrived, so tables exported by separate jobs make each load wait.
- Use a Python script task, not a notebook. The script reads its parameters from the command line, which a notebook task doesn't pass.
-
Every table can share the same
--state-table. The script adds one row per run, so tables syncing in parallel don't conflict.
Step 6: Set Run As and Save the Job
-
In the job's Job details panel, set Run as to the user or service principal granted permissions in Step 4 (Grant permissions).
- Click Save. Don't run the job or turn on the schedule yet.
--force-snapshot (see Step 8) after One Model confirms the data source is set up.
Step 7: Send One Model the Final Details
Once the first job run has completed, send your One Model team:
- The tables being exported, in the format catalog.schema.table
- The key column(s) that uniquely identify a row in each table
- Which tables each job exports, if you export tables in more than one job
Wait for One Model to confirm the data source is set up before continuing.
Step 8: Run the Job and Turn on the Schedule
- Open the job and click Run now. The first run sends a full snapshot of each table.
- Check the run succeeds, then tell One Model so we can check the data has loaded.
-
Under Schedules & Triggers, click Add trigger > Scheduled and choose how often your One Model data should refresh:
Tips
To resend a table in full, add --force-snapshot to its parameters for one run (use the arrow next to Run now to run it with different parameters), then remove it.
Next Steps
You have now set up your side of the Databricks connector. Each scheduled run of the sync job will export your tables to One Model. You can:
- Monitor your sync job runs from Jobs & Pipelines in Databricks.
- Monitor data loads in One Model as each export arrives. See Data Loads for more information.
- Contact your One Model Customer Success team before adding or removing tables from the job, so the data source can be updated to match.
- Reach out to your One Model Customer Success team with any questions.
Comments
0 comments
Article is closed for comments.