Understanding the API Run Page

When a data source runs, One Model moves through four stages to get data from the API into your tables: extraction, where the connector calls the API endpoints and retrieves the response data; consolidation, where large or multi-task extractions are combined into a unified table structure; ingestion, where the staged data is loaded into tables; and finally, a data load is created, making the data available to the processing script and to query in SQL Explorer. The API Runs page and API Run Details page let you track the progress and status of each of these stages in real time. 

Understanding the API Runs Page

The API Runs page contains a list of all API runs and their progress with retrieving data from an API. You can access this page in two ways:

1. Data > Sources page: Click View API Runs. Note: only connectors that are APIs will display this option.
2. Data Loads page: Click the i icon on any API data load that is in a running state.

On the API Runs page you can:

1. Filter the API runs by Data Source, Status, or Type.
2. View quick details about the status of the run, including its extraction type and when it started or completed (if applicable).
3. Sort (ascending or descending) the API runs.
4. View further details of the run, or retry the run.

Understanding the API Run Details Page

The API Run Details page contains a list of all tasks for an API run and their status. On this page you can:

1. View quick details of the individual API run.
2. Search for tasks within the API run. The Search function will return results for:
   1. Acronym match on task name (where any underscore or space denotes a word boundary)
   2. Task name containing the search string
   3. Status that starts with the search string (for example, typing waiting instead of waiting to start to see matches)
   4. Table names that contain the search string
   5. Messages that contain the search string
3. Sort (ascending or descending) the API run tasks.
4. View the API run details, which includes a paginated list of run tasks. The number of tasks per page can be adjusted at the bottom of the list.

How an API Run Works

An API run typically involves several stages orchestrated by various services. The tasks for these stages can be viewed in the API Run Details page.

Extraction

This stage retrieves data from the API. The API Processor is the central orchestrator, initiating the data pull from the API endpoints. Once extracted, the data is temporarily stored as objects using Amazon S3. The API Processor generates and saves delimited file inputs and creates manifest files: system files that record which S3 objects should be loaded into a table, acting as an instruction list for the consolidation and ingestion stages. Extraction tasks can be identified by the label Process [Endpoint Name].

Consolidation

An API response with a large volume of data may be extracted by multiple tasks, with each task handling a portion of the total data. Note: consolidation tasks are not applicable for all data sources. These portions of data are then consolidated into tables using two sub-stages:

• Metadata Consolidation combines the table structures, including schema and column definitions, from each extraction task to create a final, comprehensive table structure that acts as a blueprint for all incoming data. Metadata Consolidation tasks can be identified by the label Metadata Consolidation [].
• Table Consolidation uses the Metadata Consolidation blueprint as a guide to align and merge the actual data from each individual task into this unified structure. Table Consolidation tasks can be identified by the label [Endpoint] Consolidation.

Think of Metadata Consolidation as defining the columns for a spreadsheet, and Table Consolidation as filling in the rows.

Ingestion

The final stage is Redshift Ingestion, where the staged data is loaded from the S3 objects into Redshift tables. First, the data is loaded into temporary tables, then a Data Load is created. Once loaded, these tables can be queried via SQL Explorer by users with the appropriate permissions.

View API Run Statuses

The status of an API run is tracked as it progresses.

Find the API Run ID

The API run ID distinguishes between API runs and may be required when submitting support cases. It can be found in the address bar of your browser when viewing the run. In this example, the API run ID is 27348.

Determine the API Run Time

Understanding how long incremental or destructive data runs typically take helps you identify when a process is running unusually long. The completion time of the last API run is also useful when deciding when to start another run - for example, you may want to be selective about timing a destructive run if a similar run has historically taken a lengthy period.

Use the Type filter to find this information. Use Started At or Completed At to compare how long destructive versus incremental API runs typically last.

Retry an API Run

You may retry an API run if it failed or was cancelled. Retry is only available on the most recent API run. We recommend submitting a support ticket that includes the API run ID before attempting a retry.

Retry can be accessed from:

1. The individual API Run Details page.
2. The API Runs page.

Find Failed API Runs

The Status column on the API Runs page displays the status of each API run. Click the Status column header to sort the list by status, or use the Status filter to display only failed runs. Click View to go to the run details page.

Clicking on the number in the Message column provides additional detail about why the failure occurred. If a failure occurs, submit a support ticket that includes the API run ID.

Find Table Names for an API Run

The API Run Details page includes a Tables column that displays the name and record count of any tables created by a task (where applicable).