Universal Connector

Building a Custom Integration

The general prerequisites for building a Custom Integration with any app provider on One Model are:

• The Integration provider must have a public API.
• Have access to the Integration provider’s API documentation.
• If the Integration provider uses OAuth authentication, create an account from the Integration provider.

To start building a Custom Integration, go to the data sources page. You’ll need to complete the following steps:

Step 1. Basic Information

1. Click on the Universal Connector menu

2. Enter a name. This will be appear in the Data Sources page.
    
3. Data Loads Should Process Data checkbox denotes if the data load should trigger processing such as custom SQL, processing script and cache warming after data has been loaded into the database.
    
4. Enable Debug Mode checkbox denotes if the API request and response needs to be uploaded to S3.
    
5. Tick the Restricted Data Checkbox to restrict downloads of the files that are used for that Data Source. The goal is to make it so that any data that is marked as Restricted at the Data Source can't be directly referenced by a customer user anywhere.
    
6. Message Queue Capacity denotes the maximum number of SQS messages that can be processed at a time by the SQS queue of the API service.
    
7. Enter a schema name where you would like to house the connectors’ data.
    
8. Processed To field will be populated with the most recent date that the connector process the data.
    
9. Endpoint base URI consists of Connectors hostname for your organization. You should refer to your connectors documentation to find the right URL to connect to. 

This is an example of a setup for a connector called JIRA.

Step 2. Authentication Method

Pick the Authentication type that your connectors support

1. For Basic Authentication type: You need to enter a username and password setup at your connector.
    
2. For Bearer Token: Bearer authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens. The name “Bearer authentication” can be understood as “give access to the bearer of this token.” The bearer token is a cryptic string, usually generated by the server in response to a login request. Please refer to your connector documentation about how to setup a bearer token.
    
3. For oAuth Client Credential: Enter the client id, secret, and the required scope according to your connector documentation.
    
4. For oAuth static refresh token: Enter the client id, secret, refresh token, and the required scope according to your connector documentation.
    

Step 3. API Setup

Once you have set up these primary connection setup, it is time for you to setup connection to an end point for your connector. An end point refers to a REST API end point in your connector. 

This is how to setup an Endpoint

1. Click on the Plus Icon to add a new Endpoint.
2. Select any end point that this endpoint depends on from the Depend drop down
3. You can click on the rubbish bin icon to delete any end point
4. Enter a meaningful endpoint name. 
5. The Relative URI should be documented in the connector's API documentation.

6. The Root Data Element specifies the highest-level JSON object you want to extract and save. If left blank, the system will try to ingest everything. For example, you could left the root data element blank if you wish to extract both survey and responses. Alternatively you could enter  survey.question to extract only the question portion of the JSON response from an API end point. 
          

   {
   "survey": { 
     "question" : {
        ...
     }
   },
   "responses": { ... }   
   }

    

7. Enter a table name that will house this data in the schema, for example: you may enter surveys as the table name to house this API data.
    
8. Select the paging type for that API. Some API documentation provides a description on how to page results from an API.
   1. The link paging type means: the paging is included as a URL in the API response body or headers
   2. The Query Parameter type means: The paging is included as a cursor in the API response body or headers which will be added as a query parameter in the URL This cursor always points to the next page..
   3. The RFC Link header means: the paging information is included in a special format in the API response headers under the name ‘Link’. (This format is a web standard and is defined here: https://datatracker.ietf.org/doc/html/rfc8288) 
       
9. Expand the transformers drop down, where you will configure a way to get the data from the API endpoint.
    
10. There are 3 types of transformers: 
    1. Translating transformer: A transformer that takes something from somewhere in the response, and puts it somewhere else in the next request. So we could take some value out of the response of the first page and use it to query the second. Or we could take some value out of the header of a response and use it as the URL for the next request.
    2. Output: Take something from somewhere in a given response and output it as a parameter that can be used when templating out dependent tasks. Any task that depends on another task, will receive that dependencies outputs as inputs which can be used in URI templates.
    3. Incrementing: Take something from somewhere and increment it by some amount (up to an optional maximum limit) before making the next request. For example: this incrementing transformer increment the offset parameter in the connector’s query string by 100 

11. Enter the location and path of the next set of pages for the data. It could be in the response body, or in the header.
     
12. Enter the inserter location. This represents where the URI for getting the next set of data from a particular endpoint.
     
13. Terminator.
     
14. The headers section lets you set header values that get sent as part of a request. In this example, the connector will set a user agent header with a value of Mozilla.

This is an example of a setup for an API end point:

Setting up Dependent API Endpoints

Sometimes, an API endpoint needs data from the response of another API endpoint. To configure this dependency, follow these steps:

1. Add an API endpoint.
2. Configure an output transformer.

For instance, we can create an API endpoint named "list projects." The output of this endpoint includes a field called "project_key." This "project_key" can then be used as input for another API endpoint.

To retrieve data using the list issues endpoint, which requires a project_key, the list projects API was selected as a dependency in the "depends on" dropdown. Subsequently, the project_key is utilized within the parameters section.