Overview of Data Processing in One Model

One Model delivers people analytics infrastructure. It pulls together the data from all your HR systems and related tools. Then One Model intelligently reorganizes this data and joins it all together, presenting it back as if it all came from a single source.

This article is for those of you who want to know what's going on "under the hood" so to speak. We're going to break that data processing journey up into four sections and provide a video demonstration for each. The article will also mix in links to related content along the way.

In this first video, I'll set up a new flat file data source and upload its data into One Model. To keep things somewhat real-world, I'll use a report extract from our friends over at SmartRecruiters for the example. After you watch this video, you'll know how to add a new data source and how to do a file upload. 

Here are some related links based on the content in that first step:
Adding a File Data Source  

Setting up a Delimited Input

Delimited Input - Date and Timestamp columns

Incremental File Upload

Below in this second video, I'll add in some processing scripting to transform our raw data import into a metric and dimension table that we can hook into the One Model query engine. 

Below, in the third video, Josh adds in the configuration necessary for the One Model application to find our new tables, understand the relationships between them, and display our dimension. 

Finally, in video four below, I'll define a new metric based on our new data source. I'll also use the time dimension and our application state dimension to do a couple different breakouts of that data.

And there you have it folks! One complete trip through the One Model data processing pipeline. It's a basic example, but gives you a first hand view of the steps necessary to ingest, model, and deliver data in One Model. 

Getting your data into One Model

There are a few different options for getting your data into One Model, including APIs, SFTP or a flat file. Watch this video series to learn more about data processing in One Model.

Files uploaded to One Model are tracked on the Data Loads page. To access the Data Loads page, click on Data in the main navigation bar, then select Loads. 

Data loads include, but are not limited to; 

• APIs: 
  • Workday
  • SuccessFactors
  • Greenhouse
  • Oracle
  • Degreed
  • SmartRecruiters
  • …and more…!
• Redshift: 
  • Automation files 
  • Mapping files 
  • Recruiting Data (Additional) 
• SFTP: 
  • Survey files 
  • Custom Reports
  • Data from on-premise systems, e.g. SAP HCM
  • Anything really
• Manual uploads:
  • Flat files
  • Mapping files
  • Learn how to upload a flat file. 

Each data load will have a series of messages indicating the status of the load listed on the Data Loads page. The upload status, completion or error, processing, and cache status are all highlighted through a variety of these status messages. The data upload steps and accompanying status messages are explained in detail in the following table;

Data Loads Explained:

Rejected - Previous load still running

This can happen when an API run or data load has not been completed, and a user starts a new API run. User won’t be able to start a new API until the existing one has been completed.

How to read the Data Loads page 

What follows is an example of what you might expect to see when reviewing the Data Loads page. The illustrated file sources are for example purposes only (and will not match your source data), but the status meanings are consistent with the table above. 

Rejected - Previous load still running

Recommended action: 

• Contact One Model Support.

The dataload widget displays an icon for active, expired, rejected and error statuses, allowing you to monitor the progress of the API run. There will  be a description below the widget about the name of the API run.

Selecting each data load widget or clicking on the will load a series of Messages outlining each event in the data load process. 

• These contain alerts for each change in event the load goes through while being uploaded, validated, processed, and cached to the site. 
• Most recent events are located at the top of each list of events
• To load older event messages, select the Load More button located to the right of the Messages header. 

Uploaded files from each load will be sorted under the File Status Summary column located to the left 

• To view the full list of files, select the Files dropdown
• Files that are successfully loaded will enter the waiting status
• Files that have errors will be sorted separately and labeled as such 

Each data load details;

• The source file
• The number of records in that load
• Valid or Invalid
• The date and time its upload was initiated
• A short summary of the action, which includes the source (API, Redshift, etc.) that the file was uploaded from. 

Click on the file name to reveal details;

Missing a file?

If an expected file is missing from a Data Load, we recommend reviewing the transfer logs for that system to check for loading errors or to see if the file was sent from there as expected.  

Uploading the same file 

You can upload the same file twice, subject to errors, as any full files will completely replace the old data with data in the new file.  

Incremental files are set up with key columns so that if data comes through with values in the key columns that already exist in the dataset, it replaces the old record with the new - as if it's an update. When reloading the same file, this means that it will simply replace the first file's records with the second file's records, as if it's an update, even when no change to the data has actually occurred. It won't duplicate the data in the database. 

Data Loads Alerts Framework

Subscribe to receive an email notification when your data loads have completed (when the metrics are updated) or when there is an error for all or selected data sources. To set up your alerts:

• Ensure that you have the CanConfigureNotifications permission. 
• Click on the ‘User Account’ icon at the top right of the screen and select ‘Preferences’.
• Open the ‘Notifications’ pane, and then click ‘Add New Subscription’. 
• Complete the form and click ‘Save’.

Once you have subscribed to receive an alert notification, an email with your data load details, similar to what is shown below, will be sent to your email address.

If you want to dig into the details, hit the 'Details' button and you will be taken to the Data Loads page in the One Model application.

If you do not receive your subscribed alerts to your inbox, be sure to check your spam / junk folder. Please note that if you proxy as another user, you will not be able to view their preferences, including the data loads alert framework settings.