Dataflow Architecture in One Model

Dataflow in One Model uses a data pipeline model where data is ingested from several different data sources, held behind a firewall, cleaned, validated, transformed, and processed to be made available through the query engine or sent to a data destination. The diagram below illustrates how data flows into, through, and out of One Model.

* For One AI Assistant - click here to view your Dataflow Architecture. 

The left side of the diagram illustrates how raw data flows into One Model using SFTP, API connectors or flat file uploads, moved into a data source and held in the S3 storage bucket behind a firewall. 

The right side shows how that data is transferred into the database loader before being sent to Redshift for transformation and processing. From there, the processed data is available in the Redshift (One Schema) and can be sent to a destination or made available to One Model users through Explore and Storyboards.  

Key features include:

• The ability to ingest data from a wide variety of sources using different tools.
  • This is supported by security authentication, ensuring the source of incoming data is authenticated.
• Uses the Amazon S3 Storage Service to hold raw data behind a firewall.
  • All data-at-rest is encrypted with AES256 at a minimum.
  • All data-in-transit is encrypted with TLS1.2 at a minimum.
• Uses the Amazon Redshift data warehouse cloud. 
  • Access controls for Amazon Redshift are highly regulated, access is ephemeral and reviewed.
• SQL processing scripts to transform the data.
  • SQL scripts are developed and maintained to ensure vulnerabilities are not present.
• Caching to keep your most frequently used metrics at your fingertips. 
  • Cached data is encrypted and gated behind access controls.
• Query engine to relay the data through Explore and Storyboards.
  • Site Usage Activity is collected and provides information on logins and site content views and activity.
• Separation between the end-user and the raw data ingested by One Model. 
  • Data isolation is enforced via data segmentation, access controls, data encryption, and monitoring.
• The ability to send your transformed and raw data out to different data destinations. 
  • Data-in-transit is encrypted with TLS1.2, at a minimum.
• The power to control who can see or access your data. 
  • Supported by access management.