This article explains the difference between destructive and incremental data loads in One Model. It is intended for Data Admins configuring or planning a data source. By the end you will understand how each load type works, when to use each, and where to go to configure them.
When you trigger a data source in One Model, two things happen: the connector runs (retrieving data from the API) and the retrieved data is loaded (processed into tables). Destructive loads and incremental loads apply to both stages - determining what data is collected from the API and how it is written into your tables. A destructive load replaces all existing data with the current API response. An incremental load adds or updates only the records that have changed since the last successful run. Choosing the right approach depends on your data volume, how often your source data changes, and whether your API supports filtering by date.
What is a Destructive Load?
A destructive load (also known as a full load) involves completely replacing the existing data in your People Data Cloud with the new data.
Key Characteristics
- Complete replacement: All existing data in the target table is deleted or overwritten, and the new data is loaded afresh.
- Simpler process: The process is straightforward because it does not need to consider changes or differences between the new and old data.
- Potential for data loss: There is a risk of losing data if the load process fails after the existing data has been deleted but before the new data is completely loaded.
Use Cases
- First run of a newly configured data source connector.
- When the source system doesn't support date-based filtering.
- Small datasets where running a full replacement on a regular schedule is more practical than configuring incremental loads.
- Capturing record deletions for APIs that don't support them.
What is an Incremental Load?
An incremental load (also known as a delta load) involves updating your People Data Cloud by loading only the new or changed data since the last load.
Key Characteristics
- Partial update: Only new, modified, or deleted records are processed and loaded into the target table.
- Efficiency: More efficient in terms of resource consumption (CPU, memory, and disk usage) because it processes only a subset of the total data.
- Frequency: Suitable for frequent updates, especially in environments where data changes regularly but only a portion of the data is affected.
- Potential for data inaccuracy: Some APIs don't track deletes, which can mean that incremental data loads might need to be supplemented by destructive loads to keep data aligned with its source.
Use Cases
- Maintaining your data where only incremental changes need to be applied.
- Frequently scheduled runs where only a small proportion of records change between runs.
- Scenarios with large datasets where destructive loads are impractical due to time or resource constraints.
Comparing Destructive and Incremental Loads
| Aspect | Destructive Load | Incremental Load |
|---|---|---|
| Process | Complete replacement of existing data | Partial update with new/changed data |
| Resource Consumption | High (processes all data) | Low (processes only a subset of data) |
| Data Integrity Risk | Low | Varying (depends on vendor support) |
| Frequency of Use | Less frequent | More frequent |
| Use Cases | Initial loads, full refreshes | Regular updates, real-time sync |
Next Steps
You should now understand the difference between destructive and incremental data loads and when to use each. To configure the load type for your data source:
- For the Universal Connector, see Configuring the Universal Connector - in particular the Parameters, Keys, and Variables sections which are required for incremental loads.
- For source-specific configuration guides, refer to the Incremental/Destructive section of the relevant connector article in this section.
If you are unsure which load type is appropriate for a specific endpoint, contact your One Model Customer Success team.
Comments
0 comments
Please sign in to leave a comment.