Covered in this article
Related pages
Latest Changelog
Version 2.0.1 ()
Delta Detection Component

Delta Detection Component

A component to manage records from a system which does not provide timebased change tracking or webhooks.

Description

Component learn about new records, updates to records and deletions of records from a system which does not provide timebased change tracking or webhooks.

Requirements

The component would not work without the platform native object-storage service. The service is configured as a part of standard installation.

Credentials

External id - field to identify component. Should be unique per maester instance

Actions

Delta Detection

Action helps to learn about new records, updates to records from a system that does not provide time-based change tracking or webhooks.

Common mechanism of action:

image

Example Integration Flow Using This Action:

  • Extracts all records from a system on a daily basis
  • Runs the entire dataset through a delta filter
  • Performs upsert operations in some other system

Example Integration Flow

Hashing logic consider that the following two JSON strings represent two JSON objects that are semantically identical produces the same hash:

{"foo":"bar","bar":"baz"} = {"bar":"baz","foo":"bar"}

Hashes are done with cryptographically secure algorithms and non-reversible.

Expected input metadata

Click to expand metadata schema
  {
    "type": "object",
    "properties": {
      "objectId": {
        "type": ["object", "string"],
        "required": true,
        "title": "Object ID"
      },
      "objectData": {
        "type": ["object", "string", "array"],
        "required": true,
        "title": "Object Data"
      }
    }
  }
  • objectId - set unique identifier for storing object information at external object-storage.
  • objectData - set object data, that component can detect changes.

Please use developer mode in case when objectId or objectData should be another object type (array or object).

Expected output metadata depends on incoming object structure.

Known Limitations

  1. The hash function generates the same hash for different types of quotes: '', "", ` `` `
  2. Pre-fetch count should be equal 1. Correct behavior with pre-fetch > 1 is not guaranteed
  3. One credential cannot be used in more than one step per all platform instances that are connected to one maester installation as it would lead to a collision

Get your free 14-day elastic.io trial account to use this component.