Batch component

A component that provides an opportunity to collect messages to a batch.

Latest changelog

0.0.6 (November 15, 2020)

Fix MongoError: Authentication failed by authSource
Upgrade to sailor 2.6.18
Update batching-library version
Annual audit of the component code to check if it exposes a sensitive data in the logs
Annual npm vulnerabilities audit

To see the full changelog please use the following link.

Purpose

In some cases, integrator needs a mechanism for collection batches by special criteria. The main purpose is to provide a user with a simple collection and an opportunity to process messages in batches.

How works

You need to split an integration logic into 2 parts. The first one should use the action Add message to batch which starts to collect messages. The second one should start with the trigger Get ready batches.

Flow parts

Environment variables

Please Note: From the platform version 20.51 we deprecated the component LOG_LEVEL environment variable. Now you can control logging level per each step of the flow.

Credentials

Property name	Required	Description	Example
prefix	true	MongoDB URI prefix	`mongo://`; `mongodb+srv://`
host	true	IP address database host.	`10.22.0.5`
port	false	Database port. Mandatory if selected `mongo://` prefix	`27017`
dbName	true	Specifies which database to connect to	`batchDB`
login	true	Username for authentication	`admin`
password	true	Password for authentication	`adminPassw`
Auth Source	false	Authentication DB	`SomeDB`

Batch credentials

Triggers

Get ready batches

Emits all batches that are ready to be processed.

Get ready batches

Input fields description

Input field	Required	Description	Example
Max time in ms before Batch is ready	false	Timeout in millisecond after batch creation, which makes batch ready for processing. Defaults to `60000 ms`	3000
Max records number in Batch	false	Maximum count of items in batch which makes batch ready for processing. Defaults to `100`	10
Max size of Batch in bytes	false	Maximum bytes size of items in batch which makes batch ready for processing `1000000(1 MB)`	500000
Correlation Id	true	Correlation Id between action and trigger. Correlation Id helps identify what collection of batches should be used	Flow1
Delete Batch After Retrieval	false	If checked, the batch will be deleted after emit	false

Important: Use the same configuration in action for correct batch processing

Actions

Add message to batch

Stores a message in a batch and emits a created/updated batch with a processed item only.

Add message to batch

Input fields description

Input field	Required	Description	Example
Max time in ms before Batch is ready	false	Timeout in millisecond after batch creation, which makes batch ready for processing. Defaults to `60000 ms`	3000
Max records number in Batch	false	Maximum count of items in batch which makes batch ready for processing. Defaults to `100`	10
Max size of Batch in bytes	false	Maximum bytes size of items in batch which makes batch ready for processing `1000000(1 MB)`	500000
Correlation Id	true	Correlation Id between action and trigger. Correlation Id helps identify what collection of batches should be used	Flow1

Important: Use the same configuration in trigger for correct batch processing

Additional info

Please be aware that the component is not responsible for maintaining batch data in the database.

Limitations

Version: 1.0.0

Currently only Mongo database storage for batches available. Supported Mongo database versions: 4.0 and higher stable releases.
Retry logic is not implemented.
Emit rate logic is not implemented.
The library does not guarantee a sequence of batch items.

Get your free 14-day elastic.io trial account to use this component.