Batch component
General information
Description
Сomponent that provides an opportunity to collect messages to a batch.
The main purpose is to provide a user with a simple collection and an opportunity to process messages in batches.
Integrators would need to split an integration logic into 2 parts. The first one should use the action Add message to batch which starts to collect messages. The second one should start with the Get ready batches trigger.
It uses the Maester service as a package repository, the legacy component versions support using MongoDB (versions: 4.0 and higher stable releases) as a batches repository.
Please be advised that the component does not support external MongoDB as a data store since version 2.0.0 (disabled support for credentials in the component).
Requirements
Maester microservice should be enabled on installation and what would happen if it doesn’t.
Triggers
Get ready batches
Emits all batches that are ready to be processed.
Input fields description
| Input field | Required | Description | Example |
|---|---|---|---|
| Max time in ms before Batch is ready | false | Timeout in millisecond after batch creation, which makes batch ready for processing. Defaults to 60000 ms |
3000 |
| Max records number in Batch | false | Maximum count of items in batch which makes batch ready for processing. Defaults to 100 |
10 |
| Max size of Batch in bytes | false | Maximum bytes size of items in batch which makes batch ready for processing 1000000(1 MB) |
500000 |
| Correlation Id | true | Correlation Id between action and trigger. Correlation Id helps identify what collection of batches should be used | Flow1 |
| Do Not Delete Batch After Retrieval | false | If checked, the batch will not be deleted after emit | false |
Important: Use the same configuration in action for correct batch processing.
Actions
Add message to batch
Stores a message in a batch and emits a created/updated batch with a processed item only.
Input fields description
| Input field | Required | Description | Example |
|---|---|---|---|
| Max time in ms before Batch is ready | false | Timeout in millisecond after batch creation, which makes batch ready for processing. Defaults to 60000 ms |
3000 |
| Max records number in Batch | false | Maximum count of items in batch which makes batch ready for processing. Defaults to 100 |
10 |
| Max size of Batch in bytes | false | Maximum bytes size of items in batch which makes batch ready for processing 1000000(1 MB) |
500000 |
| Correlation Id | true | Correlation Id between action and trigger. Correlation Id helps identify what collection of batches should be used | Flow1 |
Important: Use the same configuration in trigger for correct batch processing
Additional info
- Created objects in Maester would have default TTL, please contact support in case if you need value of this property.
- Please make sure that values of
Max time in ms before Batch is readyare less than Maester default TTL, as you would never get your batch ready (object in Maester would be deleted before batch is ready).
Known Limitations
- Component uses Maester microservice, so the component is affected by it’s limitations.
- Emit rate logic is not implemented.
- The library does not guarantee a sequence of batch items.
Parallel Processingdoesn’t affect on component performance.
Click here to learn more about the elastic.io iPaaS