Product Update - v21.23

Details of the product update v21.23 from 10th of June 2021.

Features

The release 21.23 introduces new features for component developers and high-end integration projects.

Fine-tuning the timeout per component

We continue the work dedicated to defining and setting custom timeout intervals. In this round we extended the capability to include verify credentials, select model and dynamic data processes. We call them one-time executions. To define a custom interval for one-time executions set the newly introduced environment variable SCHEDULED_EXECUTION_TIMEOUT (seconds) in the component repository. If the variable is not set the platform will use the system defined value.

Scaling of flow steps

With 21.23 we are happy to introduce the project dedicated to providing a high availability of integration flow steps by scaling them to replica sets.

Please Note : Most integration scenarios do not require step scaling Using this feature might cause high costs.

The implementation has specific guidelines:

  • Use step scaling on steps which need scaling. For these steps platform will use deployments which control the pods.
  • Never enable startup and shutdown hooks for scaled steps.
  • Set the same number of instances for the following mapper step.
  • At the moment we have set an upper limit for scaling - maximum 5 replicas.
  • If you set scaling to 1, the system will not create deployments.

To set number of replicas for the integration flow step use the API call to PATCH v2/flows/:id endpoint. Here is an example of body for API call.

{
    "data": {
        "type": "flow",
        "id": "",
        "attributes": {
            "nodes_config": {
                "step_1": {
                    "replicas": 3
                }
            }
        }
    }
}

Improvements and Updates

Support for infinity tokens

Some OAuth2 providers supply one access token and no refresh token. These are the cases of infinity tokens. To support these cases we extend the functionality of Secrets feature. We implemented the following changes in the Secrets feature:

  • Refresh token is no longer a required parameter in the Secrets feature,
  • Refresh token is not a required field while creating a secret on the UI,
  • If no refresh token is supplied then the parameter will be ignored and the system will not try to refresh the secret.

Java Sailor 3.3.6

We released a new Java Sailor 3.3.6 which introduces a parallel processing of messages. All components written in Java using the new sailor version (3.3.6+) can use newly introduced environment variable ELASTICIO_CONSUMER_THREAD_POOL_SIZE to set the size of thread pool for AMQP consumer. If not specified (by default) it equals Prefetch Count value.

Validation for timeout variables

With introduction of SCHEDULED_EXECUTION_TIMEOUT and DEBUG_TASK_TIMEOUT (last release), we also implemented a validation mechanism for them.

  • The platform will allow only numbers in seconds for DEBUG_TASK_TIMEOUT and SCHEDULED_EXECUTION_TIMEOUT environment variables.
  • DEBUG_TASK_TIMEOUT and SCHEDULED_EXECUTION_TIMEOUT values have maximum values controlled by the system defined MAX_FORCE_DESTROY_DEBUG_TASK_TIMEOUT_SEC and MAX_FORCE_DESTROY_ONE_TIME_EXEC_SEC parameter values.
  • We deprecate the LOG_LEVEL, PREFETCH and AMQP_PUBLISH_CONFIRM_ENABLED previously defined environment variables. These variables MUST be deleted from the component repositories if they exist.

News in this section are for our customers who use OEM version of the elastic.io platform.

HELM3 deployment

The platform deployments migrated to HELM3 from the Ksonnet (last deployment 21.19). Since the intermediate release 21.21 deployments done with HELM3. The current 21.23 release is the second successful deployment using HELM3 charts.

We are using Helm OCI Registry, credentials available upon request.

Fixed Bugs

  • Addressed the network issue caused by the web-sockets timeouts while loading responses to sample retrieval during the flow design process.
  • Fixed the bug when updating the Salesforce credentials would fail with 400 response.

Components

Petstore-nodejs component

  • UPDATED statuses are now case-insensitive.
  • UPDATED to sailor version 2.6.24.
  • UPDATED README documentation.
  • UPDATED petstore password in the component.json.

Dun-and-bradstreet component

  • FIXED a bug in Cleanse and Match action when a single field was not treated as an array item.

Magento2 component

  • ADDED support for Magento2 versions 2.3.5 and 2.3.6