Product Update - v22.14

Details of the product update v22.14 from 11th of April 2022.

Improvements and Updates

Step designer: Sample data part

We have improved the sample data retrieval to provide a more unified user experience during flow design. The look and feel is now more in line with our new approach.

Step designer: Sample Data part

The above images show the new Sample data retrieval view. This release 22.14 adds new functionality, allowing users to skip sample generation by clicking on the Skip Sample button. When this function is used the platform will add an empty JSON {} instead of the Sample data.

When you retrieve the Sample the platform UI will show it only in JSON format. In 22.14 we removed the Integrator view of Data Sample.

Step designer: Sample data retrieved

Here you can view the Sample, collapse different sections of the JSON data structure and edit it.

Quota usage page improvements

To show the RAM quota overuse we improved the quota Usage History histogram. If you go over the set RAM quota value the overused part will show in red.

Quota overuse

SSH Keys and SSH 8.8

To support the newest SSH versions we deprecated the old ssh-dss signature and we added ssh-ed25519 instead. Now when you upload your SSH key to our platform you will see the information about the supported types of signatures.

SSH Key upload screen

Please Note the old ssh-rsa keys are still supported. If you want to use them you have two options:

Use an OpenSSH 8.8+ client and generate your SSH Key using the -oPubkeyAcceptedKeyTypes=+ssh-rsa parameter. It will generate an RSA+SHA-2 type SSH key which provides a high level of security and works with the OpenSSH 8.8+ clients.
Use an OpenSSH <=8.7 client to generate your SSH Key.

Flow deletion times

The platform 22.14 includes new capability to hard-delete the flows after an expiration time set by the platform management. Every time you try to delete the flow a warning message will show with the following text:

Are you sure you want to delete “FLOW NAME” flow permanently? Deleting this flow is an irreversible process, we cannot undo a deletion if you complete the process by accident. Please note that the credentials linked with the flow will not be deleted.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements done in this round.

Changes to `cadvisor` and `fluentd` demons

To enable better coexistence of platform microservices within the multi-service Kubernetes clusters, we modified the logic used to run cadvisor and fluentd services. The platform will run these services when the node has uniquely defined selectors.

To achieve this we introduced a new HELM3 configuration for the node labels for the Kubernetes pod allocation.

global:
  nodeSelectors:
    platform: "platform"
    longRunningTask: "longrunning"
    ordinaryTask: "ordinary"

For these changes to take affect we deprecate the following HELM3 configurations:

KUBERNETES_ORDINARY_LABEL_VALUE
KUBERNETES_LONG_RUNNING_LABEL_VALUE

These variables ware removed from the secrets HELM3 chart and are no longer used by the admiral service starting from 22.14 release. Please use Values.global.nodeSelectors platform HELM3 chart configuration instead.

The cadvisor service will run on the nodes with longRunningTask and ordinaryTask labels.
The fluentd service will run on the nodes with platform, longRunningTask and ordinaryTask labels.

Richer Ingress Management

To extend the Ingress setup capability we made changes in the Handmaiden microservice to provide 2 new ways to provide annotations to the Ingress records. With this new capability you can set unique Ingress values per each tenant in your cluster.

Set in the HELM3 chart.
Set per tenant using an API call.

Set HELM3 configuration

You can set the Ingress values directly in your HELM3 chart like:

global:
  services:
    handmaiden:
      annotations:
        kubernetes.io/ingress.class: 'nginx'
        testAnnotation: 'testValue'

If you set the above configuration the platform passes it to the new environment variable INGRESS_OPERATOR_ANNOTATIONS used by the Handmaiden service. This adds Ingress annotations to all tenants in your multi-tenant cluster.

Setting via API call

You can set unique Ingress annotations for each tenant using the new configuration while creating the tenant via an API call. A typical call body could look like:

{
  "data": {
    "type": "tenant",
    "attributes": {
      "ingress_config" : {
        "annotations": {
          "annotationName": "value"
        }
      }
    }
  }
}

Annotation Priorities

Our platform will serves these annotations based on priority of annotation the configurations (from low to high):

Default Ingress annotations,
Annotations provided in the INGRESS_OPERATOR_ANNOTATIONS environment variable,
Annotations, provided for the tenant using API call,
Static, hard-coded annotations.

Flow deletion job

Introducing a new cron job called clear-deleted-flows which permanently deletes flows from the MongoDB with the DELETED status. Every user trying to delete a flow will see a warning message informing that this is a permanent and irreversible action.

HELM3 optional variables

We introduce new environment variables which you can use to control the clear-deleted-flows job:

WIPER_CLEAR_DELETED_FLOWS_AGE_SECONDS - Time in seconds the job must wait before deleting the flow permanently after it is marked as DELETED in MongoDB. We set the default value to 86400 seconds (1 day).
WIPER_CLEAR_DELETED_FLOWS_LIMIT - Maximum number of flows the clear-deleted-flows job will permanently delete each time it runs.

Release process

This release process applies if you already have a cluster running previous versions of the platform code. The release will start an automatic migration.

Backup the MongoDB data base before the deployment to have an opportunity to reverse the process.
Set WIPER_CLEAR_DELETED_FLOWS_LIMIT environment variable to an applicable number for your case to perform this operation faster of you have large number of flows in your DB.

After the migration the marathonevents, requestbins and taskstartlogs deprecated collection will be removed.

Iron Bank table creation on all Clickhouse instances

New Environment Variables:

IRON_BANK_CLICKHOUSE_NODES (required) - array of objects with host (required), port, user and password values
IRON_BANK_CLICKHOUSE_DATABASE (optional) - ClickHouse database name (default iron_bank)

IRON_BANK_CLICKHOUSE_NODES:
   - host: "10.0.0.1"
     port: "8123"
     user: "default"
     password: "123"
   - host: "10.0.0.2"
     port: "8123"
     user: "user"
     password: "1234"
IRON_BANK_CLICKHOUSE_DATABASE: "iron_bank"

New Iron Bank provision logic

If all nodes in IRON_BANK_CLICKHOUSE_NODES are not available, an error is thrown and iron bank shuts down
If one of the nodes is not available, a warning is logged and provision to all other available nodes is executed
The main connection (to which all querying is executed) is set to the first available node from IRON_BANK_CLICKHOUSE_NODES

Removed Environment Variables:

IRON_BANK_CLICKHOUSE_URI

Fixed bugs

Addressed issue when the platform token refresher service would return 500 status code in case when the /refresh/[id] Oauth server would return an error. Now the service reports 4xx indicating the third party problem.
Fixed the bug connected with the Profile Page usage example returning 401 in case of API-Key regeneration.

Components

Hubspot Component `1.5.1`

FIXED the bug in Get New and Updated Objects Polling preventing it from extracting more than 10K records.
UPDATED sailor library to 2.6.27 version.
ADDED added the component pusher job to Circle.ci configuration.

Salesforce Component `2.2.2`

ADDED reconnect logic on errors.
UPDATED sailor library to 2.6.27 version.
ADDED added the component pusher job to Circle.ci configuration.

Vtex component `1.3.0`

ADDED a new Lookup Object (at Most One) action

Paypal component `1.1.0` and `1.2.0`

ADDED a new Get New and Updated Objects Polling trigger,
ADDED a new Lookup Object (at Most One) action,
ADDED a new Make a Payment action.

SOAP component `1.2.9`

IMPROVED Metadata generation,
ADDED a possibility to handle WSDL files where the “message” doesn’t contain an element,
UPDATED the Java Sailor libraries to 3.3.9 version.

SFTP component `1.4.6`

IMPROVED large files handling. Now downloading file sizes bigger 10MB would not truncate the file.

JDBC component `2.5.2`

UPDATED the Java Sailor libraries to 3.3.9 version.

Petstore component Java `1.0.4`

UPDATED the Java Sailor libraries to 3.3.9 version.