Product Update - v21.31

Details of the product update v21.31 from 7th of August 2021.

Features

Regenerate your REST API Key

The 21.31 introduced a new opportunity to regenerate your personal API key you use to authenticate your calls with the platform REST API.

You have two options, either using a new Generate new API Key button in your profile page or using a HTTP POST to the new API endpoint /v2/users/me/generate-api-key. In both cases the platform will generate a new API Key and store it in the Database

Please Note Generating a new API key will replace the existing API key. If a new API key is generated, then API calls made with the old API key will fail.

Two-factor Authentication

We introduce two-factor (2FA) login authentication to provide an additional layer of security. We use Google Authenticator for the 2FA. If the feature is enabled in the tenant then you can enable it for your account following the instructions in the profile page.

Component Whitelisting

With the platform 21.31 version we introduce a new feature called Component Whitelisting. In verbatim this means we can enable or disable certain integration components in each contract based on the list of components - the Component Whitelist. If the component whitelist is enabled in the tenant the following rules would apply in each contract:

  • You have access to all components available in your tenant while working in the developer (limited) workspaces.
  • You can only build integration flows using the available components (whitelisted components) in your production workspaces. Not included components would not show in your production workspaces while building an integration flow.
  • You can not export an integration flow from the developer workspace to production workspace if the flow contains a component not included in the whitelist.
  • You can not create a recipe with a contract level access when the recipe uses a component not included in the whitelisted components for the contract.
  • The developer workspace will not be converted to a production workspace in case when at least one step of one flow in your developer workspace uses not whitelisted component.
  • All native components from your contract with team access level would still be available in your production workspaces.

Remove pending Contract invites

With platform 21.31 we added support for deleting pending invitations to contracts using UI and API.

Now you can delete the user invitation by visiting Members section of the Contract Settings, scrolling down to find the list of Pending Members and clicking to delete. Check the following documentation page for more details.

You can also delete the invitation using the API call.

HELM3

We are in middle of constantly improving various aspects of HELM3 deployments in collaborations with our partners. These are updates and improvements done in this round.

Kubernetes Namespaces

With 21.31 release we remove Kubernetes namespace hard-coding from our charts to integrate the platform into more Enterprise friendly infrastructures.

IMPORTANT To deploy the platform version 21.31 on top of your existing cluster you would need to re-create the entire kubernetes cluster. We recommend to announce maintenance run since the platform would be down for 10-15 minutes.

Here are the steps necessary for a successful platform deployment:

  1. Delete existing namespaces OR uninstall helm release (IMPORTANT: this step will delete the cluster).
  2. Create following namespaces in kubernetes cluster (the name and labels.name should be equal to values provided in the values.global.namespaces of HELM3 charts):
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
    name: <monitoring_namespace_name_from_values>
      name: <monitoring_namespace_name_from_values>
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
    name: <platform_namespace_name_from_values>
      name: <platform_namespace_name_from_values>
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
    name: <tasks_namespace_name_from_values>
      name: <tasks_namespace_name_from_values>
    

    The labels are important to platform’s network functionality.

  3. Install the helm release.

AWS Network Load Balancer support

Platform version 21.31 includes the support of the AWS Network Load Balancer for ingress LoadBalancer. Here are the details of this change:

  1. We introduce a new storage type (for the AWS cloud provider) aws-efs-csi-driver. This means the AWS EFS storage type is supported in case when Amazon EFS CSI driver is installed on the EKS cluster. You can see more details in the Amazon EFS CSI driver configuration documentation. Here is an example configuration for your guidance:
    storage:
    slugsSubPath: “slugs”
    stewardSubPath: “steward”
    storageClassName: “platform-storage-slugs”
    persistentVolumeClaimName: “platform-storage-slugs-volume-claim”
    type: “aws-efs-csi-driver”
    config:
      name: “platform-storage-slugs-volume”
      path: “/pss”
      size: “20Gi”
      gid: 1502
      fileSystemId: fs-03f24358 # AWS EFS file system ID
    
  2. We added the parameter for the AWS Network Load Balancer (NLB). Use global.cloudProvider: AWS in your configuration to enable this (HELM3 values).
  3. We added also support for the Secured docker registry. To enable the following configuration must be provided in the HELM3 chart:
    global.services.dockerRegistry.secured: true
    global.services.dockerRegistry.tlsSecretName: "docker-registry-tls"
    global.services.certificates.dockerRegistry.name: "docker-registry-tls"
    global.services.certificates.dockerRegistry.key: "<<base64 encoded key>>"
    global.services.certificates.dockerRegistry.crt: "<<base64 encoded certificate>>"
    

Please Note:

  • Certificate CN should match domain name specified in the global.services.dockerRegistry.loadBalancerIp,
  • The docker registry load balancer does not support external load balancer type yet. Created domain must be bound to the private ip address.

Two-factor Authentication Enabling

For 2FA we use Google Authenticator mechanism. To enable the 2FA on the tenant you must perform the following steps:

  • Set a new parameter called USER_TOTP_CRYPTO_PASSWORD in the platform secrets part of the HELM3 charts. You must generate it in advance (like pwgen -s 32 -n 1 -B). If this value is not set all the following steps will fail.
  • Set the new tenant level parameter (switch) attributes.feature_flags.enabled_totp to true using a PATCH to /v2/tenants/{TENANT_ID} record (admin access).
  • When user enabled the 2FA and fails to login for some reason you can POST to a new API endpoint /v2/users/:id/disable-totp to clear 2FA setup and enable user to login again using only username and password (admin access).

Support HTTPS for docker registry URI

With this release we added HTTPS for external docker registry URI.

News in this section are for our customers who use OEM version of the elastic.io platform.

ClickHouse replication instructions

To facilitate installation of ClickHouse and the required dependencies to run in the High Availability mode we added the following documentation pages:

Enabling Component Whitelisting

The newly introduced Component Whitelisting feature includes changes and parameters only accessible to the tenant administration. Before you can use the component whitelisting you must enable a dedicated tenant level parameter (switch) attributes.feature_flags.contract_component_whitelist. Set it to true using a PATCH to /v2/tenants/{TENANT_ID} record (admin access).

When the tenant level switch is enabled, the system will start the migration and add IDs of all contracts to the components records in cases when the component is used in integration flows of production workspaces. Any subsequent new contracts would only have access to all components in the developer workspaces.

Flow network isolation

With this release we investigated and implemented integration flow complete network isolation. The traffic is now restricted to/from step pods. Only allowed connection to services required for the runtime.

Fixed Bugs

  • Addressed the problem when it was not possible to join the Contract via invitation letter in case login with OIDC (Login using Google credentials)
  • Fixed UI bug Invalid '0' value in the required fields with type 'number'.
  • Addressed and standardised Maester’s Redis. Sometimes it used to leave incomplete object keys and Maester would not regard this as problem.

Components

Splitter component 1.2.1

  • ADDED support of maester storage in Re-assembled message action (maester-client library 3.3.0)

CSV component 3.0.0

We released a new version of the component which is not backwards compatible. Users are encouraged to investigate the CSV component documentation to learn about the new component.

The component will not be deployed on top of the old component repository, you can keep using the old component. At some stage we will deprecate the old one. The new component is available as csv-v3 component in your components list.

  • ADDED new following actions:
    • Read CSV attachment
    • Create CSV From Message Stream
    • Create CSV From JSON Array
  • DELETED the following functions:
    • Read CSV attachment trigger
    • Write CSV attachment
    • Write CSV attachment from JSON Array
    • Write CSV attachment from JSON Object
    • Read CSV file from URL

id-linking-component 1.0.1

  • UPDATED to sailor version 2.6.26
  • UPDATED to Maester client library version 3.0.0

Delta Detection Component 2.0.1

  • UPDATED to Maester-client version 3.3.0

Google PubSub component 1.7.3

  • FIXED the bug with not-awaited emits and endless cycle

Maester component 1.0.0

We introduce new component for working with platform native object storage called Maester.

  • ADDED the following actions:
    • Delete Object
    • Lookup Object (at most one)
    • Lookup Objects
    • Upsert Object

Utility Component 1.3.0

  • ADDED new action: Delay
  • UPDATED the sailor version to 2.6.26 and build type to docker