Released: 2022-07-14
As part of ongoing codebase optimisation several redundant parameters have been removed from our API - these changes are reflected in our API docs.
PLease visit our API documentation for more information.
We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.
To increase the flexibility of in ingress configurations for different hosts we made a breaking change. We have changed the structure of the model properties tenant.ingressConfig.annotations and helm.Values.global.services.handmaiden.annotations. Please be aware that upgrading to elastic.io Tenants to this latest version requires the small changes described below:
tenant.ingress_config.annotations format changed. This parameter must be reset via API call before updating to the latest version. Please use this API call: {
"data": {
"type": "tenant",
"attributes": {
"ingress_config": {}
}
}
}
global.services.handmaiden.annotations HELM variable format changed, please set it to the new format during upgradeHere you can see the corresponding code for PATCH v2/tenants endpoint:
{
"data": {
"type": "tenant",
"attributes": {
"ingress_config": {
"annotations": {
"app": {
"nginx/123": "123",
"nginx/456": "456"
},
"api": {
"nginx/123": "123",
"nginx/456": "456"
},
"webhooks": {
"nginx/mtls": "true",
"nginx/456": "456"
},
"apidocs": {
"nginx/mtls": "true",
"nginx/456": "456"
}
}
}
}
}
}
Please visit our API documentation for more information.
To improve push behaviour we added --push-retry and made this configurable only via gitreceiver (deafult: 3). Please setup using .Values.global.gitReceiver.dockerImageBuildJobPushRetry.
Information in this section is intended for our customers who use the OEM version of the elastic.io platform.
We are introducing a new сomponent-build-helper version 0.1.0. It also means that the old appbuilder is totally deprecated and is no longer supported. We have added custom dockerfile support for Java components.
Please Note: For now our new component builder supports Java 8 based components only.
Component vulnerabilities check in the build docker image is done by сomponent-build-helper during component build process. Components-build-helper has build_component_docker command which checks docker image vulnerabilities using grype tool.
For more information please visit our article on Components build configuration requirements.
tenant.attributes.customScripts.PLease visit our API documentation for more information.
1.0.0 (v22.28)Make Raw Request Action1.6.3 (v22.28)Lookup Objects action, now the method search is used instead of list for the customer object type1.1.0 (v22.28)Released: 2022-07-28
We have improved the credentials part to provide more unified user experience during the flow design. The look and feel is now more in line with our new user interface approach.
In this new view we moved the Verify, Edit and Delete functions for each credential to a contextual menu which you can open by selecting three dots in the end of each credential record.
2.6.29 (v22.30)A new version of the Node.JS Sailor library 2.6.29 is released. For Users who
develop their own components using our SDKs this update enables keep-alive functionality
for the https outgoing connections as well. This functionality reduces the number
of https reconnects in favour of using already open connections.
1.1.0 (v22.30)A new version of the component-build-helper library 1.1.0 is released. This
library is used by the platform to build components when a new version of an
integration component is deployed. For more information check the
component build configuration requirements
document for supported Java versions and limitations.
Additional parameters field in the Rest-API-component credential would not render properly.1.0.0 (v22.30)Make Raw Request actionLookup Object (at most one) actionPost Message action1.2.0 (v22.30)Upsert Object actionGet Updated Objects Polling trigger3.0.0 (v22.30)Create/Upsert/Update Spreadsheet Row actionRead Spreadsheet actionReleased: 2022-08-11
As part of ongoing program of improving our code base and in-line with our ISO27001 processes we have recently completed a number of improvements to both address existing vulnerabilities and prevent potential vulnerabilities being released. Instrumental to this approach is the use of regular scans of our code base during back ups.
An high-level overview of our InfoSec standards and processes are available here.
We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.
elastic.io platform runs many services under the hood, such as webhooks, api, frontend, etc.
Most of these services can be replicated, which allows the services to be scaled
horizontally to address the required load and volume. However, some platform
services are inherently not scalable. In this update we removed the possibility
to define scaling within HELM chart configurations for bloodygate and handmaiden
services.
Information in this section is intended for our customers who use the OEM version of the elastic.io platform.
We introduce a new possibility to add a custom metadata to the contracts. A new
section called custom_data can be used to store non-sensitive string data up-to 1MB in
size. To add, edit and read contract custom metadata you need to have
contracts.contract.edit_custom_data permission (tenantAdmin has this permission
by default) to perform POST, PATCH and GET calls to the /v2/contract API endpoint.
This new feature enables Tenant Owners to better differentiate different types of Contracts they supply to their Clients and Users. For instance, a selection of different Contract types can be defined and automatically created, deployed, and maintained using this metadata as basis. Each contract type can have a wide range of specified features, such as GB RAM quota limits, numbers of users or workspaces, and even connector whitelists.
PATCH call to /v2/flows/{FLOW_ID} when an invalid team name or an invalid trigger/action name are defined in the command parameter. Now you will get 4xx type error instead of 5xx which better reflects the situation.1.6.4 (v22.32)1.3.0 (v22.32)Dependencies updated for the following Node.js components where updated:
1.5.02.0.21.2.41.0.3Released: 2022-08-25
When we introduced support for newest SSH versions
for the user SSH keys used to deploy components, we deprecated the old ssh-dss
types. If users enter an unsupported or incorrect SSH the user interface now
displays an informative error message instead of simply disabling the ‘Save’ button.
This capability will provide an alternative to Git push deployment of user developed components and is of interest to enterprise clients wishing to condense entire integration flows into single custom components and maintain control at the docker level.
At this earliest stages we lay our groundwork by updating the component
commons and api to support components pull from external docker registries.
The following new attributes were added:
docker_registry object into the HTTP POST call to /v2/teams/
attributes.docker_registry.uri - Docker Registry URIattributes.docker_registry.credentials - Docker Registry credentialsdocker_repo_name and docker_target_tag in the POST HTTP call to /v2/components (can be set if its team has docker_registry object)
attributes.docker_repo_name - String representing component’s name in docker registry. Available only if team supports docker registryattributes.docker_target_tag - String representing component’s tag in docker registry. Available only if team supports docker registryPlease note, as of this stage these attributes have no functionality attached to them.
We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.
The flow execution statistics presented on the Dashboard Runlog are stored in the platform Mongo Database. This is in contrast to the flow execution threads, shown on the Executions pages, which are stored separately in Clickhouse.
When you access the Dashboard Runlog, the platform queries the MongoDB to get these records. These database calls can delay display of the runlog for users with many active flows. To enable users to manage such circumstances we have added an environment variable which defines the retention period of runlogs in the database.
Previously you could only control the retention period by enforcing the expireAfterSeconds
option in the Database index holding the execution records. In this release we
introduce a new TASK_STAT_START_INDEX_TTL environment variable to the HELM3 charts,
which you can use to set retention suitable for your environment. It is mandatory
that a value is set for TASK_STAT_START_INDEX_TTL and the
default value is set at 432000 seconds (5 days). Which means the platform
will store the Dashboard Runlog records for 5 days in the Database.
Setting the TASK_STAT_START_INDEX_TTL environment variable to control the retention
period of the Dashboard Runlog is recommended for the new installations of the platform.
If your platform installation already has the retention period set in the Database prior to your installation of the platform version then your environment variable will not be applied. If you need to change the value you must change the Database index setup prior to the deployment of the current release.
If you need to change the retention period of Dashboard runlogs in the future, remove
the expireAfterSeconds option of the Database index containing the Dashboard Runlog records,
set the value of TASK_STAT_START_INDEX_TTL environment variable and then deploy
the platform version again.
Our platform is now compatible with the Kubernetes version 1.22. Check out the
new features enabled here.
You can now enforce MongoDB TLS verification in all platform services by using newly introduced HELM3 chart configurations:
.Values.global.secrets.mongodbTlsCertificateKey - secret name with the TLS certificate and key. If specified, will be mounted to the services and specified in the tlsCertificateKeyFile connection option..Values.global.secrets.mongodbTlsCA - secret name with CA certificate to validate MongoDB server certificate on the client side. If specified, will be mounted to the services and specified in the tlsCAFile connection option.Information in this section is intended for our customers who use the OEM version of the elastic.io platform.
Here we present further improvements to the recently introduced White-label credential management feature. Now you can use HTML inline frame or iFrame to embed the credentials.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Embedded Credentials Demo</title>
<style>html, body, iframe { height: 85%; width: 90%; margin: 0;}</style>
</head>
<body>
<iframe src="https://[eio_platform_domain]/embedded-credentials/[repoId]?workspaceId=[workspaceId]&ssoProviderType=[ssoProviderType]&ssoProviderId=[ssoProviderId]" />
</body>
</html>
The embed URL construction scheme is the same. Refer to the 22.20 release for details.
Pleass Note: End Users must enable pop-ups in their browser settings in order to use the embedded credential management.
POST to /v2/workspaces/:id/secrets/:id/refresh endpoint would return outdated access_token instead of the newly refreshed value.1.0.0 (v22.34)Get New and Updated Objects polling triggerMake Raw Request actionUpsert Object actionGenerate Custom Distribution Link actionGetting Survey Responses actionLookup Object (at most one) action1.0.1 (v22.34)429 status code. Now it will retry the messages with an exponential backoff.5xx errors. The messages will be retried with an exponential backoff as well.1.5.1 (v22.34)component-commons-library to the version 3.0.11.1.6 (v22.34)email1 and email2 fields for metadata of Contacts module in the Upsert Actionelasticio-sailor-nodejs to v2.6.291.0.0 (v22.34)Make Raw Request Action1.2.5 (v22.34)maester-client library version to 4.0.2component-commons-library to the version 3.0.12.0.0 (v22.34)OAuth 2.0 and migrated to the Secrets service2.6.29Released: 2022-09-08
We continue to update our User Interface to improve ease of use and reduce UI lag. This time it is the UI surrounding Credentials that is updated. When you navigate to Integrate > Credentials you will notice larger icons and a more coherent design.
When you select a component from the Component Overview page, you will see the redesigned Component Credential view. The Salesforce component is shown here as an example:
In this view you can select the kebab menu (3 vertical dots) at the right of each credential to Verify, Edit, or Delete the Credentials. Alternatively, you can select the + icon to the right of the search field if you want to create a new Credential.
Here is how each credential page looks when you Edit it:
While developing new integration flows it is often necessary to retrieve new sample data after each flow step is created. This was previously possible by navigating Back. In this release we have added a new button Retrieve new sample from component that allows you to create new sample data without navigating away from your work.
We improved the user interface of the Node.js Code component configuration field. Now you can use long lines of code witout spaces and the interface will wrap it for you.
We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.
Information in this section is intended for our customers who use the OEM version of the elastic.io platform.
If your tenant is using the registration page and the email_verification parameter
is enabled (set to true) then only the email address field will be displayed.
When an User enters their E-mail address and presses Register, an Email verification link is sent to the mail address. Through this link Users can then enter their password:
1.0.5 (v22.36)maester-client library version to 4.0.32.0.4 and 2.0.5 (v22.36)batching library version to 2.0.31.2.6 (v22.36)maester-client library version to 4.0.3component-commons library version to 3.0.21.5.2 (v22.36)component-commons library version to 3.0.21.2.1 and 1.2.2 (v22.36)Sailor library version to 2.6.29component-commons library version to 3.0.2components-build-helper version 2.0.03.1.5 (v22.36)Sailor library version to 2.6.29component-commons library version to 3.0.21.0.0 (v22.36)Make Raw Request action2.4.0 (v22.36)Get New and Updated Objects Polling trigger, old one set to deprecatedSailor library version to 2.6.29component-commons library version to 3.0.21.5.0 (v22.36)Calculate Flow Dependencies actionValidate Deployability actionLookup Objects (plural) actionRetrieve only base fields for Lookup Objects (plural) actionSailor library version to 2.6.29Released: 2022-09-22
In our quest to improve every aspect of our User Interface, we added a search function to the Credentials page. When you navigate to Integrate - Credentials you can now Search by component name to find the credential card you need instead of scrolling to find it.
When you configure a webhook component step in your integration flow, you can either Send sample request or Add sample manually. When you select Send sample request a WebHook address is provided which you can use to send your example sample.
We also highlight that this address to be used only one-time to send a representative sample which you can use to configure subsequent steps in your integration flow.
To improve the user experience and prevent an accidental JSON data structure pasting
into the Body field while in an Integrator mode, we changed the User Interface
behaviour to default into JSONata mode when an application/json content type is
selected for the POST and PUT HTTP calls.
This change avoids possible errors when users forget to change the input field settings deom Integrator to JSONata mode.
Components exchange information with various platform services using run-time environment variables during execution of each step of an integration flow.
We are constantly working to standardise and add new environment variables to streamline the information exchange. The following environment variables were added to the run-time:
WEBHOOK_URI - This environment variable will be available to each step in the flow which exchanges data with the webhook service, eg. WebHook or Flow-linking. This environment variable from the Tenant attributes webhooks_domain or from the HELM3 environment variable WEBHOOKS_BASE_URI if the Tenant record is missing.
ELASTICIO_CONTRACT_ID - To help with troubleshooting of errors in multi-contract cases this environment variable will be available during the run-time of every execution including:
2.7.0 (v22.38)A new version of Node.JS Sailor library 2.7.0 is released. This update is
preparatory groundwork for the message persistence feature.
3.4.0 (v22.38)A new version of Java Sailor library 3.4.0 is released. This update is
preparatory groundwork for the message persistence feature.
We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.
Iron-bank service collects statistics from the running integration flows. The
service is now running as Kubernetes DeamonSet together with
the cadvisor. This reduces the load on the system significantly and increases
the scalability of Iron Bank.
Information in this section is intended for our customers who intend to extend the automatic component delivery to include custom developed components.
Added new endpoint to PATCH /v2/teams to add docker registry to a team. Check
the API documentation for the details.
To extend the automatic component delivery to include custom developed components, we
modified the componentsConfiguration section of HELM3 charts in the following way:
componentsConfiguration:
developmentTeam: teamName
components:
componentName:
# Image tag for pulling into the internal docker registry. Required, can be "latest"
targetTag: latest
# Image name by which to pull repo from registry. Optional, `team_name/repo_name` by default
dockerRepoName: "orgName/repoName"
# Team where create RepoBuild. Optional. Will override root `developmentTeam` on the `components` level
developmentTeam: teamName
# Access for the component in the platform context. Optional, `global` by default
access: team
Please Note: These are breaking changes and you should carefully revise your setup if you intend to use this feature.
developmentTeam - unchangedcomponentName - unchangedregistry, url, repo and tag got changed into dockerRepoName, developmentTeam, targetTag and access. Please review these changes carefully.Along with these changes the following new required variables were introduced:
DOCKER_REGISTRIES_ENCRYPTION_KEY used to encrypt credentials for docker registriesCOMPONENTS_PUSHER_AUTH_USERNAME used to authenticate to components-pusher serverCOMPONENTS_PUSHER_AUTH_PASSWORD used to authenticate to components-pusher serverThe components-pusher expects to find these variables to deploy the component docker images into your default team.
The new approach expects to also find a new optional TEAM_DOCKER_REGISTRY environment
variable in case of custom developed components. It is defined in the following way:
TEAM_DOCKER_REGISTRY:
uri: "https://index.docker.io/v1/"
credentials:
username: ""
password: ""
For the uri system expects string containg the address of docker registry.
The credentials object can have the following properties:
authFile - path of the authentication fileregistryToken - bearer token for accessing the source registryusername and passwordIf none of the above are used - components-pusher will try to access the registry
anonymously.
The components-pusher is not longer a cronjob but a fully integrated service of
the platform. It runs in Kubernetes cluster in a single instance (replicas non-configurable).
The service runs following way:
dockerTargetTag fields in the repo documents..Values.componentsPusher.processLoopInterval parameter of the component-pusher service. The frequency of checks is set in millisecond(ms). The default is 86400000 ms (1 day).component-pusher uses the newly added internal endpoint (/copy/:component_id?version=:version) to pull of the specified component and version from docker registry. Uses basic authentication with COMPONENTS_PUSHER_AUTH_USERNAME and COMPONENTS_PUSHER_AUTH_PASSWORD values defined in the HELM3 charts.createTeam task now creates/updates default team with docker registry data from the TEAM_DOCKER_REGISTRY environment variablecreateDefaultRepos task creates/updates components listed in componentsConfiguration. Teams, in which these repositories are created, require to have docker registry data associated with them. If default team used - docker registry data will be taken from TEAM_DOCKER_REGISTRY variable. If any other team used - it needs to be patched with correct docker registry data using API endpoint PATCH /v2/teams.oauth and additional fields, baseUrl, etc ) could not be retrieved in the OAuthFieldView. Now all fields can be accessed directly from the cfg object.1.0.0 (v22.38)Introducing new component for Microsoft Dynamics CRM.
Make Raw Request action2.2.5 (v22.38)1.1.0 (v22.38)CLIENT_ID and CLIENT_SECRET credentialsLookup Object (at most one) actionLookup Objects (plural) actionUpsert Object actionDelete Object By ID actionGet New and Updated Objects Polling trigger1.0.3 (v22.38)Sailor library version to 2.6.29component-commons library version to 3.0.21.0.2 (v22.38)Sailor library version to 2.6.29component-commons library version to 3.0.21.4.3 (v22.38)Lookup Object (at Most 1) action for Object Type - OrdersAllow ID to be omitted option in the Lookup Object (at Most 1) actionSailor library version to 2.6.291.6.5 (v22.38)429 error (respecting the Retry-After header)1.5.3 (v22.38)Verify credentials