Product Update - v21.27

Details of the product update v21.27 from 8th of July 2021.

Improvements and Updates

Node.js Sailor 2.6.26

We released a new Node.js Sailor 2.6.26 which improves the communication and speed with queueing system. In particular with new sailor we:

  • Return to consume-based approach in communication with RabbitMQ.
  • Improve the re-connections to RabbitMQ.

Here is the NPM package release link.

Executions page slow loading

With the 21.27 release we improved the loading speeds of the execution threads. In this scope the following improvements were done:

  • Optimised the internal communications between the UI and back-end services.
  • The revised and improved the threads list (execution page) and individual threads pages.
  • The default time intervals for listing the threads is now 7 days instead of previous 30 days.

We also added a new environment variable to the system (BRAN_RETENTION_MONTHS_MESSAGES) This variable defines the number of months to store the data for Executions page. The default value is 1 (1 month).

HELM3

We are in middle of constantly improving various aspects of HELM3 deployments in collaborations with our partners. These are updates and improvements done in this round.

Redis deployment is optional

To support external Redis service we made the service as optional and the HELM3 charts were changed accordingly. To use en external Redis service follow these steps:

  1. Set the values.global.services.maesterRedis.enabled and values.global.services.cache.enabled values as false. This means both Redis services will not be deployed during the deployment.
  2. Add values for the following parameters to use external Redis:
    • values.global.config.CACHE_REDIS_URI: redis://uri-to-external.redis,
    • and values.global.config.MAESTER_REDIS_URI : redis://uri-to-external.redis.

Charts Security Audit

An audit of HELM3 charts were performed to remove any sensitive data like keys, password and credentials from main charts. All sensitive data were migrated into the secrets.

In scope of this audit a new environment variable FRONTEND_SESSION_SECRET was added. Configure it in Values.global.config.FRONTEND_SESSION_SECRET. It is required value, without it the platform deployment would not work.

News in this section are for our customers who use OEM version of the elastic.io platform.

Internal IP address for internal communication

With 21.27 version we changed a logic of internal service communications. All internal calls to the API would use internal API address from now on. For that reason the parameter API_URI environment variable value will be computed by Kubernetes API service URL and not hard-coded. Here are the details of this approach:

  • Do not use external IP addresses for communication between internal services. Use API_URI everywhere. The tenant record parameter responsible for this is data/attributes.api_domain.
  • Deprecate the use of EXTERNAL_API_URI parameter.
  • Use api.elastic.io default in the platform UI when displaying the API address if not set in tenant.

Control Number of Contract Users

You can now control number of users in any contract in your tenant. This new feature works through the platform quota service and has a unique API endpoint /v2/quotas/per_contract_user_count_limit/TENANT_ID.CONTRACT_ID.

Note: The default value of this quota is -1, which means unlimited.

As with all types of quotas you need global.quota_limits.edit permission to apply any values.

When the quota is set for a contract the following rules apply:

  • You can not invite a new member to your contract when quota is exhausted. The invitation will not be created.
  • You can not add a user via an API call when the quota is exhausted.
  • You can not redeem previously sent invitation to the contract when the quota is exhausted in the meantime.
  • If the support user functionality is enabled in your tenant then adding support user to your workspace is not counted towards your contract quota.

Quota Release and Apply Process

We extended the /sidedoor/quotatxns/sync endpoint used to sync the quotas during the release process. This endpoint must be called once to sync the correct numbers of users in contracts and workspaces with the quotatxns data base.

You must add an extra parameter that represents resource type to sync. Available types are user and workspace. For example:

  • /sidedoor/quotatxns/sync/user,
  • /sidedoor/quotatxns/sync/workspace.

For this release you must call the following command to properly sync user data:

curl -u 'username@password' -X POST API/sidedoor/quotatxns/sync/user

You need to use have AccessManager.ACTIONS.GLOBAL.QUOTATXN.SYNC permission to work with these endpoints.

There is a known issue with sync, it might return 504 if your DB contains large number of contracts. The sync operations can take anywhere from 30 to 60 seconds causing a timeout on HTTP proxy. However, the sync operation will continue in background to a successful completion.

Bran and Clickhouse Optimisation

If you are running ClickHouse in replica set continue reading.

On this round of optimisation we optimised the bran service by creating new tables and views in ClickHouse. To apply you must run the following command for each replica after the installation of 21.27 version:

docker run \
-e LOG_LEVEL=info \
-e AMQP_URI= \
-e MONGO_URI= \
-e BRAN_CLICKHOUSE_URI="http://localhost:8123/bran" \
--rm --network=host elasticio/bran:21.27 npm run start:migrate

Don’t forget to change the BRAN_CLICKHOUSE_URI to your replica URL.

Fixed Bugs

  • Addressed the failed to fetch step_id and related problems from amqp routing key in case when an old node.js sailor (earlier than 2.3.0) library is used in the component.
  • Fixed the problem of hard-coded session secret. A new required environment variable FRONTEND_SESSION_SECRET is introduced.

Components

REST API component

  • FIXED OAuth2 authentication strategy limitation: refresh_token property is now optional for Access Token Response (also optional in OAuth2 standard).

Delta Detection Component

  • REFACTORED to maester-client library based on object-storage-client library.
  • UPDATED to sailor version 2.6.24.