Our platform is designed to connect a variety of applications and systems. The platform uses dynamic and static flow control to manage the flow of data. Understanding the differences between these two methods of flow control is important to ensure efficient, reliable, and scalable data integration.
Static flow control involves setting a fixed rate of data transfer, regardless of their current performance or capacity. While static flow control can be useful in situations where the performance of the applications and network is predictable and stable, it can lead to problems if the data flow rate is set too high or too low. Static flow control can cause data loss, latency issues, or other problems if the rate of data flow is not adjusted based on real-time feedback from each application.
The Dynamic Flow Control is based on RabbitMQ Publisher Confirms and RabbitMQ Flow Policies which enables a dynamic slow-down of a publisher/producer based on the current queue state. Each task queue has a messages limit. RabbitMQ
overflow: 'reject-publish' policy is set for all task queues to reject steps to publish into queue which is overflowed by messages. Sailor will retry publish infinitely until message will be successfully sent.
Sailors use Publish Confirm RabbitMQ feature. They prefetch some amount of messages (”Parallel Processing” in UI, default: 1) and send confirmation back to RabbitMQ after processing each incoming message. Processing of incoming message ends when sailor successfully sends a message to the next step. So a step can’t process more then “Parallel Processing” messages at a time, other messages are waiting in a queue.
RabbitMQ Flow Control automatically reduces the speed to connections which are publishing too quickly to keep the rate of message ingress at one that the rest of the server (e.g. queues those messages are route to) can handle.
When you retry erroneous messages, they get marked with
retry=true header, so lookout doesn’t write data record.
Sailor retries publishing messages to next step infinitely, for those cases when the next queue is overloaded. Retries will happen with exponential back-off. For example, if it starts with retry in 5 seconds, the next will be in 10, then 20, then 40, then 80 seconds, etc. The maximum delay is configurable with an environment variable. By default, there is no limit to retries.
The following variables control the retry process:
AMQP_PUBLISH_RETRY_DELAY: 100 ms
AMQP_PUBLISH_MAX_RETRY_DELAY: 5 minutes.
Please Note: Dynamic Flow Control affects Flow Queues. For more information read this article.
As the feature may have a performance impact, you can disable it. Currently disable is supported only if step is using sailor-jvm
3.3.5+ or it’s a webhook step. In this case you will see a “Dynamic flow control” switch in UI during step configuration. When Dynamic Flow Control is disabled for a step, RabbitMQ Publish Confirm feature is not used by sailor and, of course, there will be no retries on error.
Please note taht you can also disable dynamic flow control using the corresponding API call. More on this in our API Documentation.
In an e-commerce order management system, orders may need to be processed in real-time to ensure timely delivery to customers. A static flow control system may be configured to send a fixed number of orders to the warehouse management system every minute. However, if there is a surge in orders, the warehouse may become overloaded, and orders may not be processed on time, resulting in customer complaints.
With dynamic flow control, the platform can monitor the warehouse’s performance and adjust the rate of orders being sent to ensure that the warehouse is not overloaded. The platform may also adjust the rate of orders being sent based on network conditions, such as latency or available bandwidth.