Before we can proceed any further one should clearly examine the API provider’s documentation to get familiarized with the accepted method of pagination usage.
In order to avoid producing too many data, a component developer should use paging which is done in a combination with the snapshot feature.
First, we would need to setup the snapshot object which would only have 2 properties: nextPage and timestamp. The property timestamp is used to record the most recent update time of an object in the target system and the property nextPage is used to remember the index of the next page to retrieve. Here is the algorithm behind:
At the beginning snapshot is empty, so that trigger queries based on timestamp >= 1970.
If the API supports paging, trigger checks if the result is paged. If yes then the next page to retrieve is calculated and saved into the nextPage property in the snapshot.
On next iteration, trigger checks if nextPage is available in the snapshot. If so it queries now the next page and overwrites the nextPage with a new value. At the same time, it should preserve the timestamptimestamp for the later comparison.
After iterating through all the pages, the nextPage parameter is removed from the snapshot. However, the timestamp property must stay in the snapshot.
It is important to define max timestamp through all the pages. This is accomplished either by filtering or by comparing all the timestamp values in all pages with the timestamp in the snapshot.