Basic Data Mapping
In this article we will explain how to map the data between integration components to keep them in sync.
An integration flow on elastic.io platform must have at least two components, one trigger and one action. Trigger emits a data for the action component to receive and process it. Between these two components seats the elastic.io data mapper, which maps or matches the incoming data to the specific fields where the next component expects them to receive.
To understand how the data-mapping works in practice visit our tutorials and platform feature sections. We recommend starting from steps-by-step instructions in how to create your first integration flow and creating a webhook flow followed by the data samples articles as an introduction to the data-mapping.
If you have already followed the tutorials you realise that the data-mapping on elastic.io platform is an important part of the integration process which warrants detailed explanation in its own.
Mapping simple values
Let us consider a simple scenario when we have an incoming data through the Webhook component like the following JSON structure.
{
"name": "Gromit",
"status": "Sold"
}
You can add this structure manually using the appropriate function:
We intend to map these values into outgoing fields in Petstore API component. Let us jump into the integration flow design right at the mapping stage.
The screenshot above shows the stage in integration flow designer where the actual mapping or matching of the values is done. The Petstore API operation for creating a Pet requires a name and a status for the new pet. That’s why the component asks the user to provide input in two fields: Name and Status. These two fields are required (red exclamation marks), which means without providing the values for them we won’t be able to proceed further and click on Continue button. Let us proceed with the mapping.
To map the Name field click on a drop-down menu (1), choose the step you need (2) and select the matching value from the provided values (3) . For example, our incoming data has a field name which we match with the Name field from Petstore API component.
The screenshot above shows the successful mapping result which is Gromit. Notice that you have to click to the Eye button here which shows the successful evaluation result.
Before going further we can check the JSONata mode here (1). Here you can look under the hood of the JSONata powered mapper and see the list (2) of expressions and functions that is possible to use. We have already used them in the how to create your first integration flow tutorial.
We can match the Status field with the incoming status value as well to complete the mapping and go forward. Using the Status field as an example, you can see that there are fields that accept fixed values only. In this case, these are: available, pending, sold:
If you enter an incorrect value, the field will be highlighted in orange:
Note During the mapping process the mapping expressions is evaluated on the samples. During the component execution elastic.io evaluates these expressions based on the incoming data which will differ from the presented sample.
Using advanced mapping mode
In the previous section you learned how to map data between two steps of an integration flow using a JSONata expression. This is a convenient approach, however, the graphical mapping in the Integrator mode has certain limitations.
The following screenshot demonstrates the Developer mode.
In the Developer mode you can write a single JSONata expressions for the entire input object instead of defining expressions for each individual field as in case of the Integrator mode.
To use the Developer mode, you need to know the meta-model of input data: the structure of the object and the property names. Without having access to component’s source code knowing the input model is impossible.
Therefore, when you switch from Integrator mode to Developer mode, the mapper input field is pre-populated with a required JSON expression. elastic.io generates it from the incoming meta-model parameter values and random property values. You can write your mapping by overriding the property values.
The above screenshot shows how the property values is replaced by the variables name and status. These variables get their values from the incoming payload of the Webhook component. To get a mapping result, you should evaluate the expression by clicking the corresponding button.
Please note your expression is evaluated as you are typing it. The result is shown at once below the mapper input field, in the Mapping result section. If you give an invalid expression, an error message is displayed.
The screenshot above shows the incomplete JSON structure which is not accepted by elastic.io and the error is shown in the Mapping result section.
There is a way to filter mapping view to see mandatory fields only by hiding optional fields:
Note You can switch between Developer and Integrator modes during the design of integration flow but not after the flow is published. To change the mapping mode a new draft version of a flow must be created.
Array-to-array Mapping
Now you can map not just individual objects, but entire arrays in the Integrator mode. Array-to-array mapping was only possible in the Developer mode before. If you saw arrays in Integrator mode, you would get the following message:
Obviously, the mapping result will also be an array. The mapper will take objects from the original array, and map them to the resulting array. Note that mapping of arrays has following limitations:
- It is only possible to map data from objects inside arrays
- It is impossible to map properties from objects located in multiple arrays
- If you map an array in Integrator mode, the mapping will not be saved if you switch to Developer mode, and vice-versa.
Here is an example of array mapping:
