Mapping Data
This article shows how to map the data between integration components to keep them in sync.
An integration flow on elastic.io platform must have at least two components, one trigger and one action. Trigger emits a data for the action component to receive and process it. Between these two components seats the elastic.io data mapper, which maps or matches the incoming data to the specific fields where the next component expects them to receive.
To understand how the data-mapping works in practice visit our tutorials and platform feature sections. We recommend starting from steps-by-step instructions in how to create your first integration flow and creating a webhook flow followed by the data samples articles as an introduction to the data-mapping.
If you have already followed the tutorials you realise that the data-mapping on elastic.io platform is an important part of the integration process which warrants detailed explanation in its own.
Mapping simple values
Let us consider a simple scenario when we have an incoming data through the
Webhook component like the following JSON structure.
{
"name": "Gromit",
"status": "Sold"
}
You can add this structure manually using the appropriate function:
We intend to map these values into outgoing fields in the Petstore component. Let us jump into the integration flow design right at the mapping stage.
The screenshot above shows the stage in integration flow designer where you can do the actual mapping or matching of the values. Here we choose to create a Pet and need to provide values for Name and Status required fields (red asterisks) to proceed further.
Here we will map these required fields using incoming data from the previous step(s).
To map the Name field click on cross to open a drop-down menu (1), choose
the step you need (2) and select the matching value from the provided values (3) .
For example, our incoming data has a field name which we match with the Name
field from the Petstore component.
The screenshot above shows the successful mapping result which is Gromit. You
have to click to the Eye button here which shows the successful evaluation result.
JSONata Mode
Before going further we can check the JSONata Mode here (1). Here you can look under the hood of the JSONata powered mapper and see the list (2) of expressions and functions you can use. We have already used them in the how to create your first integration flow tutorial.
We can match the Status field with the incoming status value as well to
complete the mapping and go forward. Using the Status field as an example,
you can see that there are fields that accept fixed values. In this case,
the Petstore component accepts: available, pending and sold values.
If you enter an incorrect value, the platform user field as well as the step number will be highlighted in orange:
Note During the flow building process the platform evaluates mapping expressions using samples. During the component execution it evaluates these expressions based on the incoming data which can differ from the presented sample. Use data samples as close to your real data as possible to avoid inconsistencies in outcome.
Required and optional fields
There is a way to filter mapping view to see mandatory fields only by hiding optional fields:
Developer mode
In the previous section you learned how to map data between two steps of an integration flow using a JSONata expression. This is a convenient approach, however, the graphical mapping in the Integrator mode has certain limitations.
The following short animation demonstrates how to switch to the Developer mode.
In the Developer mode you can write a JSON containing JSONata expressions for the entire input object instead of defining expressions for each individual field as in case of the Integrator mode.
Here you can write your own JSON if you know the meta-model of input data structure or open the Fields menu which lists all incoming meta-model fields. For example the Petstore component has two fields to choose from:
The Fields menu will contain more values for components with extensive meta-model structure such as the case of the Salesforce component:
In this long list you can Search to find the parameter you need and click on plus sign to add to the JSON structure.
Please Note The platform user interface assumes that you know what you are mapping when you switch to Developer mode. It will allow an empty JSON
{}as an input structure. You are responsible to ensure the validity of your mapping.
To avoid unexpected results we recommend to:
- Include and map the Required fields to avoid unexpected behaviour from the API provider.
- Make sure to use correct data Type while filling-in the parameter values.
When you write a JSONata expression which contains property not included in input meta-data the platform user interface will block switching back to Integrator mode.
Evaluate results
You are already familiar with evaluation function. In the Developer mode it evaluates the whole JSON structure in input field rather than individual fields like in the Integrator mode.
The following animation shows how you can use the Fields menu to choose values, fill-in, get their property values and Evaluate the result.
We can see in the evaluation stage how the platform interface replaces the property
values name and status to their real values from the incoming payload.
When you enter an incorrect JSON and press Evaluate the platform UI will check it and you will get an error in the Mapping result section:
Note You can switch between Developer and Integrator modes during the design of integration flow but not after you publish it. To change the mapping mode, create a new draft version of a flow.
Array-to-array Mapping
You can map the incoming array values with an outgoing array fields, we call this an array-to-array mapping.
You can use both Integrator or Developer modes to map array structure. However, when you have complex array-to-array case we recommend using the developer mode since the Integrator mode has certain limitations.
- You can map data from objects inside the arrays.
- You can’t map properties from objects located in multiple arrays.
- Due to complexity and limitations, when you map array-to-array Integrator mode, the mapping will not be saved if you switch to Developer mode and, vice-versa.
When in Integrator mode, the system checks if the incoming array has properties to map. If not then the previous step is disabled.
Alternatively, when there are array parameters to map you can access the incoming data fields.
For complex cases of array-to-array mapping we recommend switching to Developer mode where you can write extensive JSONata expressions and use data from different arrays simultaneously.