Covered in this article
Related articles

Mapping Data

This article shows how to map the data between integration components to keep them in sync.

An integration flow on elastic.io platform must have at least two components, one trigger and one action. Trigger emits a data for the action component to receive and process it. Between these two components seats the elastic.io data mapper, which maps or matches the incoming data to the specific fields where the next component expects them to receive.

To understand how the data-mapping works in practice visit our tutorials and platform feature sections. We recommend starting from steps-by-step instructions in how to create your first integration flow and creating a webhook flow followed by the data samples articles as an introduction to the data-mapping.

If you have already followed the tutorials you realise that the data-mapping on elastic.io platform is an important part of the integration process which warrants detailed explanation in its own.

Mapping simple values

Let us consider a simple scenario when we have an incoming data through the Webhook component like the following JSON structure.

{
  "name": "Gromit",
  "status": "Sold"
}

You can add this structure manually using the appropriate function:

Mapping: Configure input

We intend to map these values into outgoing fields in the Petstore component. Let us jump into the integration flow design right at the mapping stage.

Mapping: Configure input

The screenshot above shows the stage in integration flow designer where you can do the actual mapping or matching of the values. Here we choose to create a Pet and need to provide values for Name and Status required fields (red asterisks) to proceed further.

Here we will map these required fields using incoming data from the previous step(s).

Mapping: Selecting drop-down

To map the Name field click on cross to open a drop-down menu (1), choose the step you need (2) and select the matching value from the provided values (3) . For example, our incoming data has a field name which we match with the Name field from the Petstore component.

Mapping result of the field

The screenshot above shows the successful mapping result which is Gromit. You have to click to the Eye button here which shows the successful evaluation result.

JSONata Mode

Expressions tab

Before going further we can check the JSONata Mode here (1). Here you can look under the hood of the JSONata powered mapper and see the list (2) of expressions and functions you can use. We have already used them in the how to create your first integration flow tutorial.

We can match the Status field with the incoming status value as well to complete the mapping and go forward. Using the Status field as an example, you can see that there are fields that accept fixed values. In this case, the Petstore component accepts: available, pending and sold values.

Expressions tab

If you enter an incorrect value, the platform user field as well as the step number will be highlighted in orange:

Expressions tab

Note During the flow building process the platform evaluates mapping expressions using samples. During the component execution it evaluates these expressions based on the incoming data which can differ from the presented sample. Use data samples as close to your real data as possible to avoid inconsistencies in outcome.

Required and optional fields

There is a way to filter mapping view to see mandatory fields only by hiding optional fields:

Hiding and showing optional fields

Developer mode

In the previous section you learned how to map data between two steps of an integration flow using a JSONata expression. This is a convenient approach, however, the graphical mapping in the Integrator mode has certain limitations.

The following short animation demonstrates how to switch to the Developer mode.

Switch to Developer mode

In the Developer mode you can write a JSON containing JSONata expressions for the entire input object instead of defining expressions for each individual field as in case of the Integrator mode.

Here you can write your own JSON if you know the meta-model of input data structure or open the Fields menu which lists all incoming meta-model fields. For example the Petstore component has two fields to choose from:

Field menu: incoming meta-model fields

The Fields menu will contain more values for components with extensive meta-model structure such as the case of the Salesforce component:

Field menu: complex incoming meta-model fields

In this long list you can Search to find the parameter you need and click on plus sign to add to the JSON structure.

Please Note The platform user interface assumes that you know what you are mapping when you switch to Developer mode. It will allow an empty JSON {} as an input structure. You are responsible to ensure the validity of your mapping.

To avoid unexpected results we recommend to:

  • Include and map the Required fields to avoid unexpected behaviour from the API provider.
  • Make sure to use correct data Type while filling-in the parameter values.

When you write a JSONata expression which contains property not included in input meta-data the platform user interface will block switching back to Integrator mode.

Mapping when the property not included in meta-data.

Evaluate results

You are already familiar with evaluation function. In the Developer mode it evaluates the whole JSON structure in input field rather than individual fields like in the Integrator mode.

The following animation shows how you can use the Fields menu to choose values, fill-in, get their property values and Evaluate the result.

Using Fields menu in Developer mode

We can see in the evaluation stage how the platform interface replaces the property values name and status to their real values from the incoming payload.

When you enter an incorrect JSON and press Evaluate the platform UI will check it and you will get an error in the Mapping result section:

Error in input JSON

Note You can switch between Developer and Integrator modes during the design of integration flow but not after you publish it. To change the mapping mode, create a new draft version of a flow.

Array-to-array Mapping

You can map the incoming array values with an outgoing array fields, we call this an array-to-array mapping.

You can use both Integrator or Developer modes to map array structure. However, when you have complex array-to-array case we recommend using the developer mode since the Integrator mode has certain limitations.

  • You can map data from objects inside the arrays.
  • You can’t map properties from objects located in multiple arrays.
  • Due to complexity and limitations, when you map array-to-array Integrator mode, the mapping will not be saved if you switch to Developer mode and, vice-versa.

When in Integrator mode, the system checks if the incoming array has properties to map. If not then the previous step is disabled.

Disabled step in array mapping

Alternatively, when there are array parameters to map you can access the incoming data fields.

Array mapping

For complex cases of array-to-array mapping we recommend switching to Developer mode where you can write extensive JSONata expressions and use data from different arrays simultaneously.