Scatter-Gather router in Mule 4, part 1

  • March 17, 2020

What is Scatter-Gather?

The Scatter-Gather component is a routing event processor that processes a Mule event through different parallel processing routes that contain different event processors.

Each route receives a reference to the Mule event and executes a sequence of one or more event processors. Each of these routes uses a separate thread to execute the event processors, and the resulting Mule event can be either the same Mule event without modifications or a new Mule event with its own payload, attributes and variables. The Scatter-Gather component then combines the Mule events returned by each processing route into a new Mule event that is passed to the next event processor only after every route completes successfully.

The Scatter-Gather component executes each route in parallel, not sequentially. Parallel execution of routes can greatly increase the efficiency of your Mule application and may provide more information than sequential processing.

The following diagram details the behavior of the Scatter-Gather component:

Scatter-Gather router1

Fig a) Scatter Gather Component

  • The Scatter-Gather component receives a Mule event and sends a reference of this Mule event to each processing route.
  • Each of the processing routes starts executing in parallel. After all processors inside a route finish processing, the route returns a Mule event, which can be either the same Mule event without modifications or a new Mule event created by the processors in the route as a result of the modifications applied.
  • After all processing routes have finished execution, the Scatter-Gather component creates a new Mule event that combines all resulting Mule events from each route, and then passes the new Mule event to the next component in the flow.

Variable propagation in Scatter-Gather

Every route starts with the same initial variable values. Modifications to a variable within a specific route do not affect other routes. So, if a variable is added or modified in one route, then, after aggregation, the value is defined by that route. If a variable is added or modified by more than one route, the value is added to a list of all the values defined for that variable within all the routes.

Below are the flow xml and screenshot of the flow.

Scatter-Gather router2

Fig b) Scatter-Gather Variable Flow

Scatter-Gather router3

Fig c) Variables in Scatter-Gather

After aggregation, the variables are:

{var1: “var1”, var2: “newValue”, var3: [“appleVal, bananaVal, otherVal”], var4: “val4”}

Output combining all routes

{
  "0": {
    "inboundAttachmentNames": [],
    "exceptionPayload": null,
    "inboundPropertyNames": [],
    "outboundAttachmentNames": [],
    "payload": "",
    "outboundPropertyNames": [],
    "attributes": {
      "headers": {
        "user-agent": "PostmanRuntime/7.22.0",
        "accept": "*/*",
        "cache-control": "no-cache",
        "postman-token": "47e20bb3-3726-4437-8b86-8ef55cd27d71",
        "host": "localhost:8081",
        "accept-encoding": "gzip, deflate, br",
        "connection": "keep-alive"
      },
      "clientCertificate": null,
      "method": "GET",
      "scheme": "http",
      "queryParams": {},
      "requestUri": "/variable",
      "queryString": "",
      "version": "HTTP/1.1",
      "maskedRequestPath": null,
      "listenerPath": "/variable",
      "relativePath": "/variable",
      "localAddress": "127.0.0.1:8081",
      "uriParams": {},
      "rawRequestUri": "/variable",
      "rawRequestPath": "/variable",
      "remoteAddress": "127.0.0.1:60309",
      "requestPath": "/variable"
    }
  },
  "1": {
    "inboundAttachmentNames": [],
    "exceptionPayload": null,
    "inboundPropertyNames": [],
    "outboundAttachmentNames": [],
    "payload": "",
    "outboundPropertyNames": [],
    "attributes": {
      "headers": {
        "user-agent": "PostmanRuntime/7.22.0",
        "accept": "*/*",
        "cache-control": "no-cache",
        "postman-token": "47e20bb3-3726-4437-8b86-8ef55cd27d71",
        "host": "localhost:8081",
        "accept-encoding": "gzip, deflate, br",
        "connection": "keep-alive"
      },
      "clientCertificate": null,
      "method": "GET",
      "scheme": "http",
      "queryParams": {},
      "requestUri": "/variable",
      "queryString": "",
      "version": "HTTP/1.1",
      "maskedRequestPath": null,
      "listenerPath": "/variable",
      "relativePath": "/variable",
      "localAddress": "127.0.0.1:8081",
      "uriParams": {},
      "rawRequestUri": "/variable",
      "rawRequestPath": "/variable",
      "remoteAddress": "127.0.0.1:60309",
      "requestPath": "/variable"
    }
  },
  "2": {
    "inboundAttachmentNames": [],
    "exceptionPayload": null,
    "inboundPropertyNames": [],
    "outboundAttachmentNames": [],
    "payload": "",
    "outboundPropertyNames": [],
    "attributes": {
      "headers": {
        "user-agent": "PostmanRuntime/7.22.0",
        "accept": "*/*",
        "cache-control": "no-cache",
        "postman-token": "47e20bb3-3726-4437-8b86-8ef55cd27d71",
        "host": "localhost:8081",
        "accept-encoding": "gzip, deflate, br",
        "connection": "keep-alive"
      },
      "clientCertificate": null,
      "method": "GET",
      "scheme": "http",
      "queryParams": {},
      "requestUri": "/variable",
      "queryString": "",
      "version": "HTTP/1.1",
      "maskedRequestPath": null,
      "listenerPath": "/variable",
      "relativePath": "/variable",
      "localAddress": "127.0.0.1:8081",
      "uriParams": {},
      "rawRequestUri": "/variable",
      "rawRequestPath": "/variable",
      "remoteAddress": "127.0.0.1:60309",
      "requestPath": "/variable"
    }
  }
}

Flow XML

<flow name="scattergathervariableflow" doc:id="123346f7-4e48-4b24-b094-2e15b039af1f" >
<http:listener doc:name="Listener" doc:id="17cb4a5b-a8d0-4614-88aa-07a6f2ad461f" config-ref="HTTP_Listener_config" path="/variable"/>
<set-variable variableName="var1" value="var1" doc:name="Set Variable var1"/>
<set-variable variableName="var2" value="var2" doc:name="Set Variable var2"/>
<scatter-gather doc:name="Scatter-Gather" doc:id="1d2a6fc2-64cb-47df-bc0d-568a32ea57da" >
	<route >
		<set-variable variableName="var2" value="newValue" doc:name="Set Variable var2"/>
        		<set-variable variableName="var3" value="appleVal" doc:name="Set Variable var3"/>
	</route>
	<route >
		<set-variable variableName="var3" value="bananaVal" doc:name="Set Variable var3"/>
	</route>
	<route >
		<set-variable variableName="var3" value="otherVal" doc:name="Set Variable var3"/>
        <set-variable variableName="var4" value="val4" doc:name="Set Variable var4"/>
	</route>
</scatter-gather>
<ee:transform doc:name="Transform Message" doc:id="00e134d1-e38c-4764-be66-b062c8a4f7e9" >
<ee:message >
			<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]>
</ee:set-payload>
	</ee:message>
</ee:transform>
</flow>

— By Gaurav Dubey