Flat files in Mule 4

January 12, 2022

What are flat files?

A flat file, also known as a text database, is a type of database that stores data in a plain text format.

Flat files present complex hierarchical structural data in a record-based storage format. Unlike XML, flat files don’t embed structural data (metadata) within the data. The data in the flat file is “flattened” by removing the hierarchical relationship between records, leaving the records intact as a single logical record of application data.

DataWeave uses a YAML format flat file definition (FFD) to represent flat file schemas. The FFD format is flexible enough to support a range of use cases but is based around the concepts of elements, composites, segments, groups and structures.

Schemas must be written in Flat File Schema Language and by convention use an .ffd extension.

What is a flat file schema?

A flat file schema is the blueprint that contains the instructions for parsing or creating a flat file.

Components in flat file schema

Element — a basic data item that has an associated type and fixed width, along with formatting options for how the data value is read and written.
Composite (optional) — a group of elements that can also include other child composites.
Segment — a line of data, or a record, made up of any number of elements and/or composites that might be repeated.
Group (optional) — several segments grouped together that can also include other child groups.
Structure — a hierarchical organization of segments that requires each segment to have unique identifier codes as part of their data.

Types of flat file segments

Single segment
Multisegment

Single segment

If you’re only working with one type of record, your FFD needs a segment definition only for that record type.

Example of a single segment flat file

Let’s convert a simple CSV file into a flat file with help of flat file schema.

Drag a Transform Message Component and define the input metadata according to your input.

Select type as Fixed Width. It will create one file with .ffd extension in the build path.

EmployeeFile.ffd file

form: FIXEDWIDTH
id: ‘EmployeeFile’
name: ‘EmployeeFile’
values:
– { name: ’empid’, usage: M, type: String, length: 10 }
– { name: ’empname’, usage: M, type: String, length: 25 }
– { name: ’empdesignation’, usage: M, type: String, length: 25 }
– { name: ’empage’, usage: M, type: String, length: 3 }
– { name: ’empsalary’, usage: M, type: String, length: 10 }

In the Transform Message, you need to use attribute schemaPath along with the .ffd file.

Multisegment

If you’re working with multiple types of records in the same transformation, you need to use a structure definition that controls how these different records are combined.

Example of multisegment flat file

Let’s convert a simple XML file into a flat file with help of flat file schema.

Drag a Transform Message Component and define the input metadata according to your input and the output metadata.

Input metadata

Employee.ffd file

form: FLATFILE
structures:
– id: ‘BatchReq’
name: BatchRequest
data:
– {idRef: ‘RQH’}
– groupId: ‘Batch’
count: ‘>1’
items:
– {idRef: ‘BCH’}
– {idRef: ‘ESN’, count: ‘>1’}
– {idRef: ‘RQF’}

segments:

– id: ‘RQH’
name: “Request File Header Record”
values:
– {name: “Record Type”, type: String, length: 3, tagValue: ‘RQH’ }
– {name: “File Creation Date”, type: String, length: 8 }
– {name: “File Creation Time”, type: String, length: 6 }

– id: ‘BCH’
name: “Batch Header Record”
values:
– {name: “Record Type”, type: String, length: 4, tagValue: ‘BCH’ }
– {name: “CompanyName”, type: String, length: 20 }
– {name: “Capacity”, type: String, length: 6 }
– {name: “Technology”, type: String, length: 25 }
– {name: “Location”, type: String, length: 15 }
– {name: “HeadQuarter”, type: String, length: 15 }

– id: ‘ESN’
name: “Employees Details”
values:
– {name: “Record Type”, type: String, length: 4, tagValue: ‘ESN’ }
– {name: “EmployeeName”, type: String, length: 20 }
– {name: “Age”, type: String, length: 3 }
– {name: “Designation”, type: String, length: 20 }
– {name: “JobLocation”, type: String, length: 15 }
– {name: “HomeTown”, type: String, length: 15 }

– id: ‘RQF’
name: “Request File Footer Record”
values:
– {name: “Record Type”, type: String, length: 3, tagValue: ‘RQF’ }
– {name: “Total Employees”, type: String, length: 5 }

Output metadata

In the Transform Message Component, you need to do the mapping according to the expected output.