File Format Connector

Overview

The File Format connectors parse and serialize structured text formats embedded inside payload fields. Each connector reads from inputKey and writes the result to outputKey; all other payload fields are passed through unchanged.

Connector Types: CsvProcessor, XmlProcessor

CSV

Parse Mode

Reads a CSV string from inputKey and writes a structured value to outputKey.

{
  "type": "CsvProcessor",
  "config": {
    "direction": "parse",
    "delimiter": ",",
    "hasHeader": true,
    "inputKey": "raw_csv",
    "outputKey": "rows"
  }
}

With hasHeader: true, output is an array of map[string]string keyed by the header row. With hasHeader: false, output is an array of arrays ([][]string).

Serialize Mode

Reads an array-of-arrays from inputKey and writes the CSV string to outputKey.

{
  "type": "CsvProcessor",
  "config": {
    "direction": "serialize",
    "delimiter": ";",
    "inputKey": "rows",
    "outputKey": "csv_text"
  }
}

Configuration Parameters

• direction (required): parse or serialize
• inputKey (required): Payload key holding the source value
• outputKey (required): Payload key to write the result to
• delimiter: Single-character field delimiter (default ,)
• hasHeader: When true during parse, treats the first row as column names

XML

Parse Mode

Reads an XML string from inputKey and writes a nested map to outputKey. Text content is stored under the _text key when elements contain mixed children.

{
  "type": "XmlProcessor",
  "config": {
    "direction": "parse",
    "inputKey": "raw_xml",
    "outputKey": "parsed"
  }
}

Serialize Mode

Reads a map from inputKey and writes an XML string to outputKey, wrapped in rootElement.

{
  "type": "XmlProcessor",
  "config": {
    "direction": "serialize",
    "inputKey": "data",
    "outputKey": "xml_text",
    "rootElement": "report"
  }
}

Configuration Parameters

• direction (required): parse or serialize
• inputKey (required): Payload key holding the source value
• outputKey (required): Payload key to write the result to
• rootElement: Root tag wrapping serialized output (default root)

Use Cases

1. Ingest legacy CSV exports from MES or SCADA systems
2. Bridge XML-based industrial protocols to JSON-native downstream connectors
3. Emit CSV files to S3 or local storage for reporting
4. Inline parsing of CSV/XML payloads received over MQTT or HTTP

Example: CSV Ingest Pipeline

{
  "type": "CsvProcessor",
  "config": {
    "direction": "parse",
    "delimiter": ",",
    "hasHeader": true,
    "inputKey": "body",
    "outputKey": "records"
  }
}

Combined with an HTTP source, this turns an incoming CSV upload into a structured records array that can be fed into Transform, Validation, or a database sink.

Best Practices

• For CSV with non-standard delimiters (;, \t, |), set delimiter explicitly
• Always set hasHeader: true when downstream connectors reference columns by name
• For deeply nested XML, follow the parser with Transform using JSONPath to extract specific paths
• Keep raw text in a dedicated key (e.g. raw_csv) so the original is preserved if a downstream stage fails

Related Connectors

• Transform - Extract values via JSONPath after parsing
• Reshape - Rename or remap parsed fields
• Protocol Translator - Convert between structured formats