Validation Connector

Overview

The Validation connector checks each payload against a list of field rules. When a rule fails, the configured onInvalid policy decides whether to drop the payload, forward it as-is, or tag it with a validity flag.

Connector Type: MeddleValidation

Configuration

{
  "type": "MeddleValidation",
  "config": {
    "onInvalid": "drop",
    "rules": [
      { "key": "temperature", "type": "number",  "required": true,  "min": -40, "max": 150 },
      { "key": "sensor_id",   "type": "string",  "required": true,  "pattern": "^S-[0-9]{4}$" },
      { "key": "active",      "type": "boolean", "required": false }
    ]
  }
}

Configuration Parameters

• rules (required, min 1): List of per-field rules
• onInvalid (required): Behaviour when at least one rule fails — drop, forward, or tag
• invalidTag: Key written to the payload under tag mode (default _valid)

Rule Fields

• key (required): Payload field name to check
• type (required): string, number, or boolean
• required: When true, an absent field fails the rule. When false, an absent field skips the rule.
• min, max: Inclusive numeric range (only valid for type: number)
• pattern: Regular expression (only valid for type: string)

Type Checks

• number — Accepts any numeric value coercible to float64 (int, float, etc.). Type mismatch is reported as an error.
• string — Accepts Go string. If pattern is set, the value must match.
• boolean — Accepts Go bool.

Range and Pattern

min / max are inclusive. A temperature of exactly -40 passes min: -40. Regex patterns are compiled once at startup with Go’s RE2 syntax; invalid patterns prevent the connector from starting.

On Invalid Policies

Drop Mode

{ "onInvalid": "drop" }

Failing payloads are silently discarded (errors still appear on the error channel). Only valid payloads are forwarded.

Forward Mode

{ "onInvalid": "forward" }

Every payload is forwarded regardless of validity. Use this for monitoring only.

Tag Mode

{
  "onInvalid": "tag",
  "invalidTag": "is_valid"
}

Invalid payloads are forwarded with the tag key set to false. Valid payloads pass through unchanged. If invalidTag is omitted, the key defaults to _valid.

Use Cases

1. Reject malformed sensor readings before they reach storage
2. Enforce ID conventions with regex patterns (e.g. asset tags)
3. Range gating for physical sensors with known operating envelopes
4. Soft validation with tag mode for downstream Router-based handling

Example: Strict Drop

{
  "type": "MeddleValidation",
  "config": {
    "onInvalid": "drop",
    "rules": [
      { "key": "device_id",   "type": "string",  "required": true, "pattern": "^DEV-[A-Z0-9]+$" },
      { "key": "temperature", "type": "number",  "required": true, "min": -20, "max": 120 },
      { "key": "online",      "type": "boolean", "required": true }
    ]
  }
}

Any payload missing a field, failing the regex, or with temperature outside [-20, 120] is dropped.

Example: Tag and Route

{
  "type": "MeddleValidation",
  "config": {
    "onInvalid": "tag",
    "invalidTag": "is_valid",
    "rules": [
      { "key": "pressure", "type": "number", "required": true, "min": 0, "max": 200 }
    ]
  }
}

Combine with a downstream Router that branches on is_valid to send clean and dirty payloads to different sinks.

Best Practices

• Use drop for hard schemas at the boundary of trusted/untrusted data
• Use tag + Router when you want to inspect or quarantine invalid payloads rather than lose them
• Keep regex patterns anchored (^...$) to avoid accidental partial matches
• Set tight numeric ranges based on physical/operational limits — this catches sensor faults early
• Place Validation near the start of pipelines so downstream connectors can assume well-formed payloads

Related Connectors

• Router - Branch on the _valid tag
• Transform - Derive or rename fields before validation
• Filter - Drop unwanted keys