Connectors Overview

What are Connectors?

Connectors are the building blocks of Meddle workflows. Each connector is a specialized component that can read data from sources, write data to destinations, or process and transform data in transit.

Connectors are connected together in a visual workflow editor to create powerful data integration pipelines without writing code.

Connector Categories

Meddle provides 18 different connector types organized into three main categories:

🏭 Industrial Connectors (3 types)

Connect to industrial automation systems and PLCs:

OPC UA

Industry-standard protocol for industrial automation. Supports multiple authentication methods and security modes.

Modbus

Serial and TCP communication with PLCs and industrial devices. Supports all register types and data formats.

Siemens S7

Direct communication with Siemens S7-300, S7-400, S7-1200, and S7-1500 PLCs.

🌐 IoT & Database Connectors (5 types)

Integrate with IoT protocols and databases:

MQTT

Publish/subscribe messaging for IoT devices with QoS support.

HTTP/REST

Connect to REST APIs and web services with customizable headers and methods.

InfluxDB

Time-series database optimized for industrial and IoT data.

MongoDB

NoSQL document database for flexible data storage.

SQL

Support for MySQL, PostgreSQL, and SQL Server databases.

⚙️ Processing Connectors (10 types)

Transform, filter, and process data:

Filter

Whitelist or blacklist specific data keys.

Conveyor

Pass data through unchanged for many-to-many routing.

Merge

Combine data from multiple sources with timing or key-based strategies.

Reshape

Rename fields or enrich data with static values.

Trigger

Conditional logic using MXL expressions.

Cron

Schedule data release at specific times.

Auth

JWT authentication and validation.

Anomaly Detection

ML-based anomaly detection using Isolation Forest.

Alert

Send notifications based on conditions.

Chart

Real-time data visualization.

How Connectors Work

Data Format

All connectors exchange data using a standard key-value payload format:

{
  "temperature": 25.5,
  "pressure": 101.3,
  "humidity": 60,
  "status": "active",
  "timestamp": 1234567890
}

This standardized format allows any connector to communicate with any other connector, regardless of the underlying protocol or system.

Connector Configuration

Each connector is configured using JSON with three main sections:

1. Type: The connector type (e.g., OpcuaReader, MqttV3Writer)
2. Config: Connector-specific configuration (endpoints, credentials, etc.)
3. Variables: (Optional) For industrial connectors, defines what data to read/write

Example:

{
  "type": "OpcuaReader",
  "config": {
    "endpoint": "opc.tcp://localhost:4840",
    "pollingRate": 1000
  },
  "variables": [
    {
      "key": "temperature",
      "nodeId": "ns=1;s=Temperature"
    }
  ]
}

Workflow Execution

Connectors are organized into worksheets (workflows) where:

1. Reader connectors collect data from sources
2. Processing connectors transform and filter data
3. Writer connectors send data to destinations

Data flows through connections between connectors, with each connector processing data in parallel for maximum performance.

Common Configuration Patterns

Polling Rate

Many reader connectors support a pollingRate parameter (in milliseconds):

{
  "pollingRate": 1000  // Poll every 1 second
}

Authentication

Connectors that require authentication typically support multiple methods:

{
  "username": "user",
  "password": "pass"
}

Or for token-based auth:

{
  "authToken": "your-token-here"
}

Error Handling

All connectors include built-in error handling and will:

• Log errors with specific error codes
• Continue operation when possible
• Provide detailed error messages for troubleshooting

Performance Considerations

Polling Rates

• Use appropriate pollingRate values to avoid overwhelming systems
• Faster polling = more data but higher CPU/network usage
• Typical values: 100ms (fast), 1000ms (normal), 5000ms (slow)

Data Accumulation

Some connectors accumulate data in memory:

• Merge: Combines data from multiple sources
• Cron: Holds data until scheduled release
• Use maxRetained parameter to limit memory usage

Parallel Execution

• Worksheets execute connectors in parallel
• Multiple worksheets run simultaneously
• Horizontally scaled across CPU cores

Best Practices

1. Start Simple

Begin with basic reader → writer workflows before adding processing connectors.

2. Use Filters Early

Place Filter connectors early in the workflow to reduce payload size and improve performance.

3. Normalize Data

Use Reshape connectors to standardize field names across different sources.

4. Implement Error Handling

Use Trigger connectors to detect error conditions and route to Alert connectors.

5. Schedule Batch Operations

Use Cron connectors for batch processing and scheduled data releases.

6. Secure Your Data

Use Auth connectors to validate JWT tokens in data flows that require authentication.

7. Monitor for Anomalies

Add Anomaly Detection connectors to identify unusual patterns in your data.

Next Steps

Explore the connector documentation by category:

• Industrial Connectors - OPC UA, Modbus, Siemens S7
• IoT & Database Connectors - MQTT, HTTP, databases
• Processing Connectors - Data transformation and logic
• MXL Expression Language - Conditional expressions for Trigger and Alert