S3 Connector

Overview

The S3 connector reads and writes JSON objects on AWS S3 and any S3-compatible service (MinIO, Ceph, Cloudflare R2, Backblaze B2).

Connector Types:

S3Reader - Poll a bucket prefix and emit each new object as a payload
S3Writer - Write each payload as an S3 object with a templated key

S3 Reader

Poll a bucket and emit each new object:

{
  "type": "S3Reader",
  "config": {
    "region": "us-east-1",
    "bucket": "meddle-ingest",
    "prefix": "incoming/",
    "accessKey": "AKIAIOSFODNN7EXAMPLE",
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "pollingRate": 5000
  }
}

Delete After Read

Remove each object once it has been emitted, useful for inbox-style ingestion:

{
  "type": "S3Reader",
  "config": {
    "region": "eu-west-1",
    "bucket": "meddle-inbox",
    "prefix": "events/",
    "pollingRate": 10000,
    "deleteAfter": true
  }
}

S3-Compatible Endpoint (MinIO, R2)

Set endpoint to target a non-AWS service. Path-style addressing is enabled automatically when endpoint is set:

{
  "type": "S3Reader",
  "config": {
    "endpoint": "https://minio.example.com:9000",
    "region": "us-east-1",
    "bucket": "sensor-data",
    "prefix": "raw/",
    "accessKey": "minio_access_key",
    "secretKey": "minio_secret_key",
    "pollingRate": 5000
  }
}

S3 Writer

Write each payload as a new object using a Go template for the key:

{
  "type": "S3Writer",
  "config": {
    "region": "us-east-1",
    "bucket": "meddle-archive",
    "keyTemplate": "events/{{.deviceId}}/{{.timestamp}}.json",
    "accessKey": "AKIAIOSFODNN7EXAMPLE",
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  }
}

S3-Compatible Endpoint

{
  "type": "S3Writer",
  "config": {
    "endpoint": "https://minio.example.com:9000",
    "region": "us-east-1",
    "bucket": "processed",
    "keyTemplate": "out/{{.line}}/{{.timestamp}}.json",
    "accessKey": "minio_access_key",
    "secretKey": "minio_secret_key",
    "contentType": "application/json"
  }
}

Configuration Parameters

region: AWS region (required for both reader and writer)
bucket: S3 bucket name (required)
endpoint: (Optional) Custom endpoint URL for S3-compatible services; enables path-style addressing
accessKey: (Optional) AWS access key ID; falls back to the default AWS credential chain when omitted
secretKey: (Optional) AWS secret access key
prefix: (Reader) Object key prefix to filter listings
pollingRate: (Reader, required) Polling interval in milliseconds
deleteAfter: (Reader) Delete each object after it has been read
keyTemplate: (Writer, required) Go template used to build the object key for each payload
contentType: (Writer) MIME type for written objects (default: application/json)

Key Templates

The writer’s keyTemplate is a Go text/template string evaluated against the payload. Top-level payload fields are referenced as {{.fieldName}}:

Template	Resulting key (sample payload)
`events/{{.deviceId}}/{{.timestamp}}.json`	`events/sensor-42/1737036000.json`
`raw/{{.year}}/{{.month}}/{{.id}}.json`	`raw/2026/05/abc-123.json`
`flat-{{.id}}.json`	`flat-abc-123.json`

Ensure every field referenced in the template exists in the payload, otherwise the template renders an empty string for missing keys and may collide on overwrite.

Credentials Resolution

When both accessKey and secretKey are set, the connector uses static credentials.
When omitted, the AWS SDK default chain applies: environment variables, EC2/ECS task role, shared ~/.aws/credentials profile, etc.

Reader Deduplication

The reader keeps an in-memory set of object keys it has already emitted, so each object is processed only once per Meddle process lifetime. Combine with deleteAfter: true for durable inbox-style ingestion across restarts.

MongoDB - Document storage alternative
InfluxDB - Time-series storage alternative
HTTP - Push data to REST APIs instead