Version: 1.5.1

Azure Event Hubs

Microsoft Azure Real-time Messaging

Synopsis

Creates a target that sends processed messages to Azure Event Hubs with support for multiple authentication methods, batch processing, and automatic retry mechanisms. Provides high-throughput event streaming to Azure Event Hubs for real-time analytics and downstream processing.

Schema

- name: <string>
  description: <string>
  type: eventhubs
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    client_connection_string: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    namespace: <string>
    event_hub: <string>
    partition:
      id: <string>
      key: <string>
    field_format: <string>
    max_bytes: <numeric>
    max_events: <numeric>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
      insecure_skip_verify: <boolean>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `eventhubs`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Connection

EventHubs target supports two authentication methods:

Method 1: Connection String Authentication

Field	Required	Default	Description
`client_connection_string`	Y*		Event Hubs connection string (required if not using method 2)
`event_hub`	Y		Event hub name to send messages to

Method 2: Service Principal Authentication

Field	Required	Description
`tenant_id`	Y*	Azure tenant ID (required if not using connection string)
`client_id`	Y*	Azure service principal client ID
`client_secret`	Y*	Azure service principal client secret
`namespace`	Y*	Event Hubs namespace (required if not using connection string)
`event_hub`	Y	Event hub name to send messages to

* = Conditionally required (see authentication methods above)

Partition Configuration

Field	Required	Default	Description
`partition.id`	N*	-	Specific partition ID to send messages to
`partition.key`	N*	-	Partition key for message routing

* = Mutually exclusive - use either partition.id OR partition.key, not both

Message Configuration

Field	Required	Default	Description
`field_format`	N	-	Data normalization format. See applicable Normalization section

Performance

Field	Required	Default	Description
`max_bytes`	N	`0`	Maximum batch size in bytes (0 for unlimited)
`max_events`	N	`1000`	Maximum number of events per batch

TLS

Field	Required	Default	Description
`tls.status`	N	`false`	Enable TLS encryption
`tls.cert_name`	N*	-	TLS certificate file path (required if TLS enabled)
`tls.key_name`	N*	-	TLS private key file path (required if TLS enabled)
`tls.insecure_skip_verify`	N	`false`	Skip TLS certificate verification (NOT recommended for production)

* = Conditionally required (only when tls.status: true)

note

TLS certificate and key files must be placed in the service root directory.

warning

Setting insecure_skip_verify: true disables certificate validation and should only be used for testing/development environments.

Scheduler

Field	Required	Default	Description
`interval`	N	realtime	Execution frequency. See Interval for details
`cron`	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
`debug.status`	N	`false`	Enable debug logging
`debug.dont_send_logs`	N	`false`	Process logs but don't send to target (testing)

Details

The EventHubs target sends processed messages to Azure Event Hubs for real-time event streaming and analytics. It supports automatic batching for optimal performance and multiple authentication methods for flexible deployment scenarios.

Messages are sent with automatic partition distribution unless a specific partition ID or key is provided. The target handles connection pooling and automatic reconnection on network failures.

Partition Management

You can control message routing to Event Hubs partitions using two mutually exclusive options:

partition.id: Routes all messages to a specific partition by ID (0-based index)
partition.key: Uses a partition key for consistent hashing across partitions

If neither is specified, Event Hubs automatically distributes messages across available partitions using round-robin distribution.

Examples

The following are commonly used configuration types.

Basic with Connection String

Creating a basic EventHubs target with connection string...

- name: basic_eventhubs_target
  type: eventhubs
  properties:
    client_connection_string: "Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=mykey;SharedAccessKey=myvalue"
    event_hub: "processed-logs"
    max_events: 100

Target sends messages to Event Hubs in batches...

{
  "timestamp": "2024-01-15T10:30:00Z",
  "host": "server01",
  "message": "User authentication successful",
  "severity": "info"
}

Service Principal Authentication

Using service principal authentication for secure access...

- name: sp_eventhubs_target
  type: eventhubs
  properties:
    tenant_id: "12345678-1234-1234-1234-123456789012"
    client_id: "87654321-4321-4321-4321-210987654321"
    client_secret: "${AZURE_CLIENT_SECRET}"
    namespace: "production-namespace"
    event_hub: "security-events"
    max_events: 250

High-Throughput Configuration

Optimizing for high-volume message sending...

- name: high_volume_target
  type: eventhubs
  properties:
    client_connection_string: "${EVENTHUBS_CONNECTION_STRING}"
    event_hub: "high-volume-events"
    max_events: 500
    max_bytes: 1048576

Partition Key Configuration

Using partition key for consistent message routing...

- name: partitioned_key_target
  type: eventhubs
  properties:
    tenant_id: "${AZURE_TENANT_ID}"
    client_id: "${AZURE_CLIENT_ID}"
    client_secret: "${AZURE_CLIENT_SECRET}"
    namespace: "analytics-namespace"
    event_hub: "partitioned-logs"
    partition:
      key: "source_system"
    max_events: 200

Partition ID Configuration

Sending all messages to a specific partition...

- name: partitioned_id_target
  type: eventhubs
  properties:
    client_connection_string: "${EVENTHUBS_CONNECTION_STRING}"
    event_hub: "specific-partition"
    partition:
      id: "0"
    max_events: 150

Secure Connection with TLS

Implementing TLS encryption for secure transmission...

- name: secure_eventhubs_target
  type: eventhubs
  properties:
    client_connection_string: "${EVENTHUBS_CONNECTION_STRING}"
    event_hub: "secure-events"
    max_events: 150
    tls:
      status: true
      cert_name: "eventhubs.crt"
      key_name: "eventhubs.key"

Pipeline Processing

Applying post-processing pipelines before sending...

- name: pipeline_eventhubs_target
  type: eventhubs
  pipelines:
    - format_timestamp
    - add_metadata
    - validate_schema
  properties:
    client_connection_string: "${EVENTHUBS_CONNECTION_STRING}"
    event_hub: "processed-events"
    max_events: 100

Synopsis​

Schema​

Configuration​

Connection​

Partition Configuration​

Message Configuration​

Performance​

TLS​

Scheduler​

Debug Options​

Details​

Partition Management​

Examples​

Basic with Connection String​

Service Principal Authentication​

High-Throughput Configuration​

Partition Key Configuration​

Partition ID Configuration​

Secure Connection with TLS​

Pipeline Processing​