Azure Service Bus

Microsoft Azure Message Queue

Synopsis

Creates a target that sends processed messages to Azure Service Bus queues or topics with support for multiple authentication methods, batch processing, and advanced message properties. Provides reliable message delivery to Azure Service Bus for decoupled application architectures and event-driven systems.

Schema

- name: <string>
  description: <string>
  type: azservicebus
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    client_connection_string: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    namespace: <string>
    queue: <string>
    topic: <string>
    session_id: <string>
    message_id: <string>
    correlation_id: <string>
    content_type: <string>
    time_to_live: <numeric>
    max_events: <numeric>
    field_format: <string>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be azservicebus
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Azure Service Bus target supports two authentication methods:

Method 1: Connection String Authentication

Field	Required	Default	Description
client_connection_string	Y*		Service Bus connection string (required if not using method 2)
queue	N**		Queue name to send messages to
topic	N**		Topic name to send messages to

Method 2: Service Principal Authentication

Field	Required	Default	Description
tenant_id	Y*		Azure tenant ID (required if not using connection string)
client_id	Y*		Azure service principal client ID
client_secret	Y*		Azure service principal client secret
namespace	Y*		Service Bus namespace (required if not using connection string)
queue	N**		Queue name to send messages to
topic	N**		Topic name to send messages to

* = Conditionally required (see authentication methods above)

** = Either queue or topic must be specified, but not both

Message Properties

Field	Required	Default	Description
session_id	N	-	Session ID for session-enabled queues/topics
message_id	N	-	Custom message ID
correlation_id	N	-	Correlation ID for request-response patterns
content_type	N	application/json	Message content type
time_to_live	N	-	Message time-to-live in seconds

Performance

Field	Required	Default	Description
max_events	N	1000	Maximum number of messages per batch
field_format	N	-	Data normalization format. See applicable Normalization section

TLS

Field	Required	Default	Description
tls.status	N	false	Enable TLS encryption
tls.cert_name	N*		TLS certificate file path (required if TLS enabled)
tls.key_name	N*		TLS private key file path (required if TLS enabled)

* = Conditionally required (only when tls.status: true)

note

TLS certificate and key files must be placed in the service root directory.

Scheduler

Field	Required	Default	Description
interval	N	realtime	Execution frequency. See Interval for details
cron	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Azure Service Bus target sends processed messages to Azure Service Bus queues or topics for reliable message delivery in distributed systems. It supports automatic batching for optimal performance, advanced message properties for complex workflows, and multiple authentication methods for flexible deployment scenarios.

Messages are sent in batches to improve throughput and reduce network overhead. The target handles connection pooling and automatic reconnection on network failures.

Queues vs Topics

Queues

• Point-to-point messaging
• Single consumer per message
• FIFO ordering (optional)
• Ideal for task distribution and load leveling

Topics

• Publish-subscribe messaging
• Multiple subscribers per message
• Message filtering with subscriptions
• Ideal for event broadcasting and fan-out scenarios

Session Support

Session-enabled queues and topics provide FIFO guarantees for messages with the same session ID. Use the session_id property to enable session-based processing for ordered message delivery.

Message Properties

The target supports setting custom message properties:

• Message ID: Unique identifier for tracking and deduplication
• Correlation ID: Links related messages in request-response patterns
• Content Type: MIME type of message body
• Time to Live: Automatic message expiration

Batch Processing

Messages are accumulated in memory and sent in batches when the batch size limit is reached or during finalization. The maximum batch size is configurable via the max_events parameter.

Dead Letter Queue

Azure Service Bus automatically moves messages to the dead letter queue when they exceed max delivery count or expire. Configure these settings in the Azure portal or via infrastructure as code.

At-Least-Once Delivery

Service Bus guarantees at-least-once delivery. Messages may be delivered more than once in case of network failures or processing errors. Design your message handlers to be idempotent.

Examples

The following are commonly used configuration types.

Basic with Connection String

Creating a basic Service Bus queue target with connection string...	- name: basic_servicebus_queue type: azservicebus properties: client_connection_string: "Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=mykey;SharedAccessKey=myvalue" queue: "processed-logs" content_type: "application/json" max_events: 1000
Target sends JSON messages to Service Bus queue in batches...	{ "timestamp": "2024-01-15T10:30:00Z", "host": "server01", "message": "User authentication successful", "severity": "info" }

Service Principal Authentication

Using service principal authentication for secure access...

- name: sp_servicebus_topic type: azservicebus properties: tenant_id: "12345678-1234-1234-1234-123456789012" client_id: "87654321-4321-4321-4321-210987654321" client_secret: "${AZURE_CLIENT_SECRET}" namespace: "production-namespace" topic: "security-events" content_type: "application/json" max_events: 500

Session-Enabled Queue

Using sessions for ordered message processing...

- name: session_servicebus_queue type: azservicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "ordered-transactions" session_id: "session-001" content_type: "application/json" max_events: 250

Request-Response Pattern

Using correlation ID for request-response messaging...

- name: correlation_servicebus_queue type: azservicebus properties: tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" namespace: "integration-namespace" queue: "request-processing" message_id: "unique-message-id" correlation_id: "request-correlation-id" content_type: "application/json"

Message Expiration

Setting time-to-live for automatic message expiration...

- name: ttl_servicebus_queue type: azservicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "temporary-notifications" time_to_live: 3600 content_type: "application/json" max_events: 500

Topic with Subscriptions

Publishing messages to a topic for multiple subscribers...

- name: pubsub_servicebus_topic type: azservicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" topic: "application-events" content_type: "application/json" max_events: 1000

Secure Connection with TLS

Implementing TLS encryption for secure transmission...

- name: secure_servicebus_queue type: azservicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "secure-events" content_type: "application/json" max_events: 500 tls: status: true cert_name: "servicebus.crt" key_name: "servicebus.key"

Pipeline Processing

Applying post-processing pipelines before sending...

- name: pipeline_servicebus_queue type: azservicebus pipelines: - format_timestamp - add_metadata - validate_schema properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "processed-events" content_type: "application/json" max_events: 750

Field Normalization

Using field normalization for standard format...

- name: normalized_servicebus_queue type: azservicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "normalized-logs" content_type: "application/json" field_format: "ecs" max_events: 1000

High-Volume Configuration

Optimizing for high-volume message sending...

- name: high_volume_servicebus_queue type: azservicebus properties: tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" namespace: "analytics-namespace" queue: "high-volume-events" content_type: "application/json" max_events: 2000

Troubleshooting

Authentication Errors

If you encounter authentication errors:

• Verify connection string format and credentials
• Check service principal has appropriate permissions
• Ensure namespace name is correct and accessible
• Verify firewall rules allow access from your source

Queue or Topic Not Found

If messages fail to send:

• Verify queue or topic exists in the namespace
• Check queue or topic name spelling
• Ensure queue or topic is not disabled
• Verify you have send permissions

Message Size Exceeded

If you see "message too large" errors:

• Reduce message size before sending
• Check Service Bus tier limits (256 KB for Standard, 1 MB for Premium)
• Consider splitting large messages into smaller chunks

Session Errors

If session-related errors occur:

• Verify queue or topic has sessions enabled
• Ensure session ID is provided when required
• Check session lock duration settings

Batch Limit Exceeded

If batch processing fails:

• Reduce max_events parameter
• Monitor message sizes to prevent batch size overflow
• Check Service Bus quota limits

Connection Failures

If connection failures occur:

• Verify network connectivity to Azure
• Check Service Bus namespace status in Azure portal
• Ensure firewall rules allow outbound connections
• Review Service Bus namespace diagnostic logs