Azure Monitor Logs

Microsoft Azure Logging & Monitoring

Synopsis

Creates a target that ingests log messages into Azure Monitor Logs (Log Analytics) workspace tables using Data Collection Rules (DCRs). Supports automatic table discovery, field normalization, and Azure Managed Identity authentication for secure, credential-free access.

tip

For Director Proxy deployment with Managed Identity, see VirtualMetric Director Proxy.

Schema

- name: <string>
  description: <string>
  type: azmonitor
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    function_app: <string>
    function_token: <string>
    rule_id: <string>
    endpoint: <string>
    streams:
      - name: <string>
        rule_id: <string>
    stream: <string[]>
    buffer_size: <numeric>
    batch_size: <numeric>
    keep_phantom_fields: <boolean>
    drop_unknown_stream_events: <boolean>
    cache:
      timeout: <numeric>
    field_format: <string>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Core Settings

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be azmonitor
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Authentication

Field	Required	Default	Description
tenant_id	N*	-	Azure tenant ID (required for direct authentication)
client_id	N*	-	Azure client ID (required for direct authentication)
client_secret	N*	-	Client secret (required for direct authentication)
function_app	N*	-	Director Proxy endpoint URL (required for proxy forwarding)
function_token	N*	-	Director Proxy authentication token (required with function_app)

* = Conditionally required. Use either direct authentication (tenant_id, client_id, client_secret) OR Director Proxy forwarding (function_app, function_token).

Table Configuration

Field	Required	Default	Description
endpoint	Y		DCE ingestion endpoint or Resource ID
rule_id	N		Default Data Collection Rule (DCR) ID
streams	N	-	Detailed stream (table) configurations
stream	N	-	Legacy string array of stream names
buffer_size	N	1048576	Buffer size in bytes (1MB)
batch_size	N	1000	Maximum number of messages per batch
keep_phantom_fields	N	false	Keep fields not defined in DCR schema
drop_unknown_stream_events	N	true	Silently drop events for undefined tables
cache.timeout	N	300	Table cache timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

Scheduler

Field	Required	Default	Description
interval	N	realtime	Execution frequency. See Interval for details
cron	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Azure Monitor Logs target enables direct ingestion into Log Analytics workspace tables using the modern Data Collection Rules (DCRs) API. This approach provides secure authentication with Azure AD, schema validation, and optional data transformation through DCRs.

Deployment Models

The target supports two deployment models:

Direct Authentication - Director connects directly to Azure using service principal credentials (tenant_id, client_id, client_secret). This model requires Director to have network connectivity to Azure endpoints and credentials for the target subscription.

Director Proxy Forwarding - Director sends processed data to VirtualMetric Director Proxy (Azure Function) deployed in customer environment. Director Proxy uses Azure Managed Identity for credential-free access to Azure Monitor Logs, eliminating the need to share Azure credentials with Director.

The Director Proxy model is particularly valuable for MSSP deployments where customers maintain complete control over Azure credentials while enabling centralized data processing and routing by the MSSP's Director infrastructure.

Dynamic Table Routing

The target supports using the SystemS3 field to route messages to specific tables. When a log message contains a SystemS3 field, it overrides the default table selection, using the format Custom-TableName.

Schema Validation

The target automatically detects table schemas from your Data Collection Rules and validates incoming data. When keep_phantom_fields is set to false, the target removes fields that aren't defined in the DCR schema before sending data.

warning

Disabling keep_phantom_fields may result in data loss for undefined fields.

Batch Processing

Data is buffered in memory until either the batch size limit is reached or an explicit flush is triggered. Each table type has different limits based on the Log Analytics ingestion API.

warning

Large buffer sizes or batch sizes increase memory usage.

Field Normalization

The field_format property allows normalizing log data to standard formats before ingestion. Field normalization is applied before the logs are sent to Azure Monitor, ensuring consistent indexing and search capabilities.

Unknown Events Handling

When drop_unknown_stream_events is enabled, the target silently discards events destined for tables that don't exist in your DCR configuration. This prevents errors but may result in data loss.

warning

Enabling drop_unknown_stream_events silently discards unmatched events.

Autodiscovery

Director provides an autodiscovery feature that automatically configures Data Collection Rules and their associated tables.

The required permissions are:

• For Data Collection Rules

  Role	Scope
  Monitoring Metrics Publisher	Each DCR with name starting with vmetric

• For Resource Groups

  Role	Scope
  Monitoring Reader	Resource Group containing your DCE

warning

Always assign the Monitoring Reader role at the Resource Group level, not at the Subscription level.

The feature uses Resource IDs to discover DCRs and their configurations. It then automatically detects table schemas and validates fields, and prevents phantom fields through schema validation.

Autodiscovery adapts to environment changes automatically.

Examples

Basic Configuration

Configuration using Resource ID-based autodiscovery:

targets:
  - name: basic_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Managed Identity

Using Managed Identity Authentication instead of App Registration:

targets:
  - name: managed_identity_azmonitor
    type: azmonitor
    properties:
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Director Proxy

Configuration using Director Proxy for credential-free forwarding:

targets:
  - name: proxy_azmonitor
    type: azmonitor
    properties:
      function_app: "https://my-director-proxy.azurewebsites.net/api/Sentinel"
      function_token: "your-proxy-authentication-token"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Specific Tables

Using specific table filtering and custom cache timeout:

targets:
  - name: filtered_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      streams:
        - name: "Custom-ApplicationLogs"
        - name: "Custom-PerformanceMetrics"
      cache:
        timeout: 300
      keep_phantom_fields: false
      drop_unknown_stream_events: true

Custom Tables

Configuration for custom log tables:

targets:
  - name: custom_tables_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      streams:
        - name: "Custom-WebServerLogs"
          rule_id: "dcr-00000000000000000000000000000001"
        - name: "Custom-DatabaseLogs"
          rule_id: "dcr-00000000000000000000000000000002"
      keep_phantom_fields: true

High-Volume

Optimization for high-volume ingestion:

targets:
  - name: highvolume_azmonitor
    type: azmonitor
    pipelines:
      - normalization
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      buffer_size: 5242880
      batch_size: 5000
      streams:
        - name: "Custom-HighVolumeLogs"

Multiple Workspaces

Configuration for sending to different workspaces:

targets:
  - name: production_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/production/providers/Microsoft.Insights/dataCollectionEndpoints/prod-dce"
      streams:
        - name: "Custom-ProductionLogs"

  - name: staging_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/staging/providers/Microsoft.Insights/dataCollectionEndpoints/staging-dce"
      streams:
        - name: "Custom-StagingLogs"

Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      field_format: "ecs"
      streams:
        - name: "Custom-NormalizedLogs"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      batch_size: 1000
      interval: "5m"

With Debugging

Configuration with debug options:

targets:
  - name: debug_azmonitor
    type: azmonitor
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      debug:
        status: true
        dont_send_logs: true

Querying Data

After ingestion, query your logs in Log Analytics using Kusto Query Language (KQL):

Custom_ApplicationLogs_CL
| where TimeGenerated > ago(1h)
| where severity_s == "error"
| summarize count() by host_s

note

Custom tables created via DCR have the _CL suffix automatically appended by Azure Monitor.

Troubleshooting

Authentication Errors

If you encounter authentication errors:

• Verify tenant ID, client ID, and client secret are correct
• Check the service principal has appropriate permissions
• Ensure the service principal is not expired
• Verify network connectivity to Azure endpoints

DCE/DCR Not Found

If the Data Collection Endpoint or Rules are not found:

• Verify the endpoint Resource ID is correct
• Check the DCE exists in the specified resource group
• Ensure autodiscovery permissions are correctly assigned
• Verify DCR names start with vmetric for autodiscovery

Data Not Appearing

If logs don't appear in Log Analytics:

• Wait 2-3 minutes for ingestion latency
• Check DCR is correctly configured with the target table
• Verify table schema matches the incoming data
• Review Azure Monitor ingestion logs for errors

Schema Validation Errors

If you see schema validation errors:

• Check keep_phantom_fields setting
• Verify incoming data matches DCR schema
• Review DCR transformations for conflicts
• Ensure required fields are present in data

Buffer/Batch Limits

If you see buffer or batch limit errors:

• Reduce batch_size parameter
• Decrease buffer_size parameter
• Check memory usage on Director
• Monitor ingestion rate in Azure

Permission Errors

If you encounter permission errors:

• Verify Monitoring Metrics Publisher role is assigned to DCRs
• Check Monitoring Reader role is assigned to Resource Group
• Ensure roles are assigned at correct scope
• Verify service principal is not disabled

Proxy Connection Failures

If Director Proxy connections fail:

• Verify function_app URL is correct and accessible
• Check function_token is valid
• Ensure Director Proxy is deployed and running
• Verify Managed Identity is correctly configured on Function