File

Long Term Storage

Synopsis

Creates a file target that writes log messages to files in various formats like JSON, Avro, OCF, Parquet, with support for various compression methods and schemas.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: file
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    location: <string>
    name: <string>
    type: <string>
    compression: <string>
    schema: <string>
    format: <string>
    no_buffer: <boolean>

Configuration

The following are the minimum requirements to define the target.

Field	Required	Default	Description
id	Y		Unique identifier
name	Y		Target name
description	N	-	Optional description
type	Y		Must be file
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Files

Field	Required	Default	Description
location	N	<service-root>	File output directory
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	File name template
type	N	"json"	File format: json, avro, ocf, or parquet
compression	N	"zstd"	Compression algorithm
schema	N	-	Data schema for Avro/OCF/Parquet formats
no_buffer	N	false	Disable write buffering
format	N	-	Field normalization format: ecs, cim, or asim

Details

If no schema is specified the for the Avro, OCF, or Parquet formats, a default schema will be used that captures epoch timestamp and message content.

note

Files with no messages (i.e. with counter=0) are automatically removed when the target is disposed.

Templates

The following template variables can be used in the file name:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Type}}	File type	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	file

Compression

Format	Compression
JSON	zstd
Avro	None
OCF	null, deflate, snappy, zstd
Parquet	uncompressed, gzip, snappy, zstd, brotli, lz4

Examples

JSON

The minimum configuration for JSON file output:

- id: 1
  name: json_logs
  type: file
  properties:
    location: "/var/log/vmetric"

Parquet

Configuration for Parquet output with compression and schema:

- id: 2
  name: parquet_logs
  type: file
  properties:
    location: "/var/log/vmetric"
    type: "parquet"
    compression: "zstd"
    schema: |
      {
        "timestamp": {
          "type": "INT",
          "bitWidth": 64,
          "signed": true
        },
        "message": {
          "type": "STRING",
          "compression": "ZSTD"
        }
      }

OCF

Configuration for OCF output with daily file rotation:

- id: 3
  name: ocf_logs
  type: file
  properties:
    location: "/var/log/vmetric"
    type: "ocf"
    compression: "zstd"
    name: "logs_{{.Year}}_{{.Month}}_{{.Day}}.ocf"

Windows

Configuration for a Windows environment with a proper path structure:

- id: 4
  name: windows_logs
  type: file
  properties:
    location: "C:\\ProgramData\\VMetric\\Logs"
    type: "json"
    compression: "zstd"
    name: "windows_{{.Year}}\\{{.Month}}\\system_logs.json"

warning

When no_buffer is enabled, each write operation will be immediately flushed to disk. This provides durability but may impact performance.