File

Long Term Storage

Synopsis

Creates a file target that writes log messages to files in various formats like JSON, MultiJSON, Avro, Parquet, with support for various compression methods and schemas.

Schema

- name: <string>
  description: <string>
  type: file
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    location: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    field_format: <string>
    no_buffer: <boolean>
    batch_size: <integer>
    max_size: <integer>
    locations: <location[]>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be file
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Files

Files can have the following properties:

Field	Required	Default	Description
location	N	<service-root>	File output directory
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	File name template
format	N	"json"	File format. See Formats below
compression	N	-	Compression algorithm. See Compression below
extension	N	Matches type	Custom file extension
schema	N	-	Data schema for Avro / Parquet formats or schema template name
batch_size	N	100000	Maximum number of messages per file
max_size	N	32MB	Maximum file size before rotating
no_buffer	N	false	Disable write buffering
field_format	N	-	Data normalization format. See applicable Normalization section

Multiple Locations

You can define multiple output locations with different settings:

targets:
  - name: multi_location_logs
    type: file
    properties:
      locations:
        - id: "security_logs"
          path: "/var/log/security"
          schema: "{{CommonSecurityLog}}"
          format: "parquet"
        - id: "system_logs"
          path: "/var/log/system"
          schema: "{{CommonSystemLog}}"
          format: "json"

Details

The file target supports writing to multiple file locations with different formats and schemas. When using SystemS3 field in your logs, the value will be used to route the message to the location with a matching ID.

If no schema is specified for Avro or Parquet formats, a default schema will be used that captures epoch timestamp and message content.

The target supports the following built-in schema templates:

• {{Syslog}} - Standard schema for Syslog messages
• {{CommonSecurityLog}} - Schema compatible with Common Event Format (CEF)

note

Files with no messages (i.e. with counter=0) are automatically removed when the target is disposed.

caution

When no_buffer is enabled, each write operation will be immediately flushed to disk. This provides durability but may impact performance.

Templates

The following template variables can be used in the file name:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	file
{{.Table}}	Location ID	security_logs

Formats

Format	Description
json	Each log entry is written as a separate JSON line (JSONL format)
multijson	All log entries are written as a single JSON array
ocf	OCF format with schema
avro	Apache Avro format with schema
parquet	Apache Parquet columnar format with schema

Compression

Files can use the following compression algorithms:

Format	Default	Compression Codecs
Avro	zstd	See Appendix
Parquet	zstd	See Appendix

Examples

JSON

Configuration for a JSON output— as "json" is the default format, no need to specify it:

targets:
  - name: json_logs
    type: file
    properties:
      location: "/var/log/vmetric"
      compression: "zstd"

Multiple Locations

Configuration for multiple output locations; a different format is used for each location:

targets:
  - name: multi_location_logs
    type: file
    properties:
      locations:
        - id: "security"
          path: "/var/log/vmetric/security"
          format: "parquet"
          schema: "{{CommonSecurityLog}}"
        - id: "system"
          path: "/var/log/vmetric/system"
          format: "json"
        - id: "application"
          path: "/var/log/vmetric/app"
          format: "multijson"
          name: "app_{{.Year}}_{{.Month}}_{{.Day}}.json"

Parquet

Configuration for a Parquet output with compression and schema:

targets:
  - name: parquet_logs
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "parquet"
      compression: "zstd"
      schema: |
        {
          "timestamp": {
            "type": "INT",
            "bitWidth": 64,
            "signed": true
          },
          "message": {
            "type": "STRING",
            "compression": "ZSTD"
          }
        }

Using built-in schema templates:

name: cef_logs
type: file
properties:
  location: "/var/log/vmetric"
  format: "parquet"
  schema: "{{CommonSecurityLog}}"

OCF

Configuration for an OCF output with daily file rotation:

targets:
  - name: ocf_logs
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "ocf"
      compression: "zstd"
      name: "logs_{{.Year}}_{{.Month}}_{{.Day}}.ocf"

Windows

Configuration for a Windows environment with a proper path structure:

targets:
  - name: windows_logs
    type: file
    properties:
      location: "C:\\ProgramData\\VMetric\\Logs"
      compression: "zstd"
      name: "windows_{{.Year}}\\{{.Month}}\\system_logs.json"