VRL example reference

Here you’ll find a comprehensive list of all VRL program examples. These examples demonstrate the breadth of the language and its observability-focused facilities.

Try Using the VRL subcommand

You can run these examples using the vector vrl subcommand with --input (input is newline delimited JSON file representing a list of events). and --program (VRL program) to pass in the example input and program as well as --print-object to show the modified object. The below examples show pretty-printed JSON for the input events, so collapse these to single lines when passing in via --input.

For example: vector vrl --input input.json --program program.vrl --print-object. This closely matches how VRL will receive the input in a running Vector instance.

VRL REPL

Additionally, if vector vrl is run without any arguments, it will spawn a REPL (Read–eval–print loop).

Assuming you have Vector installed, you can run vector vrl to start the REPL. From there, you can type help and press return to get further help.

The REPL behaves nearly identical to the programs you write for your Vector configuration, and can be used to test individual snippets of complex programs before you commit them to your production configuration.

The vector vrl command has many other capabilities, run the command using vector vrl --help to see more information.

Real world examples

Parse Syslog logs

Given this Vector log event...
{
  "message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}
...and this VRL program...
. |= parse_syslog!(.message)
...the following log event is output:
{
  "log": {
    "appname": "su",
    "facility": "ntp",
    "hostname": "vector-user.biz",
    "message": "Something went wrong",
    "msgid": "ID389",
    "procid": 2666,
    "severity": "info",
    "timestamp": "2020-12-22T15:22:31.111Z",
    "version": 1
  }
}

Parse key/value (logfmt) logs

Given this Vector log event...
{
  "message": "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
}
...and this VRL program...
. = parse_key_value!(.message)
...the following log event is output:
{
  "log": {
    "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
    "id": "ConsumerFetcherManager-1382721708341",
    "level": "info",
    "module": "kafka.consumer.ConsumerFetcherManager",
    "msg": "Stopping all fetchers",
    "tag#production": "stopping_fetchers"
  }
}

Parse custom logs

Given this Vector log event...
{
  "message": "2021/01/20 06:39:15 +0000 [error] 17755#17755: *3569904 open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: \"GET /test.php HTTP/1.1\", host: \"yyy.yyy.yyy.yyy\""
}
...and this VRL program...
. |= parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+ \+\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')

# Coerce parsed fields
.timestamp = parse_timestamp(.timestamp, "%Y/%m/%d %H:%M:%S %z") ?? now()
.pid = to_int!(.pid)
.tid = to_int!(.tid)

# Extract structured data
message_parts = split(.message, ", ", limit: 2)
structured = parse_key_value(message_parts[1], key_value_delimiter: ":", field_delimiter: ",") ?? {}
.message = message_parts[0]
. = merge(., structured)
...the following log event is output:
{
  "log": {
    "client": "xxx.xxx.xxx.xxx",
    "connid": "3569904",
    "host": "yyy.yyy.yyy.yyy",
    "message": "open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory)",
    "pid": 17755,
    "request": "GET /test.php HTTP/1.1",
    "server": "localhost",
    "severity": "error",
    "tid": 17755,
    "timestamp": "2021-01-20T06:39:15Z"
  }
}

Multiple parsing strategies

Given this Vector log event...
{
  "message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}
...and this VRL program...
structured =
  parse_syslog(.message) ??
  parse_common_log(.message) ??
  parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')
. = merge(., structured)
...the following log event is output:
{
  "log": {
    "appname": "su",
    "facility": "ntp",
    "hostname": "vector-user.biz",
    "message": "Something went wrong",
    "msgid": "ID389",
    "procid": 2666,
    "severity": "info",
    "timestamp": "2020-12-22T15:22:31.111Z",
    "version": 1
  }
}

Modify metric tags

Given this Vector metric event...
{
  "counter": {
    "value": 102
  },
  "kind": "incremental",
  "name": "user_login_total",
  "tags": {
    "email": "vic@vector.dev",
    "host": "my.host.com",
    "instance_id": "abcd1234"
  }
}
...and this VRL program...
.tags.environment = get_env_var!("ENV") # add
.tags.hostname = del(.tags.host) # rename
del(.tags.email)
...the following metric event is output:
{
  "metric": {
    "counter": {
      "value": 102
    },
    "kind": "incremental",
    "name": "user_login_total",
    "tags": {
      "environment": "production",
      "hostname": "my.host.com",
      "instance_id": "abcd1234"
    }
  }
}

Emitting multiple logs from JSON

Given this Vector log event...
{
  "message": "[{\"message\": \"first_log\"}, {\"message\": \"second_log\"}]"
}
...and this VRL program...
. = parse_json!(.message) # sets `.` to an array of objects
...the following log event is output:
[
  {
    "log": {
      "message": "first_log"
    }
  },
  {
    "log": {
      "message": "second_log"
    }
  }
]

Emitting multiple non-object logs from JSON

Given this Vector log event...
{
  "message": "[5, true, \"hello\"]"
}
...and this VRL program...
. = parse_json!(.message) # sets `.` to an array
...the following log event is output:
[
  {
    "log": {
      "message": 5
    }
  },
  {
    "log": {
      "message": true
    }
  },
  {
    "log": {
      "message": "hello"
    }
  }
]

Invalid argument type

Given this Vector log event...
{
  "not_a_string": 1
}
...and this VRL program...
upcase(42)
...you should see this error:
error[E110]: invalid argument type
  ┌─ :1:8
  │
1 │ upcase(42)
  │        ^^
  │        │
  │        this expression resolves to the exact type integer
  │        but the parameter "value" expects the exact type string
  │
  = try: ensuring an appropriate type at runtime
  =
  =     42 = string!(42)
  =     upcase(42)
  =
  = try: coercing to an appropriate type and specifying a default value as a fallback in case coercion fails
  =
  =     42 = to_string(42) ?? "default"
  =     upcase(42)
  =
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = learn more about error code 110 at https://errors.vrl.dev/110
  = see language documentation at https://vrl.dev
  = try your code in the VRL REPL, learn more at https://vrl.dev/examples

Unhandled fallible assignment

Given this Vector log event...
{
  "message": "key1=value1 key2=value2"
}
...and this VRL program...
structured = parse_key_value(.message)
...you should see this error:
error[E103]: unhandled fallible assignment
  ┌─ :1:14
  │
1 │ structured = parse_key_value(.message)
  │ ------------ ^^^^^^^^^^^^^^^^^^^^^^^^^
  │ │            │
  │ │            this expression is fallible because at least one argument's type cannot be verified to be valid
  │ │            update the expression to be infallible by adding a `!`: `parse_key_value!(.message)`
  │ │            `.message` argument type is `any` and this function expected a parameter `value` of type `string`
  │ or change this to an infallible assignment:
  │ structured, err = parse_key_value(.message)
  │
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = see functions characteristics documentation at https://vrl.dev/expressions/#function-call-characteristics
  = learn more about error code 103 at https://errors.vrl.dev/103
  = see language documentation at https://vrl.dev
  = try your code in the VRL REPL, learn more at https://vrl.dev/examples

Array examples

append

Appends each item in the items array to the end of the value array.

Append to an array

Source
append([1, 2], [3, 4])
Return
[1,2,3,4]
Learn more about the append function

chunks

Chunks value into slices of length chunk_size bytes.

Split a string into chunks

Source
chunks("abcdefgh", 4)
Return
["abcd","efgh"]
Learn more about the chunks function

Chunks do not respect unicode code point boundaries

Source
chunks("ab你好", 4)
Return
["ab�","�好"]
Learn more about the chunks function

pop

Removes the last item from the value array.

Pop an item from an array

Source
pop([1, 2, 3])
Return
[1,2]
Learn more about the pop function

push

Adds the item to the end of the value array.

Push an item onto an array

Source
push([1, 2], 3)
Return
[1,2,3]
Learn more about the push function

Empty array

Source
push([], "bar")
Return
["bar"]
Learn more about the push function

zip

Iterate over several arrays in parallel, producing a new array containing arrays of items from each source. The resulting array will be as long as the shortest input array, with all the remaining elements dropped. This function is modeled from the zip function in Python, but similar methods can be found in Ruby and Rust.

If a single parameter is given, it must contain an array of all the input arrays.

Merge two arrays

Source
zip([1, 2, 3], [4, 5, 6, 7])
Return
[[1,4],[2,5],[3,6]]
Learn more about the zip function

Merge three arrays

Source
zip([[1, 2], [3, 4], [5, 6]])
Return
[[1,3,5],[2,4,6]]
Learn more about the zip function

Merge an array of three arrays into an array of 3-tuples

Source
zip([["a", "b", "c"], [1, null, true], [4, 5, 6]])
Return
[["a",1,4],["b",null,5],["c",true,6]]
Learn more about the zip function

Merge two array parameters

Source
zip([1, 2, 3, 4], [5, 6, 7])
Return
[[1,5],[2,6],[3,7]]
Learn more about the zip function

Codec examples

decode_base16

Decodes the value (a Base16 string) into its original string.

Decode Base16 data

Source
decode_base16!("736F6D6520737472696E672076616C7565")
Return
some string value
Learn more about the decode_base16 function

Decode longer Base16 data

Source
decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")
Return
you have successfully decoded me
Learn more about the decode_base16 function

decode_base64

Decodes the value (a Base64 string) into its original string.

Decode Base64 data (default)

Source
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return
you have successfully decoded me
Learn more about the decode_base64 function

Decode Base64 data (URL safe)

Source
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return
you can't make your heart feel something it won't
Learn more about the decode_base64 function

decode_charset

Decodes the value (a non-UTF8 string) to a UTF8 string using the specified character set.

Decode EUC-KR string

Source
decode_charset!(decode_base64!("vsiz58fPvLy/5A=="), "euc-kr")
Return
안녕하세요
Learn more about the decode_charset function

Decode EUC-JP string

Source
decode_charset!(decode_base64!("pLOk86TLpMGkzw=="), "euc-jp")
Return
こんにちは
Learn more about the decode_charset function

Decode GB2312 string

Source
decode_charset!(decode_base64!("xOO6ww=="), "gb2312")
Return
你好
Learn more about the decode_charset function

decode_gzip

Decodes the value (a Gzip string) into its original string.

Decode Gzip data

Source
decode_gzip!(decode_base64!("H4sIAB8BymMAAyvISU0sTlVISU3OT0lVyE0FAJsZ870QAAAA"))
Return
please decode me
Learn more about the decode_gzip function

decode_lz4

Decodes the value (an lz4 string) into its original string. buf_size is the size of the buffer to decode into, this must be equal to or larger than the uncompressed size. If prepended_size is set to true, it expects the original uncompressed size to be prepended to the compressed data. prepended_size is useful for some implementations of lz4 that require the original size to be known before decoding.

LZ4 block with prepended size

Source
decode_lz4!(decode_base64!("LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4="), prepended_size: true)
Return
The quick brown fox jumps over 13 lazy dogs.
Learn more about the decode_lz4 function

Decode Lz4 data without prepended size.

Source
decode_lz4!(decode_base64!("BCJNGGBAgiwAAIBUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLgAAAAA="))
Return
The quick brown fox jumps over 13 lazy dogs.
Learn more about the decode_lz4 function

decode_mime_q

Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.

Decode single encoded-word

Source
decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")
Return
Hello, World!
Learn more about the decode_mime_q function

Embedded

Source
decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")
Return
From: Hello, World! <hello_world@example.com>
Learn more about the decode_mime_q function

Without charset

Source
decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")
Return
Hello, World!
Learn more about the decode_mime_q function

decode_percent

Decodes a percent-encoded value like a URL.

Percent decode a value

Source
decode_percent("foo%20bar%3F")
Return
foo bar?
Learn more about the decode_percent function

decode_punycode

Decodes a punycode encoded value, such as an internationalized domain name (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.

Decode a punycode encoded internationalized domain name

Source
decode_punycode!("www.xn--caf-dma.com")
Return
www.café.com
Learn more about the decode_punycode function

Decode an ASCII only string

Source
decode_punycode!("www.cafe.com")
Return
www.cafe.com
Learn more about the decode_punycode function

Ignore validation

Source
decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return
١٠.٦٦.٣٠.٥.
Learn more about the decode_punycode function

decode_snappy

Decodes the value (a Snappy string) into its original string.

Decode Snappy data

Source
decode_snappy!(decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg=="))
Return
The quick brown fox jumps over 13 lazy dogs.
Learn more about the decode_snappy function

decode_zlib

Decodes the value (a Zlib string) into its original string.

Decode Zlib data

Source
decode_zlib!(decode_base64!("eJxLzUvOT0mNz00FABI5A6A="))
Return
encode_me
Learn more about the decode_zlib function

decode_zstd

Decodes the value (a Zstandard string) into its original string.

Decode Zstd data

Source
decode_zstd!(decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA"))
Return
you_have_successfully_decoded_me.congratulations.you_are_breathtaking.
Learn more about the decode_zstd function

encode_base16

Encodes the value to Base16.

Encode to Base16

Source
encode_base16("some string value")
Return
736f6d6520737472696e672076616c7565
Learn more about the encode_base16 function

encode_base64

Encodes the value to Base64.

Encode to Base64 (default)

Source
encode_base64("please encode me")
Return
cGxlYXNlIGVuY29kZSBtZQ==
Learn more about the encode_base64 function

Encode to Base64 (without padding)

Source
encode_base64("please encode me, no padding though", padding: false)
Return
cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g
Learn more about the encode_base64 function

Encode to Base64 (URL safe)

Source
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return
cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=
Learn more about the encode_base64 function

Encode to Base64 (without padding and URL safe)

Source
encode_base64("some string value", padding: false, charset: "url_safe")
Return
c29tZSBzdHJpbmcgdmFsdWU
Learn more about the encode_base64 function

encode_charset

Encodes the value (a UTF8 string) to a non-UTF8 string using the specified character set.

Encode UTF8 string to EUC-KR

Source
encode_base64(encode_charset!("안녕하세요", "euc-kr"))
Return
vsiz58fPvLy/5A==
Learn more about the encode_charset function

Encode UTF8 string to EUC-JP

Source
encode_base64(encode_charset!("こんにちは", "euc-jp"))
Return
pLOk86TLpMGkzw==
Learn more about the encode_charset function

Encode UTF8 string to GB2312

Source
encode_base64(encode_charset!("你好", "gb2312"))
Return
xOO6ww==
Learn more about the encode_charset function

encode_gzip

Encodes the value to Gzip.

Encode to Gzip

Source
encode_base64(encode_gzip("please encode me"))
Return
H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA
Learn more about the encode_gzip function

encode_json

Encodes the value to JSON.

Encode object to JSON

Source
encode_json({"field": "value", "another": [1,2,3]})
Return
s'{"another":[1,2,3],"field":"value"}'
Learn more about the encode_json function

Encode object to as pretty-printed JSON

Source
encode_json({"field": "value", "another": [1,2,3]}, true)
Return
{
  "another": [
    1,
    2,
    3
  ],
  "field": "value"
}
Learn more about the encode_json function

encode_key_value

Encodes the value into key-value format with customizable delimiters. Default delimiters match the logfmt format.

Encode with default delimiters (no ordering)

Source
encode_key_value(
    {
        "ts": "2021-06-05T17:20:00Z",
        "msg": "This is a message",
        "lvl": "info"
    }
)
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Learn more about the encode_key_value function

Encode with default delimiters (fields ordering)

Source
encode_key_value!(
    {
        "ts": "2021-06-05T17:20:00Z",
        "msg": "This is a message",
        "lvl": "info",
        "log_id": 12345
    },
    ["ts", "lvl", "msg"]
)
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Learn more about the encode_key_value function

Encode with default delimiters (nested fields)

Source
encode_key_value(
    {
        "agent": {"name": "foo"},
        "log": {"file": {"path": "my.log"}},
        "event": "log"
    }
)
Return
agent.name=foo event=log log.file.path=my.log
Learn more about the encode_key_value function

Encode with default delimiters (nested fields ordering)

Source
encode_key_value!(
    {
        "agent": {"name": "foo"},
        "log": {"file": {"path": "my.log"}},
        "event": "log"
    },
    ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=foo
Learn more about the encode_key_value function

Encode with custom delimiters (no ordering)

Source
encode_key_value(
    {"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
    field_delimiter: ",",
    key_value_delimiter: ":"
)
Return
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Learn more about the encode_key_value function

Encode with custom delimiters and flatten boolean

Source
encode_key_value(
    {"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
    field_delimiter: ",",
    key_value_delimiter: ":",
    flatten_boolean: true
)
Return
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Learn more about the encode_key_value function

encode_logfmt

Encodes the value to logfmt.

Encode to logfmt (no ordering)

Source
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Learn more about the encode_logfmt function

Encode to logfmt (fields ordering)

Source
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Learn more about the encode_logfmt function

Encode to logfmt (nested fields)

Source
encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=foo event=log log.file.path=my.log
Learn more about the encode_logfmt function

Encode to logfmt (nested fields ordering)

Source
encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=foo
Learn more about the encode_logfmt function

encode_lz4

Encodes the value to Lz4. This function compresses the input string into an lz4 block. If prepend_size is set to true, it prepends the original uncompressed size to the compressed data. This is useful for some implementations of lz4 that require the original size to be known before decoding.

Encode to Lz4

Source
encode_base64(encode_lz4!("The quick brown fox jumps over 13 lazy dogs.", true))
Return
LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4=
Learn more about the encode_lz4 function

encode_percent

Encodes a value with percent encoding to safely be used in URLs.

Percent encode all non-alphanumeric characters (default)

Source
encode_percent("foo bar?")
Return
foo%20bar%3F
Learn more about the encode_percent function

Percent encode only control characters

Source
encode_percent("foo \tbar", ascii_set: "CONTROLS")
Return
foo %09bar
Learn more about the encode_percent function

Percent encode special characters

Source
encode_percent("foo@bar?")
Return
foo%40bar%3F
Learn more about the encode_percent function

encode_proto

Encodes the value into a protocol buffer payload.

Encode to proto

Source
encode_base64(encode_proto!({ "name": "someone", "phones": [{"number": "123456"}]}, "test_protobuf.desc", "test_protobuf.v1.Person"))
Return
Cgdzb21lb25lIggKBjEyMzQ1Ng==
Learn more about the encode_proto function

encode_punycode

Encodes a value to punycode. Useful for internationalized domain names (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.

Encode an internationalized domain name

Source
encode_punycode!("www.café.com")
Return
www.xn--caf-dma.com
Learn more about the encode_punycode function

Encode an internationalized domain name with mixed case

Source
encode_punycode!("www.CAFé.com")
Return
www.xn--caf-dma.com
Learn more about the encode_punycode function

Encode an ASCII only string

Source
encode_punycode!("www.cafe.com")
Return
www.cafe.com
Learn more about the encode_punycode function

Ignore validation

Source
encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return
xn--8hbb.xn--fiba.xn--8hbf.xn--eib.
Learn more about the encode_punycode function

encode_snappy

Encodes the value to Snappy.

Encode to Snappy

Source
encode_base64(encode_snappy!("The quick brown fox jumps over 13 lazy dogs."))
Return
LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==
Learn more about the encode_snappy function

encode_zlib

Encodes the value to Zlib.

Encode to Zlib

Source
encode_base64(encode_zlib("please encode me"))
Return
eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7
Learn more about the encode_zlib function

encode_zstd

Encodes the value to Zstandard.

Encode to Zstd

Source
encode_base64(encode_zstd("please encode me"))
Return
KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ==
Learn more about the encode_zstd function

Coerce examples

to_bool

Coerces the value into a boolean.

Coerce to a Boolean (string)

Source
to_bool!("yes")
Return
true
Learn more about the to_bool function

Coerce to a Boolean (float)

Source
to_bool(0.0)
Learn more about the to_bool function

Coerce to a Boolean (int)

Source
to_bool(0)
Learn more about the to_bool function

Coerce to a Boolean (null)

Source
to_bool(null)
Learn more about the to_bool function

Coerce to a Boolean (Boolean)

Source
to_bool(true)
Return
true
Learn more about the to_bool function

Integer (other)

Source
to_bool(2)
Return
true
Learn more about the to_bool function

Float (other)

Source
to_bool(5.6)
Return
true
Learn more about the to_bool function

False

Source
to_bool(false)
Learn more about the to_bool function

True string

Source
to_bool!(s'true')
Return
true
Learn more about the to_bool function

Y string

Source
to_bool!(s'y')
Return
true
Learn more about the to_bool function

Non-zero integer string

Source
to_bool!(s'1')
Return
true
Learn more about the to_bool function

False string

Source
to_bool!(s'false')
Learn more about the to_bool function

No string

Source
to_bool!(s'no')
Learn more about the to_bool function

N string

Source
to_bool!(s'n')
Learn more about the to_bool function

Invalid string

Source
to_bool!(s'foobar')
Learn more about the to_bool function

Timestamp

Source
to_bool!(t'2020-01-01T00:00:00Z')
Learn more about the to_bool function

Array

Source
to_bool!([])
Learn more about the to_bool function

Object

Source
to_bool!({})
Learn more about the to_bool function

Regex

Source
to_bool!(r'foo')
Learn more about the to_bool function

to_float

Coerces the value into a float.

Coerce to a float

Source
to_float!("3.145")
Return
3.145
Learn more about the to_float function

Coerce to a float (timestamp)

Source
to_float(t'2020-12-30T22:20:53.824727Z')
Return
1609366853.824727
Learn more about the to_float function

Integer

Source
to_float(5)
Return
5
Learn more about the to_float function

Float

Source
to_float(5.6)
Return
5.6
Learn more about the to_float function

True

Source
to_float(true)
Return
1
Learn more about the to_float function

False

Source
to_float(false)
Learn more about the to_float function

Null

Source
to_float(null)
Learn more about the to_float function

Invalid string

Source
to_float!(s'foobar')
Learn more about the to_float function

Array

Source
to_float!([])
Learn more about the to_float function

Object

Source
to_float!({})
Learn more about the to_float function

Regex

Source
to_float!(r'foo')
Learn more about the to_float function

to_int

Coerces the value into an integer.

Coerce to an int (string)

Source
to_int!("2")
Return
2
Learn more about the to_int function

Coerce to an int (timestamp)

Source
to_int(t'2020-12-30T22:20:53.824727Z')
Return
1609366853
Learn more about the to_int function

Integer

Source
to_int(5)
Return
5
Learn more about the to_int function

Float

Source
to_int(5.6)
Return
5
Learn more about the to_int function

True

Source
to_int(true)
Return
1
Learn more about the to_int function

False

Source
to_int(false)
Learn more about the to_int function

Null

Source
to_int(null)
Learn more about the to_int function

Invalid string

Source
to_int!(s'foobar')
Learn more about the to_int function

Array

Source
to_int!([])
Learn more about the to_int function

Object

Source
to_int!({})
Learn more about the to_int function

Regex

Source
to_int!(r'foo')
Learn more about the to_int function

to_regex

Coerces the value into a regex.

Coerce to a regex

Source
to_regex!("^foo$")
Return
r'^foo$'
Learn more about the to_regex function

to_string

Coerces the value into a string.

Coerce to a string (Boolean)

Source
to_string(true)
Return
s'true'
Learn more about the to_string function

Coerce to a string (int)

Source
to_string(52)
Return
s'52'
Learn more about the to_string function

Coerce to a string (float)

Source
to_string(52.2)
Return
s'52.2'
Learn more about the to_string function

String

Source
to_string(s'foo')
Return
foo
Learn more about the to_string function

False

Source
to_string(false)
Return
s'false'
Learn more about the to_string function

Null

Source
to_string(null)
Learn more about the to_string function

Timestamp

Source
to_string(t'2020-01-01T00:00:00Z')
Return
2020-01-01T00:00:00Z
Learn more about the to_string function

Array

Source
to_string!([])
Learn more about the to_string function

Object

Source
to_string!({})
Learn more about the to_string function

Regex

Source
to_string!(r'foo')
Learn more about the to_string function

Convert examples

from_unix_timestamp

Converts the value integer from a Unix timestamp to a VRL timestamp.

Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Convert from a Unix timestamp (seconds)

Source
from_unix_timestamp!(5)
Return
t'1970-01-01T00:00:05Z'
Learn more about the from_unix_timestamp function

Convert from a Unix timestamp (milliseconds)

Source
from_unix_timestamp!(5000, unit: "milliseconds")
Return
t'1970-01-01T00:00:05Z'
Learn more about the from_unix_timestamp function

Convert from a Unix timestamp (microseconds)

Source
from_unix_timestamp!(5000, unit: "microseconds")
Return
t'1970-01-01T00:00:00.005Z'
Learn more about the from_unix_timestamp function

Convert from a Unix timestamp (nanoseconds)

Source
from_unix_timestamp!(5000, unit: "nanoseconds")
Return
t'1970-01-01T00:00:00.000005Z'
Learn more about the from_unix_timestamp function

to_syslog_facility

Converts the value, a Syslog facility code, into its corresponding Syslog keyword. For example, 0 into "kern", 1 into "user", etc.

Coerce to a Syslog facility

Source
to_syslog_facility!(4)
Return
auth
Learn more about the to_syslog_facility function

invalid

Source
to_syslog_facility!(500)
Learn more about the to_syslog_facility function

to_syslog_facility_code

Converts the value, a Syslog facility keyword, into a Syslog integer facility code (0 to 23).

Coerce to Syslog facility code

Source
to_syslog_facility_code!("authpriv")
Return
10
Learn more about the to_syslog_facility_code function

invalid

Source
to_syslog_facility_code!(s'foobar')
Learn more about the to_syslog_facility_code function

to_syslog_level

Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

Coerce to a Syslog level

Source
to_syslog_level!(5)
Return
notice
Learn more about the to_syslog_level function

invalid

Source
to_syslog_level!(500)
Learn more about the to_syslog_level function

to_syslog_severity

Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

Coerce to Syslog severity

Source
to_syslog_severity!("alert")
Return
1
Learn more about the to_syslog_severity function

invalid

Source
to_syslog_severity!(s'foobar')
Learn more about the to_syslog_severity function

to_unix_timestamp

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Convert to a Unix timestamp (seconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return
1609459200
Learn more about the to_unix_timestamp function

Convert to a Unix timestamp (milliseconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return
1609459200000
Learn more about the to_unix_timestamp function

Convert to a Unix timestamp (microseconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "microseconds")
Return
1609459200000000
Learn more about the to_unix_timestamp function

Convert to a Unix timestamp (nanoseconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return
1609459200000000000
Learn more about the to_unix_timestamp function

Debug examples

assert

Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

Assertion (true) - with message

Source
assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return
true
Learn more about the assert function

Assertion (false) - with message

Source
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")
Learn more about the assert function

Assertion (false) - simple

Source
assert!(false)
Learn more about the assert function

assert_eq

Asserts that two expressions, left and right, have the same value. The program is aborted with message if they do not have the same value.

Successful assertion

Source
assert_eq!(1, 1)
Return
true
Learn more about the assert_eq function

Unsuccessful assertion

Source
assert_eq!(127, [1, 2, 3])
Learn more about the assert_eq function

Unsuccessful assertion with custom log message

Source
assert_eq!(1, 0, message: "Unequal integers")
Learn more about the assert_eq function

log

Logs the value to stdout at the specified level.

Log a message

Source
log("Hello, World!", level: "info", rate_limit_secs: 60)
Learn more about the log function

Log an error

Source
. = { "field": "not an integer" }
_, err = to_int(.field)
if err != null {
    log(err, level: "error")
}
Learn more about the log function

Enrichment examples

find_enrichment_table_records

Searches an enrichment table for rows that match the provided condition.

For file enrichment tables, this condition needs to be a VRL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently three forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Wildcard match search. The given fields specified by the exact match search may also be matched exactly to the value provided to the wildcard parameter. A wildcard match search can also use an index directly into the dataset.

  3. Date range search. The given field must be greater than or equal to the from date and/or less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip and mmdb enrichment tables, this condition needs to be a VRL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization

  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone

  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Exact match

Source
find_enrichment_table_records!(
    "test",
    {"surname": "Smith"}
)
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]
Learn more about the find_enrichment_table_records function

Case insensitive match

Source
find_enrichment_table_records!(
    "test",
    {"surname": "smith"},
    case_sensitive: false
)
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]
Learn more about the find_enrichment_table_records function

Wildcard match

Source
find_enrichment_table_records!(
    "test",
    {"firstname": "Bob"},
    wildcard: "fred",
    case_sensitive: false
)
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]
Learn more about the find_enrichment_table_records function

Date range search

Source
find_enrichment_table_records!(
    "test",
    {
        "surname": "Smith",
        "date_of_birth": {
            "from": t'1985-01-01T00:00:00Z',
            "to": t'1985-12-31T00:00:00Z'
        }
    }
)
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]
Learn more about the find_enrichment_table_records function

get_enrichment_table_record

Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.

For file enrichment tables, this condition needs to be a VRL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently three forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Wildcard match search. The given fields specified by the exact match search may also be matched exactly to the value provided to the wildcard parameter. A wildcard match search can also use an index directly into the dataset.

  3. Date range search. The given field must be greater than or equal to the from date and/or less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip and mmdb enrichment tables, this condition needs to be a VRL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization

  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone

  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Exact match

Source
get_enrichment_table_record!("test", {"id": 1})
Return
{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}
Learn more about the get_enrichment_table_record function

Case insensitive match

Source
get_enrichment_table_record!(
    "test",
    {"surname": "bob", "firstname": "John"},
    case_sensitive: false
)
Return
{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}
Learn more about the get_enrichment_table_record function

Date range search

Source
get_enrichment_table_record!(
    "test",
    {
        "surname": "Smith",
        "date_of_birth": {
            "from": t'1985-01-01T00:00:00Z',
            "to": t'1985-12-31T00:00:00Z'
        }
    }
)
Return
{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}
Learn more about the get_enrichment_table_record function

Enumerate examples

compact

Compacts the value by removing empty values, where empty values are defined using the available parameters.

Compact an object with default parameters

Source
compact({"field1": 1, "field2": "", "field3": [], "field4": null})
Return
{
  "field1": 1
}
Learn more about the compact function

Compact an array with default parameters

Source
compact(["foo", "bar", "", null, [], "buzz"])
Return
["foo","bar","buzz"]
Learn more about the compact function

Compact an array using nullish

Source
compact(["-", "   ", "\n", null, true], nullish: true)
Return
[true]
Learn more about the compact function

Compact a complex object with default parameters

Source
compact({ "a": {}, "b": null, "c": [null], "d": "", "e": "-", "f": true })
Return
{
  "e": "-",
  "f": true
}
Learn more about the compact function

Compact a complex object using null: false

Source
compact({ "a": {}, "b": null, "c": [null], "d": "", "e": "-", "f": true }, null: false)
Return
{
  "b": null,
  "c": [
    null
  ],
  "e": "-",
  "f": true
}
Learn more about the compact function

filter

Filter elements from a collection.

This function currently does not support recursive iteration.

The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Filter elements

Source
. = { "tags": ["foo", "bar", "foo", "baz"] }
filter(array(.tags)) -> |_index, value| {
    value != "foo"
}
Return
["bar","baz"]
Learn more about the filter function

Filter object

Source
filter({ "a": 1, "b": 2 }) -> |key, _value| { key == "a" }
Return
{
  "a": 1
}
Learn more about the filter function

Filter array

Source
filter([1, 2]) -> |_index, value| { value < 2 }
Return
[1]
Learn more about the filter function

flatten

Flattens the value into a single-level representation.

Flatten array

Source
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return
[1,2,3,4,5,6,7,8,9]
Learn more about the flatten function

Flatten object

Source
flatten({
    "parent1": {
        "child1": 1,
        "child2": 2
    },
    "parent2": {
        "child3": 3
    }
})
Return
{
  "parent1.child1": 1,
  "parent1.child2": 2,
  "parent2.child3": 3
}
Learn more about the flatten function

Flatten object with custom separator

Source
flatten({ "foo": { "bar": true }}, "_")
Return
{
  "foo_bar": true
}
Learn more about the flatten function

for_each

Iterate over a collection.

This function currently does not support recursive iteration.

The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Tally elements

Source
.tags = ["foo", "bar", "foo", "baz"]
tally = {}
for_each(array(.tags)) -> |_index, value| {
    count = int(get!(tally, [value])) ?? 0
    tally = set!(tally, [value], count + 1)
}
tally
Return
{
  "bar": 1,
  "baz": 1,
  "foo": 2
}
Learn more about the for_each function

Iterate over an object

Source
count = 0
for_each({ "a": 1, "b": 2 }) -> |_key, value| {
    count = count + value
}
count
Return
3
Learn more about the for_each function

Iterate over an array

Source
count = 0
for_each([1, 2, 3]) -> |index, value| {
    count = count + index + value
}
count
Return
9
Learn more about the for_each function

includes

Determines whether the value array includes the specified item.

Array includes

Source
includes(["apple", "orange", "banana"], "banana")
Return
true
Learn more about the includes function

Includes boolean

Source
includes([1, true], true)
Return
true
Learn more about the includes function

Doesn’t include

Source
includes(["foo", "bar"], "baz")
Learn more about the includes function

keys

Returns the keys from the object passed into the function.

Get keys from the object

Source
keys({
    "key1": "val1",
    "key2": "val2"
})
Return
["key1","key2"]
Learn more about the keys function

length

Returns the length of the value.

  • If value is an array, returns the number of elements.
  • If value is an object, returns the number of top-level keys.
  • If value is a string, returns the number of bytes in the string. If you want the number of characters, see strlen.

Length (object)

Source
length({
    "portland": "Trail Blazers",
    "seattle": "Supersonics"
})
Return
2
Learn more about the length function

Length (nested object)

Source
length({
    "home": {
        "city":  "Portland",
        "state": "Oregon"
    },
    "name": "Trail Blazers",
    "mascot": {
        "name": "Blaze the Trail Cat"
    }
})
Return
3
Learn more about the length function

Length (array)

Source
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return
3
Learn more about the length function

Length (string)

Source
length("The Planet of the Apes Musical")
Return
30
Learn more about the length function

map_keys

Map the keys within an object.

If recursive is enabled, the function iterates into nested objects, using the following rules:

  1. Iteration starts at the root.
  2. For every nested object type:
    • First return the key of the object type itself.
    • Then recurse into the object, and loop back to item (1) in this list.
    • Any mutation done on a nested object before recursing into it, are preserved.
  3. For every nested array type:
    • First return the key of the array type itself.
    • Then find all objects within the array, and apply item (2) to each individual object.

The above rules mean that map_keys with recursive enabled finds all keys in the target, regardless of whether nested objects are nested inside arrays.

The function uses the function closure syntax to allow reading the key for each item in the object.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Upcase keys

Source
. = {
    "foo": "foo",
    "bar": "bar",
    "baz": {"nested key": "val"}
}
map_keys(.) -> |key| { upcase(key) }
Return
{
  "BAR": "bar",
  "BAZ": {
    "nested key": "val"
  },
  "FOO": "foo"
}
Learn more about the map_keys function

De-dot keys

Source
. = {
    "labels": {
        "app.kubernetes.io/name": "mysql"
    }
}
map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
Return
{
  "labels": {
    "app_kubernetes_io/name": "mysql"
  }
}
Learn more about the map_keys function

Recursively map object keys

Source
val = {
    "a": 1,
    "b": [{ "c": 2 }, { "d": 3 }],
    "e": { "f": 4 }
}
map_keys(val, recursive: true) -> |key| { upcase(key) }
Return
{
  "A": 1,
  "B": [
    {
      "C": 2
    },
    {
      "D": 3
    }
  ],
  "E": {
    "F": 4
  }
}
Learn more about the map_keys function

map_values

Map the values within a collection.

If recursive is enabled, the function iterates into nested collections, using the following rules:

  1. Iteration starts at the root.
  2. For every nested collection type:
    • First return the collection type itself.
    • Then recurse into the collection, and loop back to item (1) in the list
    • Any mutation done on a collection before recursing into it, are preserved.

The function uses the function closure syntax to allow mutating the value for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

Check out the examples below to learn about the closure syntax.

Upcase values

Source
. = {
    "foo": "foo",
    "bar": "bar"
}
map_values(.) -> |value| { upcase(value) }
Return
{
  "bar": "BAR",
  "foo": "FOO"
}
Learn more about the map_values function

Recursively map object values

Source
val = {
    "a": 1,
    "b": [{ "c": 2 }, { "d": 3 }],
    "e": { "f": 4 }
}
map_values(val, recursive: true) -> |value| {
    if is_integer(value) { int!(value) + 1 } else { value }
}
Return
{
  "a": 2,
  "b": [
    {
      "c": 3
    },
    {
      "d": 4
    }
  ],
  "e": {
    "f": 5
  }
}
Learn more about the map_values function

match_array

Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.

Match at least one element

Source
match_array(["foobar", "bazqux"], r'foo')
Return
true
Learn more about the match_array function

Match all elements

Source
match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return
true
Learn more about the match_array function

No matches

Source
match_array(["bazqux", "xyz"], r'foo')
Learn more about the match_array function

Not all elements match

Source
match_array(["foo", "foobar", "baz"], r'foo', all: true)
Learn more about the match_array function

strlen

Returns the number of UTF-8 characters in value. This differs from length which counts the number of bytes of a string.

Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.

Count Unicode scalar values

Source
strlen("ñandú")
Return
5
Learn more about the strlen function

tally

Counts the occurrences of each string value in the provided array and returns an object with the counts.

tally

Source
tally!(["foo", "bar", "foo", "baz"])
Return
{
  "bar": 1,
  "baz": 1,
  "foo": 2
}
Learn more about the tally function

tally_value

Counts the number of times a specific value appears in the provided array.

count matching values

Source
tally_value(["foo", "bar", "foo", "baz"], "foo")
Return
2
Learn more about the tally_value function

unflatten

Unflattens the value into a nested representation.

Unflatten

Source
unflatten({
    "foo.bar.baz": true,
    "foo.bar.qux": false,
    "foo.quux": 42
})
Return
{
  "foo": {
    "bar": {
      "baz": true,
      "qux": false
    },
    "quux": 42
  }
}
Learn more about the unflatten function

Unflatten recursively

Source
unflatten({
    "flattened.parent": {
        "foo.bar": true,
        "foo.baz": false
    }
})
Return
{
  "flattened": {
    "parent": {
      "foo": {
        "bar": true,
        "baz": false
      }
    }
  }
}
Learn more about the unflatten function

Unflatten non-recursively

Source
unflatten({
    "flattened.parent": {
        "foo.bar": true,
        "foo.baz": false
    }
}, recursive: false)
Return
{
  "flattened": {
    "parent": {
      "foo.bar": true,
      "foo.baz": false
    }
  }
}
Learn more about the unflatten function

Ignore inconsistent keys values

Source
unflatten({
    "a": 3,
    "a.b": 2,
    "a.c": 4
})
Return
{
  "a": {
    "b": 2,
    "c": 4
  }
}
Learn more about the unflatten function

Unflatten with custom separator

Source
unflatten({ "foo_bar": true }, "_")
Return
{
  "foo": {
    "bar": true
  }
}
Learn more about the unflatten function

unique

Returns the unique values for an array.

The first occurrence of each element is kept.

Unique

Source
unique(["foo", "bar", "foo", "baz"])
Return
["foo","bar","baz"]
Learn more about the unique function

values

Returns the values from the object passed into the function.

Get values from the object

Source
values({"key1": "val1", "key2": "val2"})
Return
["val1","val2"]
Learn more about the values function

Get values from a complex object

Source
values({"key1": "val1", "key2": [1, 2, 3], "key3": {"foo": "bar"}})
Return
["val1",[1,2,3],{"foo":"bar"}]
Learn more about the values function

Event examples

get_secret

Returns the value of the given secret from an event.

Get the Datadog API key from the event metadata

Source
get_secret("datadog_api_key")
Return
secret value
Learn more about the get_secret function

Get a non existent secret

Source
get_secret("i_dont_exist")
Learn more about the get_secret function

remove_secret

Removes a secret from an event.

Remove the datadog api key

Source
remove_secret("datadog_api_key")
Learn more about the remove_secret function

set_secret

Sets the given secret in the event.

Set the datadog api key

Source
set_secret("datadog_api_key", "secret-value")
Learn more about the set_secret function

set_semantic_meaning

Sets a semantic meaning for an event.

Sets custom field semantic meaning

Source
set_semantic_meaning(.foo, "bar")
Learn more about the set_semantic_meaning function

Path examples

del

Removes the field specified by the static path from the target.

For dynamic path deletion, see the remove function.

Delete a field

Source
. = { "foo": "bar" }
del(.foo)
Return
bar
Learn more about the del function

Rename a field

Source
. = { "old": "foo" }
.new = del(.old)
.
Return
{
  "new": "foo"
}
Learn more about the del function

Returns null for unknown field

Source
del({"foo": "bar"}.baz)
Learn more about the del function

External target

Source
. = { "foo": true, "bar": 10 }
del(.foo)
.
Return
{
  "bar": 10
}
Learn more about the del function

Delete field from variable

Source
var = { "foo": true, "bar": 10 }
del(var.foo)
var
Return
{
  "bar": 10
}
Learn more about the del function

Delete object field

Source
var = { "foo": {"nested": true}, "bar": 10 }
del(var.foo.nested, false)
var
Return
{
  "bar": 10,
  "foo": {}
}
Learn more about the del function

Compact object field

Source
var = { "foo": {"nested": true}, "bar": 10 }
del(var.foo.nested, true)
var
Return
{
  "bar": 10
}
Learn more about the del function

exists

Checks whether the path exists for the target.

This function distinguishes between a missing path and a path with a null value. A regular path lookup, such as .foo, cannot distinguish between the two cases since it always returns null if the path doesn’t exist.

Exists (field)

Source
. = { "field": 1 }
exists(.field)
Return
true
Learn more about the exists function

Exists (array element)

Source
. = { "array": [1, 2, 3] }
exists(.array[2])
Return
true
Learn more about the exists function

Does not exist (field)

Source
exists({ "foo": "bar"}.baz)
Learn more about the exists function

get

Dynamically get the value of a given path.

If you know the path you want to look up, use static paths such as .foo.bar[1] to get the value of that path. However, if you do not know the path names, use the dynamic get function to get the requested value.

Single-segment top-level field

Source
get!(value: {"foo": "bar"}, path: ["foo"])
Return
bar
Learn more about the get function

Returns null for unknown field

Source
get!(value: {"foo": "bar"}, path: ["baz"])
Learn more about the get function

Multi-segment nested field

Source
get!(value: {"foo": { "bar": true }}, path: ["foo", "bar"])
Return
true
Learn more about the get function

Array indexing

Source
get!(value: [92, 42], path: [0])
Return
92
Learn more about the get function

Array indexing (negative)

Source
get!(value: ["foo", "bar", "baz"], path: [-2])
Return
bar
Learn more about the get function

Nested indexing

Source
get!(value: {"foo": { "bar": [92, 42] }}, path: ["foo", "bar", 1])
Return
42
Learn more about the get function

External target

Source
get!(value: ., path: ["foo"])
Return
true
Learn more about the get function

Variable

Source
var = { "foo": true }
get!(value: var, path: ["foo"])
Return
true
Learn more about the get function

Missing index

Source
get!(value: {"foo": { "bar": [92, 42] }}, path: ["foo", "bar", 1, -1])
Learn more about the get function

Invalid indexing

Source
get!(value: [42], path: ["foo"])
Learn more about the get function

Invalid segment type

Source
get!(value: {"foo": { "bar": [92, 42] }}, path: ["foo", true])
Learn more about the get function

remove

Dynamically remove the value for a given path.

If you know the path you want to remove, use the del function and static paths such as del(.foo.bar[1]) to remove the value at that path. The del function returns the deleted value, and is more performant than remove. However, if you do not know the path names, use the dynamic remove function to remove the value at the provided path.

Single-segment top-level field

Source
remove!(value: { "foo": "bar" }, path: ["foo"])
Learn more about the remove function

Remove unknown field

Source
remove!(value: {"foo": "bar"}, path: ["baz"])
Return
{
  "foo": "bar"
}
Learn more about the remove function

Multi-segment nested field

Source
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return
{
  "foo": {}
}
Learn more about the remove function

Array indexing

Source
remove!(value: ["foo", "bar", "baz"], path: [-2])
Return
["foo","baz"]
Learn more about the remove function

Compaction

Source
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)
Return
{
  "foo": {
    "baz": true
  }
}
Learn more about the remove function

Compact object

Source
remove!(value: {"foo": { "bar": true }}, path: ["foo", "bar"], compact: true)
Learn more about the remove function

Compact array

Source
remove!(value: {"foo": [42], "bar": true }, path: ["foo", 0], compact: true)
Return
{
  "bar": true
}
Learn more about the remove function

External target

Source
remove!(value: ., path: ["foo"])
Learn more about the remove function

Variable

Source
var = { "foo": true }
remove!(value: var, path: ["foo"])
Learn more about the remove function

Missing index

Source
remove!(value: {"foo": { "bar": [92, 42] }}, path: ["foo", "bar", 1, -1])
Return
{
  "foo": {
    "bar": [
      92,
      42
    ]
  }
}
Learn more about the remove function

Invalid indexing

Source
remove!(value: [42], path: ["foo"])
Return
[42]
Learn more about the remove function

Invalid segment type

Source
remove!(value: {"foo": { "bar": [92, 42] }}, path: ["foo", true])
Learn more about the remove function

set

Dynamically insert data into the path of a given object or array.

If you know the path you want to assign a value to, use static path assignments such as .foo.bar[1] = true for improved performance and readability. However, if you do not know the path names, use the dynamic set function to insert the data into the object or array.

Single-segment top-level field

Source
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")
Return
{
  "foo": "baz"
}
Learn more about the set function

Multi-segment nested field

Source
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")
Return
{
  "foo": {
    "bar": "qux"
  }
}
Learn more about the set function

Array

Source
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)
Return
["foo",42,"baz"]
Learn more about the set function

Nested fields

Source
set!(value: {}, path: ["foo", "bar"], data: "baz")
Return
{
  "foo": {
    "bar": "baz"
  }
}
Learn more about the set function

Nested indexing

Source
set!(value: {"foo": { "bar": [] }}, path: ["foo", "bar", 1], data: "baz")
Return
{
  "foo": {
    "bar": [
      null,
      "baz"
    ]
  }
}
Learn more about the set function

External target

Source
set!(value: ., path: ["bar"], data: "baz")
Return
{
  "bar": "baz",
  "foo": true
}
Learn more about the set function

Variable

Source
var = { "foo": true }
set!(value: var, path: ["bar"], data: "baz")
Return
{
  "bar": "baz",
  "foo": true
}
Learn more about the set function

Invalid indexing

Source
set!(value: [], path: ["foo"], data: "baz")
Return
{
  "foo": "baz"
}
Learn more about the set function

Invalid segment type

Source
set!({"foo": { "bar": [92, 42] }}, ["foo", true], "baz")
Learn more about the set function

Cryptography examples

decrypt

Decrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • AES-128-SIV (key = 32 bytes, iv = 16 bytes)
  • AES-256-SIV (key = 64 bytes, iv = 16 bytes)
  • Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
  • CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
  • XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
  • XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

Decrypt value using AES-256-CFB

Source
iv = "0123456789012345"
key = "01234567890123456789012345678912"
ciphertext = decode_base64!("c/dIOA==")
decrypt!(ciphertext, "AES-256-CFB", key: key, iv: iv)
Return
data
Learn more about the decrypt function

Decrypt value using AES-128-CBC-PKCS7

Source
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key: key, iv: iv)
Return
super_secret_message
Learn more about the decrypt function

encrypt

Encrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • AES-128-SIV (key = 32 bytes, iv = 16 bytes)
  • AES-256-SIV (key = 64 bytes, iv = 16 bytes)
  • Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
  • CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
  • XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
  • XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

Encrypt value using AES-256-CFB

Source
iv = "0123456789012345" # typically you would call random_bytes(16)
key = "01234567890123456789012345678912"
encrypted_message = encrypt!("data", "AES-256-CFB", key: key, iv: iv)
encode_base64(encrypted_message)
Return
c/dIOA==
Learn more about the encrypt function

Encrypt value using AES-128-CBC-PKCS7

Source
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!("super secret message", "AES-128-CBC-PKCS7", key: key, iv: iv)
encode_base64(encrypted_message)
Return
GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=
Learn more about the encrypt function

hmac

Calculates a HMAC of the value using the given key. The hashing algorithm used can be optionally specified.

For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.

This function is infallible if either the default algorithm value or a recognized-valid compile-time algorithm string literal is used. Otherwise, it is fallible.

Calculate message HMAC (defaults: SHA-256), encoding to a base64 string

Source
encode_base64(hmac("Hello there", "super-secret-key"))
Return
eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=
Learn more about the hmac function

Calculate message HMAC using SHA-224, encoding to a hex-encoded string

Source
encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))
Return
42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90
Learn more about the hmac function

Calculate message HMAC using SHA1, encoding to a base64 string

Source
encode_base64(hmac("Hello there", "super-secret-key", algorithm: "SHA1"))
Return
MiyBIHO8Set9+6crALiwkS0yFPE=
Learn more about the hmac function

Calculate message HMAC using a variable hash algorithm

Source
.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
    .hmac = encode_base16(hmac_bytes)
}
Return
78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662
Learn more about the hmac function

md5

Calculates an md5 hash of the value.

Create md5 hash

Source
md5("foo")
Return
acbd18db4cc2f85cedef654fccc4a4d8
Learn more about the md5 function

seahash

Calculates a Seahash hash of the value. Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.

Calculate seahash

Source
seahash("foobar")
Return
5348458858952426000
Learn more about the seahash function

Calculate negative seahash

Source
seahash("bar")
Return
-2796170501982571500
Learn more about the seahash function

sha1

Calculates a SHA-1 hash of the value.

Calculate sha1 hash

Source
sha1("foo")
Return
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33
Learn more about the sha1 function

sha2

Calculates a SHA-2 hash of the value.

Calculate sha2 hash using default variant

Source
sha2("foobar")
Return
d014c752bc2be868e16330f47e0c316a5967bcbc9c286a457761d7055b9214ce
Learn more about the sha2 function

Calculate sha2 hash with SHA-512/224

Source
sha2("foo", variant: "SHA-512/224")
Return
d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be
Learn more about the sha2 function

Calculate sha2 hash with SHA-384

Source
sha2("foobar", "SHA-384")
Return
3c9c30d9f665e74d515c842960d4a451c83a0125fd3de7392d7b37231af10c72ea58aedfcdf89a5765bf902af93ecf06
Learn more about the sha2 function

sha3

Calculates a SHA-3 hash of the value.

Calculate sha3 hash using default variant

Source
sha3("foobar")
Return
ff32a30c3af5012ea395827a3e99a13073c3a8d8410a708568ff7e6eb85968fccfebaea039bc21411e9d43fdb9a851b529b9960ffea8679199781b8f45ca85e2
Learn more about the sha3 function

Calculate sha3 hash with SHA3-224

Source
sha3("foo", variant: "SHA3-224")
Return
f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a
Learn more about the sha3 function

Calculate sha3 hash with SHA3-384

Source
sha3("foobar", "SHA3-384")
Return
0fa8abfbdaf924ad307b74dd2ed183b9a4a398891a2f6bac8fd2db7041b77f068580f9c6c66f699b496c2da1cbcc7ed8
Learn more about the sha3 function

IP examples

decrypt_ip

Decrypts an IP address that was previously encrypted, restoring the original IP address.

Supported Modes:

  • AES128 - Decrypts an IP address that was scrambled using AES-128 encryption. Can transform between IPv4 and IPv6.
  • PFX (Prefix-preserving) - Decrypts an IP address that was encrypted with prefix-preserving mode, where network hierarchy was maintained.

Decrypt IPv4 address with AES128

Source
decrypt_ip!("72b9:a747:f2e9:72af:76ca:5866:6dcf:c3b0", "sixteen byte key", "aes128")
Return
192.168.1.1
Learn more about the decrypt_ip function

Decrypt IPv6 address with AES128

Source
decrypt_ip!("c0e6:eb35:6887:f554:4c65:8ace:17ca:6c6a", "sixteen byte key", "aes128")
Return
2001:db8::1
Learn more about the decrypt_ip function

Decrypt IPv4 address with prefix-preserving mode

Source
decrypt_ip!("33.245.248.61", "thirty-two bytes key for pfx use", "pfx")
Return
192.168.1.1
Learn more about the decrypt_ip function

Decrypt IPv6 address with prefix-preserving mode

Source
decrypt_ip!("88bd:d2bf:8865:8c4d:84b:44f6:6077:72c9", "thirty-two bytes key for ipv6pfx", "pfx")
Return
2001:db8::1
Learn more about the decrypt_ip function

Round-trip encryption and decryption

Source
original_ip = "192.168.1.100"
key = "sixteen byte key"
mode = "aes128"

encrypted = encrypt_ip!(original_ip, key, mode)
decrypt_ip!(encrypted, key, mode)
Return
192.168.1.100
Learn more about the decrypt_ip function

encrypt_ip

Encrypts an IP address, transforming it into a different valid IP address.

Supported Modes:

  • AES128 - Scrambles the entire IP address using AES-128 encryption. Can transform between IPv4 and IPv6.
  • PFX (Prefix-preserving) - Maintains network hierarchy by ensuring that IP addresses within the same network are encrypted to addresses that also share a common network. This preserves prefix relationships while providing confidentiality.

Encrypt IPv4 address with AES128

Source
encrypt_ip!("192.168.1.1", "sixteen byte key", "aes128")
Return
72b9:a747:f2e9:72af:76ca:5866:6dcf:c3b0
Learn more about the encrypt_ip function

Encrypt IPv6 address with AES128

Source
encrypt_ip!("2001:db8::1", "sixteen byte key", "aes128")
Return
c0e6:eb35:6887:f554:4c65:8ace:17ca:6c6a
Learn more about the encrypt_ip function

Encrypt IPv4 address with prefix-preserving mode

Source
encrypt_ip!("192.168.1.1", "thirty-two bytes key for pfx use", "pfx")
Return
33.245.248.61
Learn more about the encrypt_ip function

Encrypt IPv6 address with prefix-preserving mode

Source
encrypt_ip!("2001:db8::1", "thirty-two bytes key for ipv6pfx", "pfx")
Return
88bd:d2bf:8865:8c4d:84b:44f6:6077:72c9
Learn more about the encrypt_ip function

ip_aton

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

IPv4 to integer

Source
ip_aton!("1.2.3.4")
Return
16909060
Learn more about the ip_aton function

ip_cidr_contains

Determines whether the ip is contained in the block referenced by the cidr.

IPv4 contains CIDR

Source
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return
true
Learn more about the ip_cidr_contains function

IPv4 is private

Source
ip_cidr_contains!(["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"], "192.168.10.32")
Return
true
Learn more about the ip_cidr_contains function

IPv6 contains CIDR

Source
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return
true
Learn more about the ip_cidr_contains function

Not in range

Source
ip_cidr_contains!("192.168.0.0/24", "192.168.10.32")
Learn more about the ip_cidr_contains function

Invalid address

Source
ip_cidr_contains!("192.168.0.0/24", "INVALID")
Learn more about the ip_cidr_contains function

ip_ntoa

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.

This behavior mimics inet_ntoa.

Integer to IPv4

Source
ip_ntoa!(16909060)
Return
1.2.3.4
Learn more about the ip_ntoa function

ip_ntop

Converts IPv4 and IPv6 addresses from binary to text form.

This behavior mimics inet_ntop.

Convert IPv4 address from bytes after decoding from Base64

Source
ip_ntop!(decode_base64!("wKgAAQ=="))
Return
192.168.0.1
Learn more about the ip_ntop function

Convert IPv6 address from bytes after decoding from Base64

Source
ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))
Return
2001:db8:85a3::8a2e:370:7334
Learn more about the ip_ntop function

ip_pton

Converts IPv4 and IPv6 addresses from text to binary form.

  • The binary form of IPv4 addresses is 4 bytes (32 bits) long.
  • The binary form of IPv6 addresses is 16 bytes (128 bits) long.

This behavior mimics inet_pton.

Convert IPv4 address to bytes and encode to Base64

Source
encode_base64(ip_pton!("192.168.0.1"))
Return
wKgAAQ==
Learn more about the ip_pton function

Convert IPv6 address to bytes and encode to Base64

Source
encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))
Return
IAENuIWjAAAAAIouA3BzNA==
Learn more about the ip_pton function

ip_subnet

Extracts the subnet address from the ip using the supplied subnet.

IPv4 subnet

Source
ip_subnet!("192.168.10.32", "255.255.255.0")
Return
192.168.10.0
Learn more about the ip_subnet function

IPv6 subnet

Source
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return
2404:6800::
Learn more about the ip_subnet function

Subnet /1

Source
ip_subnet!("192.168.0.1", "/1")
Return
128.0.0.0
Learn more about the ip_subnet function

ip_to_ipv6

Converts the ip to an IPv6 address.

IPv4 to IPv6

Source
ip_to_ipv6!("192.168.10.32")
Return
::ffff:192.168.10.32
Learn more about the ip_to_ipv6 function

ipv6_to_ipv4

Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

IPv6 to IPv4

Source
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return
192.168.0.1
Learn more about the ipv6_to_ipv4 function

is_ipv4

Check if the string is a valid IPv4 address or not.

An IPv4-mapped or IPv4-compatible IPv6 address is not considered valid for the purpose of this function.

Valid IPv4 address

Source
is_ipv4("10.0.102.37")
Return
true
Learn more about the is_ipv4 function

Valid IPv6 address

Source
is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Learn more about the is_ipv4 function

Arbitrary string

Source
is_ipv4("foobar")
Learn more about the is_ipv4 function

is_ipv6

Check if the string is a valid IPv6 address or not.

Valid IPv6 address

Source
is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return
true
Learn more about the is_ipv6 function

Valid IPv4 address

Source
is_ipv6("10.0.102.37")
Learn more about the is_ipv6 function

Arbitrary string

Source
is_ipv6("foobar")
Learn more about the is_ipv6 function

Map examples

haversine

Calculates haversine distance and bearing between two points.

Haversine in kilometers

Source
haversine(0.0, 0.0, 10.0, 10.0)
Return
{
  "bearing": 44.561,
  "distance": 1568.5227233
}
Learn more about the haversine function

Haversine in miles

Source
haversine(0.0, 0.0, 10.0, 10.0, measurement_unit: "miles")
Return
{
  "bearing": 44.561,
  "distance": 974.6348468
}
Learn more about the haversine function

Metrics examples

aggregate_vector_metrics

Aggregates internal Vector metrics, using one of 4 aggregation functions, filtering by name and optionally by tags. Returns the aggregated value. Only includes counter and gauge metrics.

Internal Vector metrics functions work with a snapshot of the metrics. The interval at which the snapshot is updated is controlled through the metrics_storage_refresh_period global option. Higher values can reduce performance impact of that process, but may cause stale metrics data in the snapshot.

Sum vector internal metrics matching the name

Source
aggregate_vector_metrics("sum", "utilization")
Return
0.5
Learn more about the aggregate_vector_metrics function

Sum vector internal metrics matching the name and tags

Source
aggregate_vector_metrics("sum", "utilization", tags: {"component_id": "test"})
Return
0.5
Learn more about the aggregate_vector_metrics function

Average of vector internal metrics matching the name

Source
aggregate_vector_metrics("avg", "utilization")
Return
0.5
Learn more about the aggregate_vector_metrics function

Max of vector internal metrics matching the name

Source
aggregate_vector_metrics("max", "utilization")
Return
0.5
Learn more about the aggregate_vector_metrics function

Min of vector internal metrics matching the name

Source
aggregate_vector_metrics("max", "utilization")
Return
0.5
Learn more about the aggregate_vector_metrics function

find_vector_metrics

Searches internal Vector metrics by name and optionally by tags. Returns all matching metrics.

Internal Vector metrics functions work with a snapshot of the metrics. The interval at which the snapshot is updated is controlled through the metrics_storage_refresh_period global option. Higher values can reduce performance impact of that process, but may cause stale metrics data in the snapshot.

Find vector internal metrics matching the name

Source
find_vector_metrics("utilization")
Return
[{"kind":"absolute","name":"utilization","tags":{"component_id":["test"]},"type":"gauge","value":0.5}]
Learn more about the find_vector_metrics function

Find vector internal metrics matching the name and tags

Source
find_vector_metrics("utilization", tags: {"component_id": "test"})
Return
[{"kind":"absolute","name":"utilization","tags":{"component_id":["test"]},"type":"gauge","value":0.5}]
Learn more about the find_vector_metrics function

get_vector_metric

Searches internal Vector metrics by name and optionally by tags. Returns the first matching metric.

Internal Vector metrics functions work with a snapshot of the metrics. The interval at which the snapshot is updated is controlled through the metrics_storage_refresh_period global option. Higher values can reduce performance impact of that process, but may cause stale metrics data in the snapshot.

Get a vector internal metric matching the name

Source
get_vector_metric("utilization")
Return
{
  "kind": "absolute",
  "name": "utilization",
  "tags": {
    "component_id": [
      "test"
    ]
  },
  "type": "gauge",
  "value": 0.5
}
Learn more about the get_vector_metric function

Get a vector internal metric matching the name and tags

Source
get_vector_metric("utilization", tags: {"component_id": "test"})
Return
{
  "kind": "absolute",
  "name": "utilization",
  "tags": {
    "component_id": [
      "test"
    ]
  },
  "type": "gauge",
  "value": 0.5
}
Learn more about the get_vector_metric function

Number examples

abs

Computes the absolute value of value.

Computes the absolute value of an integer

Source
abs(-42)
Return
42
Learn more about the abs function

Computes the absolute value of a float

Source
abs(-42.2)
Return
42.2
Learn more about the abs function

Computes the absolute value of a positive integer

Source
abs(10)
Return
10
Learn more about the abs function

ceil

Rounds the value up to the specified precision.

Round a number up (without precision)

Source
ceil(4.345)
Return
5
Learn more about the ceil function

Round a number up (with precision)

Source
ceil(4.345, precision: 2)
Return
4.35
Learn more about the ceil function

Round an integer up (noop)

Source
ceil(5)
Return
5
Learn more about the ceil function

floor

Rounds the value down to the specified precision.

Round a number down (without precision)

Source
floor(9.8)
Return
9
Learn more about the floor function

Round a number down (with precision)

Source
floor(4.345, precision: 2)
Return
4.34
Learn more about the floor function

format_int

Formats the integer value into a string representation using the given base/radix.

Format as a hexadecimal integer

Source
format_int!(42, 16)
Return
2a
Learn more about the format_int function

Format as a negative hexadecimal integer

Source
format_int!(-42, 16)
Return
-2a
Learn more about the format_int function

Format as a decimal integer (default base)

Source
format_int!(42)
Return
42
Learn more about the format_int function

format_number

Formats the value into a string representation of the number.

Format a number (3 decimals)

Source
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return
1,234,567.890
Learn more about the format_number function

Format a number with European-style separators

Source
format_number(4672.4, decimal_separator: ",", grouping_separator: "_")
Return
4_672,4
Learn more about the format_number function

Format a number with a middle dot separator

Source
format_number(4321.09, 3, decimal_separator: "·")
Return
4321·090
Learn more about the format_number function

mod

Calculates the remainder of value divided by modulus.

Calculate the remainder of two integers

Source
mod(5, 2)
Return
1
Learn more about the mod function

round

Rounds the value to the specified precision.

Round a number (without precision)

Source
round(4.345)
Return
4
Learn more about the round function

Round a number (with precision)

Source
round(4.345, precision: 2)
Return
4.35
Learn more about the round function

Round up

Source
round(5.5)
Return
6
Learn more about the round function

Round down

Source
round(5.45)
Return
5
Learn more about the round function

Object examples

match_datadog_query

Matches an object against a Datadog Search Syntax query.

OR query

Source
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return
true
Learn more about the match_datadog_query function

AND query

Source
match_datadog_query({"message": "contains only this"}, "this AND that")
Learn more about the match_datadog_query function

Attribute wildcard

Source
match_datadog_query({"name": "foobar"}, "@name:foo*")
Return
true
Learn more about the match_datadog_query function

Tag range

Source
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return
true
Learn more about the match_datadog_query function

merge

Merges the from object into the to object.

Object merge (shallow)

Source
merge(
    {
        "parent1": {
            "child1": 1,
            "child2": 2
        },
        "parent2": {
            "child3": 3
        }
    },
    {
        "parent1": {
            "child2": 4,
            "child5": 5
        }
    }
)
Return
{
  "parent1": {
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}
Learn more about the merge function

Object merge (deep)

Source
merge(
    {
        "parent1": {
            "child1": 1,
            "child2": 2
        },
        "parent2": {
            "child3": 3
        }
    },
    {
        "parent1": {
            "child2": 4,
            "child5": 5
        }
    },
    deep: true
)
Return
{
  "parent1": {
    "child1": 1,
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}
Learn more about the merge function

object_from_array

Iterate over either one array of arrays or a pair of arrays and create an object out of all the key-value pairs contained in them. With one array of arrays, any entries with no value use null instead. Any keys that are null skip the corresponding value.

If a single parameter is given, it must contain an array of all the input arrays.

Create an object from one array

Source
object_from_array([["one", 1], [null, 2], ["two", 3]])
Return
{
  "one": 1,
  "two": 3
}
Learn more about the object_from_array function

Create an object from separate key and value arrays

Source
object_from_array([1, 2, 3], keys: ["one", null, "two"])
Return
{
  "one": 1,
  "two": 3
}
Learn more about the object_from_array function

Create an object from a separate arrays of keys and values

Source
object_from_array(values: [1, null, true], keys: ["a", "b", "c"])
Return
{
  "a": 1,
  "b": null,
  "c": true
}
Learn more about the object_from_array function

unnest

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . results in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explode in some languages.

Unnest an array field

Source
. = {"hostname": "localhost", "messages": ["message 1", "message 2"]}
. = unnest(.messages)
Return
[{"hostname":"localhost","messages":"message 1"},{"hostname":"localhost","messages":"message 2"}]
Learn more about the unnest function

Unnest a nested array field

Source
. = {"hostname": "localhost", "event": {"messages": ["message 1", "message 2"]}}
. = unnest(.event.messages)
Return
[{"event":{"messages":"message 1"},"hostname":"localhost"},{"event":{"messages":"message 2"},"hostname":"localhost"}]
Learn more about the unnest function

Parse examples

parse_apache_log

Parses Apache access and error log lines. Lines can be in common, combined, or the default error format.

Parse using Apache log format (common)

Source
parse_apache_log!(s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326', format: "common")
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_apache_log function

Parse using Apache log format (combined)

Source
parse_apache_log!(
    s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
    "combined",
)
Return
{
  "agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_apache_log function

Parse using Apache log format (error)

Source
parse_apache_log!(
    s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
    "error"
)
Return
{
  "client": "147.159.108.175",
  "message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
  "module": "ab",
  "pid": 4803,
  "port": 24259,
  "severity": "alert",
  "thread": "3814",
  "timestamp": "2021-03-01T12:00:19Z"
}
Learn more about the parse_apache_log function

parse_aws_alb_log

Parses value in the Elastic Load Balancer Access format.

Parse AWS ALB log

Source
parse_aws_alb_log!(
    "http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return
{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "traceability_id": null,
  "type": "http",
  "user_agent": "curl/7.46.0"
}
Learn more about the parse_aws_alb_log function

Parse AWS ALB log with trailing fields (non-strict mode)

Source
parse_aws_alb_log!(
    "http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" TID_12345 \"-\" \"-\" \"-\"",
    strict_mode: false
)
Return
{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "traceability_id": "TID_12345",
  "type": "http",
  "user_agent": "curl/7.46.0"
}
Learn more about the parse_aws_alb_log function

parse_aws_cloudwatch_log_subscription_message

Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

Parse AWS Cloudwatch Log subscription message

Source
parse_aws_cloudwatch_log_subscription_message!(s'{
    "messageType": "DATA_MESSAGE",
    "owner": "111111111111",
    "logGroup": "test",
    "logStream": "test",
    "subscriptionFilters": [
        "Destination"
    ],
    "logEvents": [
        {
            "id": "35683658089614582423604394983260738922885519999578275840",
            "timestamp": 1600110569039,
            "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41-0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}"
        }
    ]
}')
Return
{
  "log_events": [
    {
      "id": "35683658089614582423604394983260738922885519999578275840",
      "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41-0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
      "timestamp": "2020-09-14T19:09:29.039Z"
    }
  ],
  "log_group": "test",
  "log_stream": "test",
  "message_type": "DATA_MESSAGE",
  "owner": "111111111111",
  "subscription_filters": [
    "Destination"
  ]
}
Learn more about the parse_aws_cloudwatch_log_subscription_message function

parse_aws_vpc_flow_log

Parses value in the VPC Flow Logs format.

Parse AWS VPC Flow log (default format)

Source
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return
{
  "account_id": "123456789010",
  "action": null,
  "bytes": null,
  "dstaddr": null,
  "dstport": null,
  "end": 1431280934,
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "NODATA",
  "packets": null,
  "protocol": null,
  "srcaddr": null,
  "srcport": null,
  "start": 1431280876,
  "version": 2
}
Learn more about the parse_aws_vpc_flow_log function

Parse AWS VPC Flow log (custom format)

Source
parse_aws_vpc_flow_log!(
    "- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
    "instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return
{
  "dstaddr": "10.0.0.220",
  "instance_id": null,
  "interface_id": "eni-1235b8ca123456789",
  "pkt_dstaddr": "203.0.113.5",
  "pkt_srcaddr": "10.0.1.5",
  "srcaddr": "10.0.1.5"
}
Learn more about the parse_aws_vpc_flow_log function

Parse AWS VPC Flow log including v5 fields

Source
parse_aws_vpc_flow_log!(
    "5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
    format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status"
)
Return
{
  "account_id": "123456789012",
  "action": "ACCEPT",
  "az_id": "apse2-az3",
  "bytes": 15044,
  "dstaddr": "10.0.0.71",
  "dstport": 34210,
  "end": 1616729349,
  "flow_direction": "ingress",
  "instance_id": "i-0c50d5961bcb2d47b",
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "OK",
  "packets": 14,
  "pkt_dst_aws_service": null,
  "pkt_dstaddr": "10.0.0.71",
  "pkt_src_aws_service": "S3",
  "pkt_srcaddr": "52.95.128.179",
  "protocol": 6,
  "region": "ap-southeast-2",
  "srcaddr": "52.95.128.179",
  "srcport": 80,
  "start": 1616729292,
  "sublocation_id": null,
  "sublocation_type": null,
  "subnet_id": "subnet-aaaaaaaa012345678",
  "tcp_flags": 19,
  "traffic_path": null,
  "type": "IPv4",
  "version": 5,
  "vpc_id": "vpc-abcdefab012345678"
}
Learn more about the parse_aws_vpc_flow_log function

parse_bytes

Parses the value into a human-readable bytes format specified by unit and base.

Parse bytes (kilobytes)

Source
parse_bytes!("1024KiB", unit: "MiB")
Return
1
Learn more about the parse_bytes function

Parse kilobytes in default binary units

Source
parse_bytes!("1KiB", unit: "B")
Return
1024
Learn more about the parse_bytes function

Parse bytes in SI unit (terabytes)

Source
parse_bytes!("4TB", unit: "MB", base: "10")
Return
4000000
Learn more about the parse_bytes function

Parse gigabytes in decimal units

Source
parse_bytes!("1GB", unit: "B", base: "10")
Return
1000000000
Learn more about the parse_bytes function

Parse bytes in ambiguous unit (gigabytes)

Source
parse_bytes!("1GB", unit: "B", base: "2")
Return
1073741824
Learn more about the parse_bytes function

Parse gigabytes in ambiguous decimal units

Source
parse_bytes!("1GB", unit: "MB", base: "2")
Return
1024
Learn more about the parse_bytes function

parse_cbor

Parses the value as CBOR.

Parse CBOR

Source
parse_cbor!(decode_base64!("oWVmaWVsZGV2YWx1ZQ=="))
Return
{
  "field": "value"
}
Learn more about the parse_cbor function

array

Source
parse_cbor!(decode_base64!("gvUA"))
Return
[true,0]
Learn more about the parse_cbor function

string

Source
parse_cbor!(decode_base64!("ZWhlbGxv"))
Return
hello
Learn more about the parse_cbor function

integer

Source
parse_cbor!(decode_base64!("GCo="))
Return
42
Learn more about the parse_cbor function

float

Source
parse_cbor!(decode_base64!("+0BFEKPXCj1x"))
Return
42.13
Learn more about the parse_cbor function

boolean

Source
parse_cbor!(decode_base64!("9A=="))
Learn more about the parse_cbor function

parse_cef

Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.

Parse output generated by PTA

Source
parse_cef!(
    "CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
Return
{
  "cefVersion": "0",
  "cs1": "None",
  "cs1Label": "ExtraData",
  "cs2": "52b06812ec3500ed864c461e",
  "cs2Label": "EventID",
  "cs3": "https://1.1.1.1/incidents/52b06812ec3500ed864c461e",
  "cs3Label": "PTAlink",
  "cs4": "None",
  "cs4Label": "ExternalLink",
  "deviceCustomDate1": "1388577900000",
  "deviceCustomDate1Label": "detectionDate",
  "deviceEventClassId": "1",
  "deviceProduct": "PTA",
  "deviceVendor": "CyberArk",
  "deviceVersion": "12.6",
  "dhost": "dev1.domain.com",
  "dst": "2.2.2.2",
  "duser": "andy@dev1.domain.com",
  "name": "Suspected credentials theft",
  "severity": "8",
  "shost": "prod1.domain.com",
  "src": "1.1.1.1",
  "suser": "mike2@prod1.domain.com"
}
Learn more about the parse_cef function

Ignore syslog header

Source
parse_cef!(
    "Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
Return
{
  "cefVersion": "1",
  "deviceEventClassId": "100",
  "deviceProduct": "threatmanager",
  "deviceVendor": "Security",
  "deviceVersion": "1.0",
  "dst": "2.1.2.2",
  "name": "worm successfully stopped",
  "severity": "10",
  "spt": "1232",
  "src": "10.0.0.1"
}
Learn more about the parse_cef function

Translate custom fields

Source
parse_cef!(
    "CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
    translate_custom_fields: true
)
Return
{
  "Device IPv6 Address": "2345:0425:2CA1:0000:0000:0567:5673:23b5",
  "cefVersion": "0",
  "deviceEventClassId": "1",
  "deviceProduct": "firewall",
  "deviceVendor": "Dev",
  "deviceVersion": "2.2",
  "name": "Connection denied",
  "severity": "5"
}
Learn more about the parse_cef function

Parse CEF with only header

Source
parse_cef!("CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|")
Return
{
  "cefVersion": "1",
  "deviceEventClassId": "100",
  "deviceProduct": "threatmanager",
  "deviceVendor": "Security",
  "deviceVersion": "1.0",
  "name": "worm successfully stopped",
  "severity": "10"
}
Learn more about the parse_cef function

Parse CEF with empty value

Source
parse_cef!("CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft||suser=mike2@prod1.domain.com shost= src=1.1.1.1")
Return
{
  "cefVersion": "0",
  "deviceEventClassId": "1",
  "deviceProduct": "PTA",
  "deviceVendor": "CyberArk",
  "deviceVersion": "12.6",
  "name": "Suspected credentials theft",
  "severity": "",
  "shost": "",
  "src": "1.1.1.1",
  "suser": "mike2@prod1.domain.com"
}
Learn more about the parse_cef function

Parse CEF with escapes

Source
parse_cef!(s'CEF:0|security|threatmanager|1.0|100|Detected a \| in message. No action needed.|10|src=10.0.0.1 msg=Detected a threat.\n No action needed act=blocked a \= dst=1.1.1.1')
Return
{
  "act": "blocked a =",
  "cefVersion": "0",
  "deviceEventClassId": "100",
  "deviceProduct": "threatmanager",
  "deviceVendor": "security",
  "deviceVersion": "1.0",
  "dst": "1.1.1.1",
  "msg": "Detected a threat.\n No action needed",
  "name": "Detected a | in message. No action needed.",
  "severity": "10",
  "src": "10.0.0.1"
}
Learn more about the parse_cef function

parse_common_log

Parses the value using the Common Log Format (CLF).

Parse using Common Log Format (with default timestamp format)

Source
parse_common_log!(s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326')
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_common_log function

Parse using Common Log Format (with custom timestamp format)

Source
parse_common_log!(
    s'127.0.0.1 bob frank [2000-10-10T20:55:36Z] "GET /apache_pb.gif HTTP/1.0" 200 2326',
    "%+"
)
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_common_log function

parse_csv

Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

Parse a single CSV formatted row

Source
parse_csv!(s'foo,bar,"foo "", bar"')
Return
["foo","bar","foo \", bar"]
Learn more about the parse_csv function

Parse a single CSV formatted row with custom delimiter

Source
parse_csv!("foo bar", delimiter: " ")
Return
["foo","bar"]
Learn more about the parse_csv function

parse_dnstap

Parses the value as base64 encoded DNSTAP data.

Parse dnstap query message

Source
parse_dnstap!("ChVqYW1lcy1WaXJ0dWFsLU1hY2hpbmUSC0JJTkQgOS4xNi4zGgBy5wEIAxACGAEiEAAAAAAAAAAAAAAAAAAAAAAqECABBQJwlAAAAAAAAAAAADAw8+0CODVA7+zq9wVNMU3WNlI2kwIAAAABAAAAAAABCWZhY2Vib29rMQNjb20AAAEAAQAAKQIAAACAAAAMAAoACOxjCAG9zVgzWgUDY29tAGAAbQAAAAByZLM4AAAAAQAAAAAAAQJoNQdleGFtcGxlA2NvbQAABgABAAApBNABAUAAADkADwA1AAlubyBTRVAgbWF0Y2hpbmcgdGhlIERTIGZvdW5kIGZvciBkbnNzZWMtZmFpbGVkLm9yZy54AQ==")
Return
{
  "dataType": "Message",
  "dataTypeId": 1,
  "extraInfo": "",
  "messageType": "ResolverQuery",
  "messageTypeId": 3,
  "queryZone": "com.",
  "requestData": {
    "fullRcode": 0,
    "header": {
      "aa": false,
      "ad": false,
      "anCount": 0,
      "arCount": 1,
      "cd": false,
      "id": 37634,
      "nsCount": 0,
      "opcode": 0,
      "qdCount": 1,
      "qr": 0,
      "ra": false,
      "rcode": 0,
      "rd": false,
      "tc": false
    },
    "opt": {
      "do": true,
      "ednsVersion": 0,
      "extendedRcode": 0,
      "options": [
        {
          "optCode": 10,
          "optName": "Cookie",
          "optValue": "7GMIAb3NWDM="
        }
      ],
      "udpPayloadSize": 512
    },
    "question": [
      {
        "class": "IN",
        "domainName": "facebook1.com.",
        "questionType": "A",
        "questionTypeId": 1
      }
    ],
    "rcodeName": "NoError"
  },
  "responseAddress": "2001:502:7094::30",
  "responseData": {
    "fullRcode": 16,
    "header": {
      "aa": false,
      "ad": false,
      "anCount": 0,
      "arCount": 1,
      "cd": false,
      "id": 45880,
      "nsCount": 0,
      "opcode": 0,
      "qdCount": 1,
      "qr": 0,
      "ra": false,
      "rcode": 16,
      "rd": false,
      "tc": false
    },
    "opt": {
      "do": false,
      "ede": [
        {
          "extraText": "no SEP matching the DS found for dnssec-failed.org.",
          "infoCode": 9,
          "purpose": "DNSKEY Missing"
        }
      ],
      "ednsVersion": 1,
      "extendedRcode": 1,
      "udpPayloadSize": 1232
    },
    "question": [
      {
        "class": "IN",
        "domainName": "h5.example.com.",
        "questionType": "SOA",
        "questionTypeId": 6
      }
    ],
    "rcodeName": "BADVERS"
  },
  "responsePort": 53,
  "serverId": "james-Virtual-Machine",
  "serverVersion": "BIND 9.16.3",
  "socketFamily": "INET6",
  "socketProtocol": "UDP",
  "sourceAddress": "::",
  "sourcePort": 46835,
  "time": 1593489007920014000,
  "timePrecision": "ns",
  "timestamp": "2020-06-30T03:50:07.920014129Z"
}
Learn more about the parse_dnstap function

parse_duration

Parses the value into a human-readable duration format specified by unit.

Parse duration (milliseconds)

Source
parse_duration!("1005ms", unit: "s")
Return
1.005
Learn more about the parse_duration function

Parse multiple durations (seconds & milliseconds)

Source
parse_duration!("1s 1ms", unit: "ms")
Return
1001
Learn more about the parse_duration function

parse_etld

Parses the eTLD from value representing domain name.

Parse eTLD

Source
parse_etld!("sub.sussex.ac.uk")
Return
{
  "etld": "ac.uk",
  "etld_plus": "ac.uk",
  "known_suffix": true
}
Learn more about the parse_etld function

Parse eTLD+1

Source
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)
Return
{
  "etld": "ac.uk",
  "etld_plus": "sussex.ac.uk",
  "known_suffix": true
}
Learn more about the parse_etld function

Parse eTLD with unknown suffix

Source
parse_etld!("vector.acmecorp")
Return
{
  "etld": "acmecorp",
  "etld_plus": "acmecorp",
  "known_suffix": false
}
Learn more about the parse_etld function

Parse eTLD with custom PSL

Source
parse_etld!("vector.acmecorp", psl: "lib/tests/tests/functions/custom_public_suffix_list.dat")
Return
{
  "etld": "acmecorp",
  "etld_plus": "acmecorp",
  "known_suffix": false
}
Learn more about the parse_etld function

parse_glog

Parses the value using the glog (Google Logging Library) format.

Parse using glog

Source
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return
{
  "file": "main.c++",
  "id": 15520,
  "level": "info",
  "line": 9,
  "message": "Hello world!",
  "timestamp": "2021-01-31T14:48:54.411655Z"
}
Learn more about the parse_glog function

parse_grok

Parses the value using the grok format. All patterns listed here are supported.

Parse using Grok

Source
value = "2020-10-02T23:22:12.223222Z info Hello world"
pattern = "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"

parse_grok!(value, pattern)
Return
{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}
Learn more about the parse_grok function

parse_groks

Parses the value using multiple grok patterns. All patterns listed here are supported.

Parse using multiple Grok patterns

Source
parse_groks!(
    "2020-10-02T23:22:12.223222Z info Hello world",
    patterns: [
        "%{common_prefix} %{_status} %{_message}",
        "%{common_prefix} %{_message}",
    ],
    aliases: {
        "common_prefix": "%{_timestamp} %{_loglevel}",
        "_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
        "_loglevel": "%{LOGLEVEL:level}",
        "_status": "%{POSINT:status}",
        "_message": "%{GREEDYDATA:message}"
    }
)
Return
{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}
Learn more about the parse_groks function

Parse using aliases from file

Source
parse_groks!(
  "username=foo",
  patterns: [ "%{PATTERN_A}" ],
  alias_sources: [ "tests/data/grok/aliases.json" ]
)
# aliases.json contents:
# {
#   "PATTERN_A": "%{PATTERN_B}",
#   "PATTERN_B": "username=%{USERNAME:username}"
# }
Return
{
  "username": "foo"
}
Learn more about the parse_groks function

parse_influxdb

Parses the value as an InfluxDB line protocol string, producing a list of Vector-compatible metrics.

Parse InfluxDB line protocol

Source
parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")
Return
[{"gauge":{"value":64},"kind":"absolute","name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":10},"kind":"absolute","name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":50.5},"kind":"absolute","name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":1},"kind":"absolute","name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":0},"kind":"absolute","name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"}]
Learn more about the parse_influxdb function

parse_int

Parses the string value representing a number in an optional base/radix to an integer.

Parse decimal

Source
parse_int!("-42")
Return
-42
Learn more about the parse_int function

Parse binary

Source
parse_int!("0b1001")
Return
9
Learn more about the parse_int function

Parse octal

Source
parse_int!("0o42")
Return
34
Learn more about the parse_int function

Parse hexadecimal

Source
parse_int!("0x2a")
Return
42
Learn more about the parse_int function

Parse explicit base

Source
parse_int!("2a", 17)
Return
44
Learn more about the parse_int function

parse_json

Parses the provided value as JSON.

Only JSON types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

Parse JSON

Source
parse_json!(s'{"key": "val"}')
Return
{
  "key": "val"
}
Learn more about the parse_json function

Parse JSON array

Source
parse_json!("[true, 0]")
Return
[true,0]
Learn more about the parse_json function

Parse JSON string

Source
parse_json!(s'"hello"')
Return
hello
Learn more about the parse_json function

Parse JSON integer

Source
parse_json!("42")
Return
42
Learn more about the parse_json function

Parse JSON float

Source
parse_json!("42.13")
Return
42.13
Learn more about the parse_json function

Parse JSON boolean

Source
parse_json!("false")
Learn more about the parse_json function

Invalid JSON value

Source
parse_json!("{ INVALID }")
Learn more about the parse_json function

Parse JSON with max_depth

Source
parse_json!(s'{"first_level":{"second_level":"finish"}}', max_depth: 1)
Return
{
  "first_level": "{\"second_level\":\"finish\"}"
}
Learn more about the parse_json function

parse_key_value

Parses the value in key-value format. Also known as logfmt.

  • Keys and values can be wrapped with ".
  • " characters can be escaped using \.

Parse simple key value pairs

Source
parse_key_value!("zork=zook zonk=nork")
Return
{
  "zonk": "nork",
  "zork": "zook"
}
Learn more about the parse_key_value function

Parse logfmt log

Source
parse_key_value!(
    "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}
Learn more about the parse_key_value function

Parse comma delimited log

Source
parse_key_value!(
    "path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
    field_delimiter: ",",
    key_value_delimiter: ":"
)
Return
{
  "bytes": "632",
  "connect": "0ms",
  "dyno": "web.1",
  "fwd": "102.30.171.16",
  "host": "store.app.com",
  "path": "/cart_link",
  "protocol": "https",
  "service": "87ms",
  "status": "304"
}
Learn more about the parse_key_value function

Parse comma delimited log with standalone keys

Source
parse_key_value!(
    "env:prod,service:backend,region:eu-east1,beta",
    field_delimiter: ",",
    key_value_delimiter: ":",
)
Return
{
  "beta": true,
  "env": "prod",
  "region": "eu-east1",
  "service": "backend"
}
Learn more about the parse_key_value function

Parse duplicate keys

Source
parse_key_value!(
    "at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
    field_delimiter: ",",
    key_value_delimiter: "=",
)
Return
{
  "at": "info",
  "method": "GET",
  "path": "/index",
  "status": "200",
  "tags": [
    "dev",
    "dummy"
  ]
}
Learn more about the parse_key_value function

Parse with strict whitespace

Source
parse_key_value!(s'app=my-app ip=1.2.3.4 user= msg=hello-world', whitespace: "strict")
Return
{
  "app": "my-app",
  "ip": "1.2.3.4",
  "msg": "hello-world",
  "user": ""
}
Learn more about the parse_key_value function

parse_klog

Parses the value using the klog format used by Kubernetes components.

Parse using klog

Source
parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")
Return
{
  "file": "klog.go",
  "id": 28133,
  "level": "info",
  "line": 70,
  "message": "hello from klog",
  "timestamp": "2026-05-05T17:59:40.692994Z"
}
Learn more about the parse_klog function

parse_linux_authorization

Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

Parse Linux authorization event

Source
parse_linux_authorization!(
    s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return
{
  "appname": "sshd",
  "hostname": "localhost",
  "message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
  "procid": 1111,
  "timestamp": "2026-03-23T01:49:58Z"
}
Learn more about the parse_linux_authorization function

parse_logfmt

Parses the value in logfmt.

  • Keys and values can be wrapped using the " character.
  • " characters can be escaped by the \ character.
  • As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

Parse simple logfmt log

Source
parse_logfmt!("zork=zook zonk=nork")
Return
{
  "zonk": "nork",
  "zork": "zook"
}
Learn more about the parse_logfmt function

Parse logfmt log

Source
parse_logfmt!(
    "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}
Learn more about the parse_logfmt function

Parse logfmt log with standalone key

Source
parse_logfmt!("zork=zook plonk zonk=nork")
Return
{
  "plonk": true,
  "zonk": "nork",
  "zork": "zook"
}
Learn more about the parse_logfmt function

parse_nginx_log

Parses Nginx access and error log lines. Lines can be in combined, ingress_upstreaminfo, main or error format.

Parse via Nginx log format (combined)

Source
parse_nginx_log!(
    s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)
Return
{
  "agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
  "client": "172.17.0.1",
  "compression": "2.75",
  "referer": "http://localhost/somewhere",
  "request": "POST /not-found HTTP/1.1",
  "size": 153,
  "status": 404,
  "timestamp": "2021-04-01T12:02:31Z",
  "user": "alice"
}
Learn more about the parse_nginx_log function

Parse via Nginx log format (error)

Source
parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)
Return
{
  "cid": 1,
  "client": "172.17.0.1",
  "host": "localhost:8081",
  "message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
  "pid": 31,
  "request": "POST /not-found HTTP/1.1",
  "server": "localhost",
  "severity": "error",
  "tid": 31,
  "timestamp": "2021-04-01T13:02:31Z"
}
Learn more about the parse_nginx_log function

Parse via Nginx log format (ingress_upstreaminfo)

Source
parse_nginx_log!(
    s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
    "ingress_upstreaminfo"
)
Return
{
  "body_bytes_size": 12312,
  "http_referer": "https://10.0.0.1/some/referer",
  "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
  "proxy_alternative_upstream_name": "some-other-upstream-5000",
  "proxy_upstream_name": "some-upstream-service-9000",
  "remote_addr": "0.0.0.0",
  "remote_user": "bob",
  "req_id": "752178adb17130b291aefd8c386279e7",
  "request": "GET /some/path HTTP/2.0",
  "request_length": 462,
  "request_time": 0.05,
  "status": 200,
  "timestamp": "2023-03-18T15:00:00Z",
  "upstream_addr": "10.0.50.80:9000",
  "upstream_response_length": 19437,
  "upstream_response_time": 0.049,
  "upstream_status": 200
}
Learn more about the parse_nginx_log function

Parse via Nginx log format (main)

Source
parse_nginx_log!(
    s'172.24.0.3 - alice [31/Dec/2024:17:32:06 +0000] "GET / HTTP/1.1" 200 615 "https://domain.tld/path" "curl/8.11.1" "1.2.3.4, 10.10.1.1"',
    "main"
)
Return
{
  "body_bytes_size": 615,
  "http_referer": "https://domain.tld/path",
  "http_user_agent": "curl/8.11.1",
  "http_x_forwarded_for": "1.2.3.4, 10.10.1.1",
  "remote_addr": "172.24.0.3",
  "remote_user": "alice",
  "request": "GET / HTTP/1.1",
  "status": 200,
  "timestamp": "2024-12-31T17:32:06Z"
}
Learn more about the parse_nginx_log function

parse_proto

Parses the value as a protocol buffer payload.

Parse proto

Source
parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "test_protobuf.desc", "test_protobuf.v1.Person")
Return
{
  "name": "someone",
  "phones": [
    {
      "number": "123456"
    }
  ]
}
Learn more about the parse_proto function

parse_query_string

Parses the value as a query string.

Parse simple query string

Source
parse_query_string("foo=1&bar=2")
Return
{
  "bar": "2",
  "foo": "1"
}
Learn more about the parse_query_string function

Parse query string

Source
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return
{
  "bar": [
    "2",
    "3"
  ],
  "foo": "+1",
  "xyz": ""
}
Learn more about the parse_query_string function

Parse Ruby on Rails’ query string

Source
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return
{
  "foo[]": [
    "1",
    "2"
  ]
}
Learn more about the parse_query_string function

parse_regex

Parses the value using the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

Parse using Regex (with capture groups)

Source
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return
{
  "number": "first"
}
Learn more about the parse_regex function

Parse using Regex (without capture groups)

Source
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return
{
  "0": "first group",
  "1": "first"
}
Learn more about the parse_regex function

Parse using Regex with simple match

Source
parse_regex!("8.7.6.5 - zorp", r'^(?P<host>[\w\.]+) - (?P<user>[\w]+)')
Return
{
  "host": "8.7.6.5",
  "user": "zorp"
}
Learn more about the parse_regex function

Parse using Regex with all numeric groups

Source
parse_regex!("8.7.6.5 - zorp", r'^(?P<host>[\w\.]+) - (?P<user>[\w]+)', numeric_groups: true)
Return
{
  "0": "8.7.6.5 - zorp",
  "1": "8.7.6.5",
  "2": "zorp",
  "host": "8.7.6.5",
  "user": "zorp"
}
Learn more about the parse_regex function

Parse using Regex with variables

Source
variable = r'^(?P<host>[\w\.]+) - (?P<user>[\w]+)';
parse_regex!("8.7.6.5 - zorp", variable)
Return
{
  "host": "8.7.6.5",
  "user": "zorp"
}
Learn more about the parse_regex function

parse_regex_all

Parses the value using the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

Parse using Regex (all matches)

Source
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]
Learn more about the parse_regex_all function

Parse using Regex (simple match)

Source
parse_regex_all!("apples and carrots, peaches and peas", r'(?P<fruit>[\w\.]+) and (?P<veg>[\w]+)')
Return
[{"fruit":"apples","veg":"carrots"},{"fruit":"peaches","veg":"peas"}]
Learn more about the parse_regex_all function

Parse using Regex (all numeric groups)

Source
parse_regex_all!("apples and carrots, peaches and peas", r'(?P<fruit>[\w\.]+) and (?P<veg>[\w]+)', numeric_groups: true)
Return
[{"0":"apples and carrots","1":"apples","2":"carrots","fruit":"apples","veg":"carrots"},{"0":"peaches and peas","1":"peaches","2":"peas","fruit":"peaches","veg":"peas"}]
Learn more about the parse_regex_all function

Parse using Regex with variables

Source
variable = r'(?P<fruit>[\w\.]+) and (?P<veg>[\w]+)';
parse_regex_all!("apples and carrots, peaches and peas", variable)
Return
[{"fruit":"apples","veg":"carrots"},{"fruit":"peaches","veg":"peas"}]
Learn more about the parse_regex_all function

parse_ruby_hash

Parses the value as ruby hash.

Parse ruby hash

Source
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')
Return
{
  "test": "value",
  "testNum": 0.2,
  "testObj": {
    "testBool": true,
    "testNull": null
  }
}
Learn more about the parse_ruby_hash function

parse_syslog

Parses the value in Syslog format.

Parse Syslog log (5424)

Source
parse_syslog!(s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!')
Return
{
  "appname": "non",
  "exampleSDID@32473": {
    "eventID": "1011",
    "eventSource": "Application",
    "iut": "3"
  },
  "facility": "user",
  "hostname": "dynamicwireless.name",
  "message": "Try to override the THX port, maybe it will reboot the neural interface!",
  "msgid": "ID931",
  "procid": 2426,
  "severity": "notice",
  "timestamp": "2020-03-13T20:45:38.119Z",
  "version": 1
}
Learn more about the parse_syslog function

parse_timestamp

Parses the value in strptime format.

Parse timestamp

Source
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")
Return
t'2020-10-10T16:00:00Z'
Learn more about the parse_timestamp function

Parse timestamp with timezone

Source
parse_timestamp!("16/10/2019 12:00:00", format: "%d/%m/%Y %H:%M:%S", timezone: "Asia/Taipei")
Return
t'2019-10-16T04:00:00Z'
Learn more about the parse_timestamp function

parse_tokens

Parses the value in token format. A token is considered to be one of the following:

  • A word surrounded by whitespace.
  • Text delimited by double quotes: "..". Quotes can be included in the token if they are escaped by a backslash (\).
  • Text delimited by square brackets: [..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).

Parse tokens

Source
parse_tokens(s'A sentence "with \"a\" sentence inside" and [some brackets]')
Return
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]
Learn more about the parse_tokens function

parse_url

Parses the value in URL format.

Parse URL

Source
parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")
Return
{
  "fragment": "123",
  "host": "example.com",
  "password": "bar",
  "path": "/foobar",
  "port": 4343,
  "query": {
    "hello": "world"
  },
  "scheme": "ftp",
  "username": "foo"
}
Learn more about the parse_url function

Parse URL with default port

Source
parse_url!("https://example.com", default_known_ports: true)
Return
{
  "fragment": null,
  "host": "example.com",
  "password": "",
  "path": "/",
  "port": 443,
  "query": {},
  "scheme": "https",
  "username": ""
}
Learn more about the parse_url function

parse_user_agent

Parses the provided value as a user agent, which has a loosely defined format.

Parses on the basis of best effort. Returned schema depends only on the configured mode, so if the function fails to parse a field it will set it to null.

Fast mode

Source
parse_user_agent(
    "Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
Return
{
  "browser": {
    "family": "Firefox",
    "version": "1.0.1"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Linux",
    "version": null
  }
}
Learn more about the parse_user_agent function

Reliable mode

Source
parse_user_agent(
    "Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
    mode: "reliable")
Return
{
  "browser": {
    "family": "Internet Explorer",
    "version": "7.66"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Windows XP",
    "version": "NT 5.1"
  }
}
Learn more about the parse_user_agent function

Enriched mode

Source
parse_user_agent(
    "Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
    mode: "enriched"
)
Return
{
  "browser": {
    "family": "Opera Mini",
    "major": "4",
    "minor": "3",
    "patch": "24214",
    "version": "10.54"
  },
  "device": {
    "brand": "Apple",
    "category": "smartphone",
    "family": "iPhone",
    "model": "iPhone"
  },
  "os": {
    "family": "iOS",
    "major": "4",
    "minor": "2",
    "patch": "1",
    "patch_minor": null,
    "version": "4.2.1"
  }
}
Learn more about the parse_user_agent function

parse_xml

Parses the value as XML.

Parse XML

Source
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';

parse_xml!(value, text_key: "value", parse_number: false)
Return
{
  "book": {
    "@category": "CHILDREN",
    "author": "J K. Rowling",
    "title": {
      "@lang": "en",
      "value": "Harry Potter"
    },
    "year": "2005"
  }
}
Learn more about the parse_xml function

Random examples

random_bool

Returns a random boolean.

Random boolean

Source
is_boolean(random_bool())
Return
true
Learn more about the random_bool function

random_bytes

A cryptographically secure random number generator. Returns a string value containing the number of random bytes requested.

Generate random base 64 encoded bytes

Source
encode_base64(random_bytes(16))
Return
LNu0BBgUbh7XAlXbjSOomQ==
Learn more about the random_bytes function

Generate 16 random bytes

Source
length(random_bytes(16))
Return
16
Learn more about the random_bytes function

random_float

Returns a random float between [min, max).

Random float from 0.0 to 10.0, not including 10.0

Source
f = random_float(0.0, 10.0)
f >= 0 && f < 10
Return
true
Learn more about the random_float function

random_int

Returns a random integer between [min, max).

Random integer from 0 to 10, not including 10

Source
i = random_int(0, 10)
i >= 0 && i < 10
Return
true
Learn more about the random_int function

uuid_from_friendly_id

Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID.

Convert a Friendly ID to a UUID

Source
uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")
Return
7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2
Learn more about the uuid_from_friendly_id function

uuid_v4

Generates a random UUIDv4 string.

Create a UUIDv4

Source
uuid_v4()
Return
1d262f4f-199b-458d-879f-05fd0a5f0683
Learn more about the uuid_v4 function

uuid_v7

Generates a random UUIDv7 string.

Create a UUIDv7 with implicit now()

Source
uuid_v7()
Return
0135ddb4-a444-794c-a7a2-088f260104c0
Learn more about the uuid_v7 function

Create a UUIDv7 with explicit now()

Source
uuid_v7(now())
Return
0135ddb4-a444-794c-a7a2-088f260104c0
Learn more about the uuid_v7 function

Create a UUIDv7 with custom timestamp

Source
uuid_v7(t'2020-12-30T22:20:53.824727Z')
Return
0176b5bd-5d19-794c-a7a2-088f260104c0
Learn more about the uuid_v7 function

String examples

basename

Returns the filename component of the given path. This is similar to the Unix basename command. If the path ends in a directory separator, the function returns the name of the directory.

Extract basename from file path

Source
basename!("/usr/local/bin/vrl")
Return
vrl
Learn more about the basename function

Extract basename from file path with extension

Source
basename!("/home/user/file.txt")
Return
file.txt
Learn more about the basename function

Extract basename from directory path

Source
basename!("/home/user/")
Return
user
Learn more about the basename function

Root directory has no basename

Source
basename!("/")
Learn more about the basename function

camelcase

Takes the value string, and turns it into camelCase. Optionally, you can pass in the existing case of the function, or else an attempt is made to determine the case automatically.

camelCase a string without specifying original case

Source
camelcase("input-string")
Return
inputString
Learn more about the camelcase function

camelcase a snake_case string

Source
camelcase("foo_bar_baz", "snake_case")
Return
fooBarBaz
Learn more about the camelcase function

camelcase specifying the wrong original case (noop)

Source
camelcase("foo_bar_baz", "kebab-case")
Return
foo_bar_baz
Learn more about the camelcase function

community_id

Generates an ID based on the Community ID Spec.

Generate Community ID for TCP

Source
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)
Return
1:wCb3OG7yAFWelaUydu0D+125CLM=
Learn more about the community_id function

Generate Community ID for UDP

Source
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 17)
Return
1:0Mu9InQx6z4ZiCZM/7HXi2WMhOg=
Learn more about the community_id function

Generate Community ID for ICMP

Source
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 8, destination_port: 0, protocol: 1)
Return
1:crodRHL2FEsHjbv3UkRrfbs4bZ0=
Learn more about the community_id function

Generate Community ID for RSVP

Source
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", protocol: 46)
Return
1:ikv3kmf89luf73WPz1jOs49S768=
Learn more about the community_id function

contains

Determines whether the value string contains the specified substring.

String contains with default parameters (case sensitive)

Source
contains("banana", "AnA")
Learn more about the contains function

String contains (case insensitive)

Source
contains("banana", "AnA", case_sensitive: false)
Return
true
Learn more about the contains function

contains_all

Determines whether the value string contains all the specified substrings.

String contains all with default parameters (case sensitive)

Source
contains_all("The NEEDLE in the Haystack", ["NEEDLE", "Haystack"])
Return
true
Learn more about the contains_all function

String doesn’t contain all with default parameters (case sensitive)

Source
contains_all("The NEEDLE in the Haystack", ["needle", "Haystack"])
Learn more about the contains_all function

String contains all (case insensitive)

Source
contains_all("The NEEDLE in the HaYsTaCk", ["nEeDlE", "haystack"], case_sensitive: false)
Return
true
Learn more about the contains_all function

dirname

Returns the directory component of the given path. This is similar to the Unix dirname command. The directory component is the path with the final component removed.

Extract dirname from file path

Source
dirname!("/usr/local/bin/vrl")
Return
/usr/local/bin
Learn more about the dirname function

Extract dirname from file path with extension

Source
dirname!("/home/user/file.txt")
Return
/home/user
Learn more about the dirname function

Extract dirname from directory path

Source
dirname!("/home/user/")
Return
/home
Learn more about the dirname function

Root directory dirname is itself

Source
dirname!("/")
Return
/
Learn more about the dirname function

Relative files have current directory as dirname

Source
dirname!("file.txt")
Return
.
Learn more about the dirname function

downcase

Downcases the value string, where downcase is defined according to the Unicode Derived Core Property Lowercase.

Downcase a string

Source
downcase("Hello, World!")
Return
hello, world!
Learn more about the downcase function

Downcase with number

Source
downcase("FOO 2 BAR")
Return
foo 2 bar
Learn more about the downcase function

ends_with

Determines whether the value string ends with the specified substring.

String ends with (case sensitive)

Source
ends_with("The Needle In The Haystack", "The Haystack")
Return
true
Learn more about the ends_with function

String ends with (case insensitive)

Source
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)
Return
true
Learn more about the ends_with function

String ends with (case sensitive failure)

Source
ends_with("foobar", "R")
Learn more about the ends_with function

find

Determines from left to right the start position of the first found element in value that matches pattern. Returns -1 if not found.

Match text

Source
find("foobar", "bar")
Return
3
Learn more about the find function

Match text at start

Source
find("foobar", "foo")
Learn more about the find function

Match regex

Source
find("foobar", r'b.r')
Return
3
Learn more about the find function

No matches

Source
find("foobar", "baz")
Learn more about the find function

With an offset

Source
find("foobarfoobarfoo", "bar", 4)
Return
9
Learn more about the find function

join

Joins each string in the value array into a single string, with items optionally separated from one another by a separator.

Join array (no separator)

Source
join!(["bring", "us", "together"])
Return
bringustogether
Learn more about the join function

Join array (comma separator)

Source
join!(["sources", "transforms", "sinks"], separator: ", ")
Return
sources, transforms, sinks
Learn more about the join function

kebabcase

Takes the value string, and turns it into kebab-case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

kebab-case a string without specifying original case

Source
kebabcase("InputString")
Return
input-string
Learn more about the kebabcase function

kebab-case a snake_case string

Source
kebabcase("foo_bar_baz", "snake_case")
Return
foo-bar-baz
Learn more about the kebabcase function

kebab-case specifying the wrong original case (noop)

Source
kebabcase("foo_bar_baz", "PascalCase")
Return
foo_bar_baz
Learn more about the kebabcase function

match

Determines whether the value matches the pattern.

Regex match on a string

Source
match("I'm a little teapot", r'teapot')
Return
true
Learn more about the match function

String does not match the regular expression

Source
match("I'm a little teapot", r'.*balloon')
Learn more about the match function

match_any

Determines whether value matches any of the given patterns. All patterns are checked in a single pass over the target string, giving this function a potential performance advantage over the multiple calls in the match function.

Regex match on a string

Source
match_any("I'm a little teapot", [r'frying pan', r'teapot'])
Return
true
Learn more about the match_any function

No match

Source
match_any("My name is John Doe", patterns: [r'\d+', r'Jane'])
Learn more about the match_any function

parse_float

Parses the string value representing a floating point number in base 10 to a float.

Parse negative integer

Source
parse_float!("-42")
Return
-42
Learn more about the parse_float function

Parse float

Source
parse_float!("42.38")
Return
42.38
Learn more about the parse_float function

Scientific notation

Source
parse_float!("2.5e3")
Return
2500
Learn more about the parse_float function

pascalcase

Takes the value string, and turns it into PascalCase. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

PascalCase a string without specifying original case

Source
pascalcase("input-string")
Return
InputString
Learn more about the pascalcase function

PascalCase a snake_case string

Source
pascalcase("foo_bar_baz", "snake_case")
Return
FooBarBaz
Learn more about the pascalcase function

PascalCase specifying the wrong original case (only capitalizes)

Source
pascalcase("foo_bar_baz", "kebab-case")
Return
Foo_bar_baz
Learn more about the pascalcase function

redact

Redact sensitive data in value such as:

This can help achieve compliance by ensuring sensitive data does not leave your network.

Replace text using a regex

Source
redact("my id is 123456", filters: [r'\d+'])
Return
my id is [REDACTED]
Learn more about the redact function

Replace us social security numbers in any field

Source
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])
Return
{
  "name": "John Doe",
  "ssn": "[REDACTED]"
}
Learn more about the redact function

Replace with custom text

Source
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})
Return
my id is ***
Learn more about the redact function

Replace with SHA-2 hash

Source
redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")
Return
my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I=
Learn more about the redact function

Replace with SHA-3 hash

Source
redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")
Return
my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw==
Learn more about the redact function

Replace with SHA-256 hash using hex encoding

Source
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
Return
my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92
Learn more about the redact function

replace

Replaces all matching instances of pattern in value.

The pattern argument accepts regular expression capture groups.

Note when using capture groups:

  • You will need to escape the $ by using $$ to avoid Vector interpreting it as an environment variable when loading configuration
  • If you want a literal $ in the replacement pattern, you will also need to escape this with $$. When combined with environment variable interpolation in config files this means you will need to use $$$$ to have a literal $ in the replacement pattern.

Replace literal text

Source
replace("Apples and Bananas", "and", "not")
Return
Apples not Bananas
Learn more about the replace function

Replace using regular expression

Source
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")
Return
Apples and Pineapples
Learn more about the replace function

Replace first instance

Source
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)
Return
Pineapples and Bananas
Learn more about the replace function

Replace with capture groups

Source
# Note that in the context of Vector configuration files, an extra `$` escape character is required
# (i.e. `$$num`) to avoid interpreting `num` as an environment variable.
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")
Return
123
Learn more about the replace function

Replace all

Source
replace("foobar", "o", "i")
Return
fiibar
Learn more about the replace function

replace_with

Replaces all matching instances of pattern using a closure.

The pattern argument accepts a regular expression that can use capture groups.

The function uses the function closure syntax to compute the replacement values.

The closure takes a single parameter, which is an array, where the first item is always present and contains the entire string that matched pattern. The items from index one on contain the capture groups of the corresponding index. If a capture group is optional, the value may be null if it didn’t match.

The value returned by the closure must be a string and will replace the section of the input that was matched.

This returns a new string with the replacements, the original string is not mutated.

Capitalize words

Source
replace_with("apples and bananas", r'\b(\w)(\w*)') -> |match| {
    upcase!(match.captures[0]) + string!(match.captures[1])
}
Return
Apples And Bananas
Learn more about the replace_with function

Replace with hash

Source
replace_with("email from test@example.com", r'\w+@example.com') -> |match| {
    sha2(match.string, variant: "SHA-512/224")
}
Return
email from adf6e1bc4415d24912bd93072ad34ef825a7b6eb3bf53f68def1fc17
Learn more about the replace_with function

Replace first instance

Source
replace_with("Apples and Apples", r'(?i)apples|cones', count: 1) -> |match| {
    "Pine" + downcase(match.string)
}
Return
Pineapples and Apples
Learn more about the replace_with function

Named capture group

Source
replace_with("level=error A message", r'level=(?P<level>\w+)') -> |match| {
    lvl = upcase!(match.level)
    "[{{lvl}}]"
}
Return
[ERROR] A message
Learn more about the replace_with function

Replace with processed capture group

Source
replace_with(s'Got message: {"msg": "b"}', r'message: (\{.*\})') -> |m| {
    to_string!(parse_json!(m.captures[0]).msg)
}
Return
Got b
Learn more about the replace_with function

Replace with optional capture group

Source
replace_with("bar of chocolate and bar of gold", r'bar( of gold)?') -> |m| {
    if m.captures[0] == null { "pile" } else { "money" }
}
Return
pile of chocolate and money
Learn more about the replace_with function

screamingsnakecase

Takes the value string, and turns it into SCREAMING_SNAKE case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

SCREAMING_SNAKE_CASE a string without specifying original case

Source
screamingsnakecase("input-string")
Return
INPUT_STRING
Learn more about the screamingsnakecase function

SCREAMING_SNAKE_CASE a snake_case string

Source
screamingsnakecase("foo_bar_baz", "snake_case")
Return
FOO_BAR_BAZ
Learn more about the screamingsnakecase function

SCREAMING_SNAKE_CASE specifying the wrong original case (capitalizes but doesn’t include _ properly)

Source
screamingsnakecase("FooBarBaz", "kebab-case")
Return
FOOBARBAZ
Learn more about the screamingsnakecase function

shannon_entropy

Generates Shannon entropy from given string. It can generate it based on string bytes, codepoints, or graphemes.

Simple byte segmentation example

Source
floor(shannon_entropy("vector.dev"), precision: 4)
Return
2.9219
Learn more about the shannon_entropy function

UTF-8 string with bytes segmentation

Source
floor(shannon_entropy("test123%456.فوائد.net."), precision: 4)
Return
4.0784
Learn more about the shannon_entropy function

UTF-8 string with grapheme segmentation

Source
floor(shannon_entropy("test123%456.فوائد.net.", segmentation: "grapheme"), precision: 4)
Return
3.9362
Learn more about the shannon_entropy function

UTF-8 emoji (7 Unicode scalar values) with grapheme segmentation

Source
shannon_entropy("👨‍👩‍👧‍👦", segmentation: "grapheme")
Learn more about the shannon_entropy function

sieve

Keeps only matches of pattern in value.

This can be used to define patterns that are allowed in the string and remove everything else.

Keep only lowercase letters

Source
sieve("vector.dev/lowerUPPER", permitted_characters: r'[a-z]')
Return
vectordevlower
Learn more about the sieve function

Sieve with regex

Source
sieve("test123%456.فوائد.net.", r'[a-z0-9.]')
Return
test123456..net.
Learn more about the sieve function

Custom replacements

Source
sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")
Return
test123X456.<REMOVED>.net.
Learn more about the sieve function

slice

Returns a slice of value between the start and end positions.

If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array, a slice up to the end of the string or array is returned.

Slice a string (positive index)

Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)
Return
califrag
Learn more about the slice function

Slice a string (negative index)

Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)
Return
califragilistic
Learn more about the slice function

String start

Source
slice!("foobar", 3)
Return
bar
Learn more about the slice function

Array start

Source
slice!([0, 1, 2], 1)
Return
[1,2]
Learn more about the slice function

snakecase

Takes the value string, and turns it into snake_case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

snake_case a string

Source
snakecase("input-string")
Return
input_string
Learn more about the snakecase function

snake_case a string with original case

Source
snakecase("input-string", original_case: "kebab-case")
Return
input_string
Learn more about the snakecase function

snake_case with excluded boundaries

Source
snakecase("s3BucketDetails", excluded_boundaries: ["lower_digit"])
Return
s3_bucket_details
Learn more about the snakecase function

split

Splits the value string using pattern.

Split a string (no limit)

Source
split("apples and pears and bananas", " and ")
Return
["apples","pears","bananas"]
Learn more about the split function

Split a string (with a limit)

Source
split("apples and pears and bananas", " and ", limit: 2)
Return
["apples","pears and bananas"]
Learn more about the split function

Split string

Source
split("foobar", "b")
Return
["foo","ar"]
Learn more about the split function

Split regex

Source
split("barbaz", r'ba')
Return
["","r","z"]
Learn more about the split function

split_path

Splits the given path into its constituent components, returning an array of strings. Each component represents a part of the file system path hierarchy.

Split path with trailing slash

Source
split_path("/home/user/")
Return
["/","home","user"]
Learn more about the split_path function

Split path from file path

Source
split_path("/home/user")
Return
["/","home","user"]
Learn more about the split_path function

Split path from root

Source
split_path("/")
Return
["/"]
Learn more about the split_path function

Empty path returns empty array

Source
split_path("")
Learn more about the split_path function

starts_with

Determines whether value begins with substring.

String starts with (case sensitive)

Source
starts_with("The Needle In The Haystack", "The Needle")
Return
true
Learn more about the starts_with function

String starts with (case insensitive)

Source
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)
Return
true
Learn more about the starts_with function

String starts with (case sensitive failure)

Source
starts_with("foobar", "F")
Learn more about the starts_with function

strip_ansi_escape_codes

Strips ANSI escape codes from value.

strip_whitespace

Strips whitespace from the start and end of value, where whitespace is defined by the Unicode White_Space property.

Strip whitespace

Source
strip_whitespace("  A sentence.  ")
Return
A sentence.
Learn more about the strip_whitespace function

Start whitespace

Source
strip_whitespace("  foobar")
Return
foobar
Learn more about the strip_whitespace function

End whitespace

Source
strip_whitespace("foo bar  ")
Return
foo bar
Learn more about the strip_whitespace function

Newlines

Source
strip_whitespace("\n\nfoo bar\n  ")
Return
foo bar
Learn more about the strip_whitespace function

truncate

Truncates the value string up to the limit number of characters.

Truncate a string

Source
truncate("A rather long sentence.", limit: 11, suffix: "...")
Return
A rather lo...
Learn more about the truncate function

Truncate a string (custom suffix)

Source
truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")
Return
A rather lo[TRUNCATED]
Learn more about the truncate function

Truncate

Source
truncate("foobar", 3)
Return
foo
Learn more about the truncate function

upcase

Upcases value, where upcase is defined according to the Unicode Derived Core Property Uppercase.

Upcase a string

Source
upcase("Hello, World!")
Return
HELLO, WORLD!
Learn more about the upcase function

System examples

dns_lookup

Performs a DNS lookup on the provided domain name.

Basic lookup

Source
res = dns_lookup!("dns.google")
# reset non-static ttl so result is static
res.answers = map_values(res.answers) -> |value| {
  value.ttl = 600
  value
}
# remove extra responses for example
res.answers = filter(res.answers) -> |_, value| {
    value.rData == "8.8.8.8"
}
# remove class since this is also dynamic
res.additional = map_values(res.additional) -> |value| {
    del(value.class)
    value
}
res
Return
{
  "additional": [
    {
      "domainName": "",
      "rData": "OPT ...",
      "recordType": "OPT",
      "recordTypeId": 41,
      "ttl": 0
    }
  ],
  "answers": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "rData": "8.8.8.8",
      "recordType": "A",
      "recordTypeId": 1,
      "ttl": 600
    }
  ],
  "authority": [],
  "fullRcode": 0,
  "header": {
    "aa": false,
    "ad": false,
    "anCount": 2,
    "arCount": 1,
    "cd": false,
    "nsCount": 0,
    "opcode": 0,
    "qdCount": 1,
    "qr": true,
    "ra": true,
    "rcode": 0,
    "rd": true,
    "tc": false
  },
  "question": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "questionType": "A",
      "questionTypeId": 1
    }
  ],
  "rcodeName": "NOERROR"
}
Learn more about the dns_lookup function

Custom class and qtype

Source
res = dns_lookup!("dns.google", class: "IN", qtype: "A")
# reset non-static ttl so result is static
res.answers = map_values(res.answers) -> |value| {
  value.ttl = 600
  value
}
# remove extra responses for example
res.answers = filter(res.answers) -> |_, value| {
    value.rData == "8.8.8.8"
}
# remove class since this is also dynamic
res.additional = map_values(res.additional) -> |value| {
    del(value.class)
    value
}
res
Return
{
  "additional": [
    {
      "domainName": "",
      "rData": "OPT ...",
      "recordType": "OPT",
      "recordTypeId": 41,
      "ttl": 0
    }
  ],
  "answers": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "rData": "8.8.8.8",
      "recordType": "A",
      "recordTypeId": 1,
      "ttl": 600
    }
  ],
  "authority": [],
  "fullRcode": 0,
  "header": {
    "aa": false,
    "ad": false,
    "anCount": 2,
    "arCount": 1,
    "cd": false,
    "nsCount": 0,
    "opcode": 0,
    "qdCount": 1,
    "qr": true,
    "ra": true,
    "rcode": 0,
    "rd": true,
    "tc": false
  },
  "question": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "questionType": "A",
      "questionTypeId": 1
    }
  ],
  "rcodeName": "NOERROR"
}
Learn more about the dns_lookup function

Custom options

Source
res = dns_lookup!("dns.google", options: {"timeout": 30, "attempts": 5})
res.answers = map_values(res.answers) -> |value| {
  value.ttl = 600
  value
}
# remove extra responses for example
res.answers = filter(res.answers) -> |_, value| {
    value.rData == "8.8.8.8"
}
# remove class since this is also dynamic
res.additional = map_values(res.additional) -> |value| {
    del(value.class)
    value
}
res
Return
{
  "additional": [
    {
      "domainName": "",
      "rData": "OPT ...",
      "recordType": "OPT",
      "recordTypeId": 41,
      "ttl": 0
    }
  ],
  "answers": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "rData": "8.8.8.8",
      "recordType": "A",
      "recordTypeId": 1,
      "ttl": 600
    }
  ],
  "authority": [],
  "fullRcode": 0,
  "header": {
    "aa": false,
    "ad": false,
    "anCount": 2,
    "arCount": 1,
    "cd": false,
    "nsCount": 0,
    "opcode": 0,
    "qdCount": 1,
    "qr": true,
    "ra": true,
    "rcode": 0,
    "rd": true,
    "tc": false
  },
  "question": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "questionType": "A",
      "questionTypeId": 1
    }
  ],
  "rcodeName": "NOERROR"
}
Learn more about the dns_lookup function

Custom server

Source
res = dns_lookup!("dns.google", options: {"servers": ["dns.quad9.net"]})
res.answers = map_values(res.answers) -> |value| {
  value.ttl = 600
  value
}
# remove extra responses for example
res.answers = filter(res.answers) -> |_, value| {
    value.rData == "8.8.8.8"
}
# remove class since this is also dynamic
res.additional = map_values(res.additional) -> |value| {
    del(value.class)
    value
}
res
Return
{
  "additional": [
    {
      "domainName": "",
      "rData": "OPT ...",
      "recordType": "OPT",
      "recordTypeId": 41,
      "ttl": 0
    }
  ],
  "answers": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "rData": "8.8.8.8",
      "recordType": "A",
      "recordTypeId": 1,
      "ttl": 600
    }
  ],
  "authority": [],
  "fullRcode": 0,
  "header": {
    "aa": false,
    "ad": false,
    "anCount": 2,
    "arCount": 1,
    "cd": false,
    "nsCount": 0,
    "opcode": 0,
    "qdCount": 1,
    "qr": true,
    "ra": true,
    "rcode": 0,
    "rd": true,
    "tc": false
  },
  "question": [
    {
      "class": "IN",
      "domainName": "dns.google",
      "questionType": "A",
      "questionTypeId": 1
    }
  ],
  "rcodeName": "NOERROR"
}
Learn more about the dns_lookup function

get_env_var

Returns the value of the environment variable specified by name.

Get an environment variable

Source
get_env_var!("HOME")
Return
/root
Learn more about the get_env_var function

get_hostname

Returns the local system’s hostname.

Get hostname

Source
get_hostname!()
Return
my-hostname
Learn more about the get_hostname function

get_timezone_name

Returns the name of the timezone in the Vector configuration (see global configuration options). If the configuration is set to local, then it attempts to determine the name of the timezone from the host OS. If this is not possible, then it returns the fixed offset of the local timezone for the current time in the format "[+-]HH:MM", for example, "+02:00".

Get the IANA name of Vector’s timezone

Source
get_timezone_name!()
Return
UTC
Learn more about the get_timezone_name function

http_request

Makes an HTTP request to the specified URL.

Basic HTTP request

Source
http_request("https://httpbin.org/get")
Return
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org"
  },
  "url": "https://httpbin.org/get"
}
Learn more about the http_request function

HTTP request with bearer token

Source
http_request("https://httpbin.org/bearer", headers: {"Authorization": "Bearer my_token"})
Return
{
  "authenticated": true,
  "token": "my_token"
}
Learn more about the http_request function

HTTP PUT request

Source
http_request("https://httpbin.org/put", method: "put")
Return
{
  "args": {},
  "data": "",
  "url": "https://httpbin.org/put"
}
Learn more about the http_request function

HTTP POST request with body

Source
http_request("https://httpbin.org/post", method: "post", body: "{\"data\":{\"hello\":\"world\"}}")
Return
{
  "data": "{\"data\":{\"hello\":\"world\"}}"
}
Learn more about the http_request function

reverse_dns

Performs a reverse DNS lookup on the provided IP address to retrieve the associated hostname.

Example

Source
reverse_dns!("127.0.0.1")
Return
localhost
Learn more about the reverse_dns function

Timestamp examples

format_timestamp

Formats value into a string representation of the timestamp.

Format a timestamp (ISO8601/RFC 3339)

Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")
Return
2020-10-21T16:00:00+00:00
Learn more about the format_timestamp function

Format a timestamp (custom)

Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")
Return
21-Oct-2020 16:00
Learn more about the format_timestamp function

Format a timestamp with custom format string

Source
format_timestamp!(t'2021-02-10T23:32:00+00:00', format: "%d %B %Y %H:%M")
Return
10 February 2021 23:32
Learn more about the format_timestamp function

Format a timestamp with timezone conversion

Source
format_timestamp!(t'2021-02-10T23:32:00+00:00', format: "%d %B %Y %H:%M", timezone: "Europe/Berlin")
Return
11 February 2021 00:32
Learn more about the format_timestamp function

now

Returns the current timestamp in the UTC timezone with nanosecond precision.

Generate a current timestamp

Source
now()
Return
2012-03-04T12:34:56.789012345Z
Learn more about the now function

Type examples

array

Returns value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects an array.

Declare an array type

Source
array!(.value)
Return
[1,2,3]
Learn more about the array function

Valid array literal

Source
array([1,2,3])
Return
[1,2,3]
Learn more about the array function

Invalid type

Source
array!(true)
Learn more about the array function

bool

Returns value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects a Boolean.

Valid Boolean

Source
bool(false)
Learn more about the bool function

Invalid Boolean

Source
bool!(42)
Learn more about the bool function

Valid Boolean from path

Source
bool!(.value)
Return
true
Learn more about the bool function

float

Returns value if it is a float, otherwise returns an error. This enables the type checker to guarantee that the returned value is a float and can be used in any function that expects a float.

Declare a float type

Source
. = { "value": 42.0 }
float(.value)
Return
42
Learn more about the float function

Declare a float type (literal)

Source
float(3.1415)
Return
3.1415
Learn more about the float function

Invalid float type

Source
float!(true)
Learn more about the float function

int

Returns value if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the returned value is an integer and can be used in any function that expects an integer.

Declare an integer type

Source
. = { "value": 42 }
int(.value)
Return
42
Learn more about the int function

Declare an integer type (literal)

Source
int(42)
Return
42
Learn more about the int function

Invalid integer type

Source
int!(true)
Learn more about the int function

is_array

Check if the value’s type is an array.

Valid array

Source
is_array([1, 2, 3])
Return
true
Learn more about the is_array function

Non-matching type

Source
is_array("a string")
Learn more about the is_array function

Boolean

Source
is_array(true)
Learn more about the is_array function

Null

Source
is_array(null)
Learn more about the is_array function

is_boolean

Check if the value’s type is a boolean.

Valid boolean

Source
is_boolean(false)
Return
true
Learn more about the is_boolean function

Non-matching type

Source
is_boolean("a string")
Learn more about the is_boolean function

Null

Source
is_boolean(null)
Learn more about the is_boolean function

is_empty

Check if the object, array, or string has a length of 0.

Empty array

Source
is_empty([])
Return
true
Learn more about the is_empty function

Non-empty string

Source
is_empty("a string")
Learn more about the is_empty function

Non-empty object

Source
is_empty({"foo": "bar"})
Learn more about the is_empty function

Empty string

Source
is_empty("")
Return
true
Learn more about the is_empty function

Empty object

Source
is_empty({})
Return
true
Learn more about the is_empty function

Non-empty array

Source
is_empty([1,2,3])
Learn more about the is_empty function

is_float

Check if the value’s type is a float.

Valid float

Source
is_float(0.577)
Return
true
Learn more about the is_float function

Non-matching type

Source
is_float("a string")
Learn more about the is_float function

Boolean

Source
is_float(true)
Learn more about the is_float function

Null

Source
is_float(null)
Learn more about the is_float function

is_integer

Check if the value’s type is an integer.

Valid integer

Source
is_integer(1)
Return
true
Learn more about the is_integer function

Non-matching type

Source
is_integer("a string")
Learn more about the is_integer function

Null

Source
is_integer(null)
Learn more about the is_integer function

is_json

Check if the string is a valid JSON document.

Valid JSON object

Source
is_json("{}")
Return
true
Learn more about the is_json function

Non-valid value

Source
is_json("{")
Learn more about the is_json function

Exact variant

Source
is_json("{}", variant: "object")
Return
true
Learn more about the is_json function

Non-valid exact variant

Source
is_json("{}", variant: "array")
Learn more about the is_json function

Valid JSON string

Source
is_json(s'"test"')
Return
true
Learn more about the is_json function

is_null

Check if value’s type is null. For a more relaxed function, see is_nullish.

Null value

Source
is_null(null)
Return
true
Learn more about the is_null function

Non-matching type

Source
is_null("a string")
Learn more about the is_null function

Array

Source
is_null([1, 2, 3])
Learn more about the is_null function

is_nullish

Determines whether value is nullish. Returns true if the specified value is null, an empty string, a string containing only whitespace, or the string "-". Returns false otherwise.

Null detection (blank string)

Source
is_nullish("")
Return
true
Learn more about the is_nullish function

Null detection (dash string)

Source
is_nullish("-")
Return
true
Learn more about the is_nullish function

Null detection (whitespace)

Source
is_nullish("
  
")
Return
true
Learn more about the is_nullish function

Null

Source
is_nullish(null)
Return
true
Learn more about the is_nullish function

is_object

Check if value’s type is an object.

Valid object

Source
is_object({"foo": "bar"})
Return
true
Learn more about the is_object function

Non-matching type

Source
is_object("a string")
Learn more about the is_object function

Boolean

Source
is_object(true)
Learn more about the is_object function

is_regex

Check if value’s type is a regex.

Valid regex

Source
is_regex(r'pattern')
Return
true
Learn more about the is_regex function

Non-matching type

Source
is_regex("a string")
Learn more about the is_regex function

Null value

Source
is_regex(null)
Learn more about the is_regex function

is_string

Check if value’s type is a string.

Valid string

Source
is_string("a string")
Return
true
Learn more about the is_string function

Non-matching type

Source
is_string([1, 2, 3])
Learn more about the is_string function

Boolean

Source
is_string(true)
Learn more about the is_string function

Null

Source
is_string(null)
Learn more about the is_string function

is_timestamp

Check if value’s type is a timestamp.

Valid timestamp

Source
is_timestamp(t'2021-03-26T16:00:00Z')
Return
true
Learn more about the is_timestamp function

Non-matching type

Source
is_timestamp("a string")
Learn more about the is_timestamp function

Boolean value

Source
is_timestamp(true)
Learn more about the is_timestamp function

object

Returns value if it is an object, otherwise returns an error. This enables the type checker to guarantee that the returned value is an object and can be used in any function that expects an object.

Declare an object type

Source
. = { "value": { "field1": "value1", "field2": "value2" } }
object(.value)
Return
{
  "field1": "value1",
  "field2": "value2"
}
Learn more about the object function

Invalid type

Source
object!(true)
Learn more about the object function

string

Returns value if it is a string, otherwise returns an error. This enables the type checker to guarantee that the returned value is a string and can be used in any function that expects a string.

Declare a string type

Source
. = { "message": "{\"field\": \"value\"}" }
string(.message)
Return
{"field": "value"}
Learn more about the string function

Invalid type

Source
string!(true)
Learn more about the string function

tag_types_externally

Adds type information to all (nested) scalar values in the provided value.

The type information is added externally, meaning that value has the form of "type": value after this transformation.

Tag types externally (scalar)

Source
tag_types_externally(123)
Return
{
  "integer": 123
}
Learn more about the tag_types_externally function

Tag types externally (object)

Source
tag_types_externally({
    "message": "Hello world",
    "request": {
        "duration_ms": 67.9
    }
})
Return
{
  "message": {
    "string": "Hello world"
  },
  "request": {
    "duration_ms": {
      "float": 67.9
    }
  }
}
Learn more about the tag_types_externally function

Tag types externally (array)

Source
tag_types_externally(["foo", "bar"])
Return
[{"string":"foo"},{"string":"bar"}]
Learn more about the tag_types_externally function

Tag types externally (null)

Source
tag_types_externally(null)
Learn more about the tag_types_externally function

timestamp

Returns value if it is a timestamp, otherwise returns an error. This enables the type checker to guarantee that the returned value is a timestamp and can be used in any function that expects a timestamp.

Declare a timestamp type

Source
timestamp(t'2020-10-10T16:00:00Z')
Return
t'2020-10-10T16:00:00Z'
Learn more about the timestamp function

Invalid type

Source
timestamp!(true)
Learn more about the timestamp function

type_def

Returns the type definition of an expression at runtime.

This is a debug function that is UNSTABLE. Behavior is NOT guaranteed even though it is technically usable.

return type definition

Source
type_def(42)
Return
{
  "integer": true
}
Learn more about the type_def function

validate_json_schema

Check if value conforms to a JSON Schema definition. This function validates a JSON payload against a JSON Schema definition. It can be used to ensure that the data structure and types in value match the expectations defined in schema_definition.

Payload contains a valid email

Source
validate_json_schema!(s'{ "productUser": "valid@email.com" }', "schema_with_email_format.json", false)
Return
true
Learn more about the validate_json_schema function

Payload contains an invalid email

Source
validate_json_schema!(s'{ "productUser": "invalidEmail" }', "schema_with_email_format.json", false)
Learn more about the validate_json_schema function

Payload contains a custom format declaration

Source
validate_json_schema!(s'{ "productUser": "a-custom-formatted-string" }', "schema_with_custom_format.json", false)
Learn more about the validate_json_schema function

Payload contains a custom format declaration, with ignore_unknown_formats set to true

Source
validate_json_schema!(s'{ "productUser": "a-custom-formatted-string" }', "schema_with_custom_format.json", true)
Return
true
Learn more about the validate_json_schema function

Checksum examples

crc

Calculates a CRC of the value.The CRC algorithm used can be optionally specified.

This function is infallible if either the default algorithm value or a recognized-valid compile-time algorithm string literal is used. Otherwise, it is fallible.

Create CRC checksum using the default algorithm

Source
crc("foo")
Return
2356372769
Learn more about the crc function

Create CRC checksum using the CRC_32_CKSUM algorithm

Source
crc("foo", algorithm: "CRC_32_CKSUM")
Return
4271552933
Learn more about the crc function

xxhash

Calculates a xxHash hash of the value.

Calculate a hash using the default (XXH32) algorithm

Source
xxhash("foo")
Return
3792637401
Learn more about the xxhash function

Calculate a hash using the XXH32 algorithm

Source
xxhash("foo", "XXH32")
Return
3792637401
Learn more about the xxhash function

Calculate a hash using the XXH64 algorithm

Source
xxhash("foo", "XXH64")
Return
3728699739546630700
Learn more about the xxhash function

Calculate a hash using the XXH3-64 algorithm

Source
xxhash("foo", "XXH3-64")
Return
-6093828362558604000
Learn more about the xxhash function

Calculate a hash using the XXH3-128 algorithm

Source
xxhash("foo", "XXH3-128")
Return
161745101148472925293886522910304009610
Learn more about the xxhash function
Sidebar