Logo

A powerful, easily deployable network traffic analysis tool suite for network security monitoring

Quick Start

Documentation

Components

Supported Protocols

Configuring

Arkime

Dashboards

Hedgehog Linux

Contribution Guide

Adding a new Zeek parser

This page walks through integrating a Zeek package that writes new logs into Malcolm, from installing the package to building its dashboard. The steps are written in the order they’re usually done, and the checklist at the end lists every file involved.

The examples use a made-up package, icsnpp-foo, that writes foo.log, foo_detail.log and foo_errors.log. Replace foo with your parser’s name throughout.

Overview

A Zeek log goes through several stages before it shows up in Malcolm, and a new parser touches each of them:

Stage What you add Where
Zeek the package, its enable/disable and port settings zeek/zeek-packages.list, zeek/zeek-packages.lock, zeek/config/local.zeek, config/zeek.env.example
Logstash parsing turning each log line into fields logstash/pipelines/zeek/10xx_zeek_foo.conf
Logstash normalization shared fields, ECS fields, value cleanup, timestamps, document IDs logstash/pipelines/zeek/1200_zeek_mutate.conf, 1300_zeek_normalize.conf, 1400_zeek_convert.conf, 2000_fingerprint.conf, logstash/maps/*.yaml, logstash/pipelines/enrichment/20_enriched_to_ecs.conf
Field definitions field types and how Arkime shows them dashboards/templates/composable/component/zeek.json or zeek_ot.json, arkime/etc/config.ini, arkime/wise/source.zeeklogs.js
Documentation the protocol and its settings docs/protocols.md, docs/malcolm-config.md
Tests capture files and expected datasets Malcolm-Test-Artifacts, Malcolm-Test
Dashboards a dashboard and its navigation link dashboards/dashboards/, dashboards/dashboards/navigation.md

Most of the parsing and field definitions can be generated from the package’s Zeek scripts with scripts/zeek_script_to_malcolm_boilerplate.py. The rest of the work is deciding how the parser’s fields fit into Malcolm’s data model, and checking the result against real traffic.

1. Before you start

Check that the package is a good fit. Malcolm pins and builds every third-party package it ships, so review the package against the standards of trust for upstream code before investing time in it.

Read the package’s Zeek scripts. You’ll need to know:

Find captures to test with. Many packages ship their own test traces (often under testing/traces/), along with baseline outputs that show what the logs should contain. Real-world captures are better still if you can find them. To check whether a capture actually contains the protocol, tshark helps; for example, for a protocol carried over MMS:

tshark -r capture.pcap -Y 'mms' | wc -l
tshark -r capture.pcap -q -z conv,tcp      # which connections and ports
tshark -r capture.pcap -q -z io,phs        # which protocols Wireshark recognizes

If the traffic is on a port Wireshark doesn’t associate with the protocol, tell it with -d (e.g., -d tcp.port==10102,tpkt).

Run the package in Zeek by itself first. Before touching Malcolm, run Zeek with the package loaded against your captures and look at the logs it writes. This is the reference you’ll compare Malcolm’s results against in Test and verify. Keep these logs.

2. Install the package

Zeek packages are installed when the zeek image is built, from two files:

https://github.com/cisagov/icsnpp-foo|main

After editing the list, generate the lock entry and review the diff:

GITHUB_TOKEN=... ./scripts/update_pins.py --zeek-packages --apply
git diff zeek/zeek-packages.lock

See Version Pinning and Updating a Pin for more on how pins work. The install itself is done by zeek/scripts/zeek_install_plugins.sh; if a package needs extra zkg arguments (such as --user-var), add them in its zkg_install_pinned function. Then rebuild the zeek image.

Packages are loaded with @load packages in local.zeek, so an installed package is active without any further changes. Packages installed with clone: aren’t zkg packages, so they need their own @load.

3. Configure the analyzer

Users control analyzers through environment variables in config/zeek.env, which local.zeek reads when Zeek starts. Follow the naming already in use:

Variable Use
ZEEK_DISABLE_ICS_FOO disables an ICS/OT analyzer (these also honor ZEEK_DISABLE_ICS_ALL)
ZEEK_DISABLE_SPICY_FOO disables a non-ICS Spicy analyzer
ZEEK_PORTS_FOO overrides the analyzer’s ports, as a comma-separated list like 1234/tcp,1235/udp
ZEEK_FOO_<OPTION> any other setting specific to the parser (e.g., ZEEK_FOO_DETAILED)

Disabling the analyzer

In local.zeek, read the variable near the other global disable_... lines:

global disable_ics_foo = (getenv("ZEEK_DISABLE_ICS_FOO") == true_regex) ? T : F;

Then disable the analyzer in the zeek_init event alongside the others. The call depends on the analyzer type:

  # Spicy analyzer: the tag comes from the .evt file's "protocol analyzer FOO_TCP over TCP"
  #   (a name declared as spicy::FOO_TCP becomes ANALYZER_SPICY_FOO_TCP)
  if (disable_ics_all || disable_ics_foo) {
    Spicy::disable_protocol_analyzer(Analyzer::ANALYZER_FOO_TCP);
  }

  # compiled plugin
  if (disable_ics_all || disable_ics_foo) {
    Analyzer::disable_analyzer(Analyzer::ANALYZER_FOO);
  }

When you test the parser, check that setting the variable to true really does stop its logs.

Configurable ports

If the package exposes a &redef port set, read the variable with the other ..._ports_str globals and redefine the set at parse time, following the existing examples. split_ports_by_protocol separates TCP from UDP ports:

global foo_ports_str = getenv("ZEEK_PORTS_FOO");

@if ((!disable_ics_all) && (!disable_ics_foo) && (foo_ports_str != ""))
  global split_foo_ports = split_ports_by_protocol(foo_ports_str);
  @if (|split_foo_ports$tcp| > 0)
    redef FOO::foo_ports = split_foo_ports$tcp;
  @endif
@endif

Environment files and documentation

Add the variables to config/zeek.env.example next to the related ones. Leave the value empty for analyzers that are on by default, or set a disable variable to true for analyzers that should be off by default. Describe each new variable in docs/malcolm-config.md.

These variables aren’t exposed in Malcolm’s configuration UI; users set them in the .env file. For Kubernetes deployments, the Malcolm Helm chart has its own copy of them: once the parser is in a Malcolm release, add the variables to the chart’s values.yaml (under zeek_env; ports go under zeek_env.zeek_ports) and templates/malcolm_configmaps.yaml, and submit them as a pull request to that repository.

Script optimization (ZAM)

Malcolm can run Zeek with its ZAM script optimizer (ZEEK_ZAM_ENABLED). ZAM compiles every loaded script when Zeek starts, and a large package can make that compile very slow. scripts/zeek-zam-bench.sh installs each package from the lock file in a fresh container and times Zeek startup with and without ZAM:

./scripts/zeek-zam-bench.sh -p capture.pcap -l zeek/zeek-packages.lock -f 'icsnpp-foo'

If startup with ZAM takes far longer for the new package than for the others, or times out, note it in the documentation and report it upstream.

4. Generate the boilerplate

scripts/zeek_script_to_malcolm_boilerplate.py reads the package’s .zeek scripts and generates a starting point for the Logstash parse filters, Arkime field definitions and views, OpenSearch template mappings and WISE field list. It needs the zeekscript and python-slugify Python packages, and malcolm_utils.py from the same directory:

python3 -m pip install zeekscript python-slugify
./scripts/zeek_script_to_malcolm_boilerplate.py \
  -i ~/src/icsnpp-foo/scripts/*.zeek \
  -p tcp \
  -s foo \
  -u https://github.com/cisagov/icsnpp-foo \
  -g zeek_foo \
  -l foo.logstash.conf \
  -a foo.arkime.ini \
  -x foo.template.json \
  -w foo.wise.txt

Review the output before using it:

The generated files cover the parse filters and the first version of the field definitions. Everything else in this guide is done by hand.

5. Plan the normalization

Before writing more configuration, decide how the parser’s fields fit into Malcolm’s data model. It’s easier to settle this once than to rework it across seven files later.

Fields shared across logs

Parsers that write several logs often repeat the same fields across them: a request ID, an object name, a connection phase, an error code. If those are moved to one shared location, analysts can filter on one field and see every log where the value appears. Malcolm puts them under the namespace of the parser’s main log. For example, a field that appears in foo.log and foo_detail.log is stored as zeek.foo.request_id for both, not as zeek.foo.request_id and zeek.foo_detail.request_id.

To find candidates, list every &log field in every record and look for names that appear in more than one. Then check each one:

Keep a short list of every rename and merge you decide on. It drives the Logstash renames, the template, the Arkime definitions and the WISE list.

ECS fields

Malcolm fills in a few ECS fields for every protocol, and dashboards such as Actions and Results depend on them:

Read the parser’s code to find which fields carry these and what values they can take. Pay attention to fields that mix kinds of values: a field that holds the operation name plus a stage (e.g., write_request, write_response) should feed only its base action into event.action, and lifecycle markers like requested may not belong in event.result at all. Make a first decision now and revisit it once you can see real data.

Protocol labeling

network.protocol comes from the log’s service. If the parser handles more than one protocol (for example, a parser that also recognizes a protocol it’s layered on), decide whether some rows should be labeled with the other protocol, and which field tells you.

6. Logstash

Zeek logs go through the Zeek parse pipeline in logstash/pipelines/zeek/, whose files run in name order. See Logstash for an overview of each stage.

Parse filters: 10xx_zeek_foo.conf

Create a file for the parser, named so it sorts after 1001_zeek_parse.conf and before 1199_zeek_unknown.conf (e.g., 1070_zeek_foo.conf), and put the generated Logstash boilerplate in it. Keep this file to parsing: one block per log that splits the line into [zeek_cols] fields and adds proto and service. Normalization belongs in the later stages.

The common Zeek fields (ts, uid, the id four-tuple, is_orig, service and so on) are renamed to their standard locations at the top of 1200_zeek_mutate.conf for every log, so the parser doesn’t need to handle them.

Maps

Three maps in logstash/maps/ need entries, in alphabetical order:

For an OT protocol, also add its network.protocol values to the OT list in logstash/pipelines/enrichment/20_enriched_to_ecs.conf. That list categorizes other logs (such as conn.log) as OT based on their protocol. If the parser relabels some rows with a different protocol, add that value too.

Shared fields and cleanup: 1200_zeek_mutate.conf

Add a branch for the parser to the [log_source] chain in 1200_zeek_mutate.conf, in alphabetical order:

  } else if ([log_source] =~ /^foo/) {
    #############################################################################################################################
    # foo*.log specific logic

Do the renames from your plan in a single mutate at the start of the branch, so everything after it can use the final field names:

    mutate {
      id => "mutate_rename_foo_normalized_fields"
      rename => { "[zeek][foo_detail][request_id]" => "[zeek][foo][request_id]"
                  "[zeek][foo_errors][request_id]" => "[zeek][foo][request_id]"
                  "[zeek][foo][result_pairing_status]" => "[zeek][foo][pairing_status]" }
    }

A rename only needs to become a merge if two source fields in the same log land on the same target. If a rename only applies to some rows, put a conditional rename before the main one, while the field still has its original name.

After the renames, add any cleanup the values need:

Wrap filters that only apply to some rows in a conditional, so Logstash doesn’t run them for every event.

ECS fields: 1300_zeek_normalize.conf

1300_zeek_normalize.conf has sections for network.protocol_version, event.action and event.result, each listing the logs alphabetically. Add the parser’s fields to each section that applies, following the existing patterns:

  if ([zeek][foo][operation])                 { mutate { id => "mutate_merge_normalize_zeek_foo_operation"
                                                        merge => { "[event][action]" => "[zeek][foo][operation]" } } }

For coded values, translate them to names with an inline dictionary, with the raw value as the fallback:

  if ([zeek][foo][error_code]) {
    translate {
      id => "translate_zeek_foo_error_code"
      source => "[zeek][foo][error_code]"
      target => "[@metadata][zeek_foo_error_code]"
      dictionary => {
        "0" => "Success"
        "1" => "Access denied"
      }
      fallback => "%{[zeek][foo][error_code]}"
    }
    mutate { id => "mutate_merge_zeek_foo_error_code"
             merge => { "[event][result]" => "[@metadata][zeek_foo_error_code]" } }
  }

When a value combines an action with a stage, derive the base action (e.g., with a small ruby filter) and merge that into event.action, leaving the original field as it is.

Timestamps: 1400_zeek_convert.conf

Zeek writes timestamps as epoch seconds. For any timestamp field other than ts, add its final field name (after renames) to the @zeek_time_fields list in 1400_zeek_convert.conf. The generated boilerplate lists the time fields it found. Fields stored as integer epoch seconds need the same treatment, but the generator can’t detect those, so look for them in the parser’s documentation and code.

Each of these fields needs a date mapping in the template and kind:termfield in Arkime. The converter drops values of 0 or less, and values greater than 10000000000, as invalid, so make sure the values really are seconds by the time they reach this stage.

Document IDs: 2000_fingerprint.conf

Each document’s ID is a hash of some of its fields, computed in 2000_fingerprint.conf. Because the hash only uses fields derived from the traffic, reprocessing the same capture produces the same IDs and overwrites the existing documents instead of duplicating them (see cisagov/Malcolm#715). The catch is that two different rows with the same values in those fields also get the same ID, and the later one silently replaces the earlier one.

Logs without a branch of their own use the default fingerprint, which hashes the timestamp, the connection’s addresses and ports, event.action, event.result and a few other fields. That’s enough for logs that write one row per connection or transaction. A parser needs its own branch if it can write several rows that share those values, for example:

To build a branch, start from the base fields every Zeek branch uses, then add the fields that identify a row in that log:

  } else if ([event][dataset] == "foo_detail") {
    fingerprint {
      id => "fingerprint_zeek_foo_detail"
      source => [ "[event][provider]",
                  "[event][dataset]",
                  "[zeek][ts]",
                  "[source][ip]",
                  "[source][port]",
                  "[destination][ip]",
                  "[destination][port]",
                  "[event][action]",
                  "[event][result]",
                  "[network][is_orig]",
                  "[zeek][foo][request_id]",
                  "[zeek][foo_detail][item_index]" ]
      concatenate_sources => true
      # uses event.hash
      ecs_compatibility => "v8"
      method => "MURMUR3_128"
      base64encode => true
    }

You can check whether a branch is needed, and whether a candidate list of fields is enough, before indexing anything. Count the distinct rows in the raw Zeek logs while ignoring the uid column, and compare that with the number of distinct combinations of your fingerprint fields. If the second number is smaller, some rows will collide. Test and verify shows how to confirm it after indexing.

7. Field definitions

Every field needs a definition in three places for Malcolm to index it correctly and show it in Arkime. Adding new log fields describes each file in detail. For a Zeek parser:

All three must list the same fields. A quick script that extracts the field names from each file and compares the sets catches typos and missed renames before you test.

8. Documentation

9. Test and verify

Process your test captures through Malcolm, then compare what was indexed against the Zeek logs you kept from Before you start.

Processing captures

Upload each capture separately. Combining captures into one file with mergecap is convenient, but synthetic test traces often reuse the same addresses, ports and timestamps, and merging them makes Zeek treat several traces as one connection. If you do merge captures:

Logstash

The Logstash node stats show how many events went through each of the parser’s filters, and how long each took:

docker compose exec logstash curl -sSL -XGET http://localhost:9600/_node/stats/pipelines | \
  jq -r '.. | .filters? // empty | .[] | objects | select (.events.in > 0) | [.id, .events.in, .events.out, .events.duration_in_millis] | join (";")' | \
  sort -n -t ';' -k4 | grep foo

Check that:

OpenSearch

Run these in OpenSearch Dashboards’ Dev Tools. arkime_sessions3-* is Malcolm’s default network index pattern.

Document counts. Compare these with the number of distinct rows in the raw logs, ignoring uid (rows that differ only in uid get the same ID, which is intended). A shortfall usually means a fingerprint collision:

GET arkime_sessions3-*/_search
{"size":0,"track_total_hits":true,"query":{"prefix":{"event.dataset":"foo"}},
 "aggs":{"dataset":{"terms":{"field":"event.dataset","size":20}},
         "failures":{"filter":{"bool":{"should":[{"prefix":{"tags":"_rubyexception"}},{"prefix":{"tags":"_dissect"}},{"prefix":{"tags":"_dateparse"}}]}}}}}

failures should be 0. For TSV logs, the distinct row count is grep -v '^#' foo.log | cut -f1,3- | sort -u | wc -l, which drops the second column (uid).

Unexpected fields. Every field should come from the template. Anything else was mapped dynamically, which usually means a missed rename:

GET arkime_sessions3-*/_field_caps?fields=zeek.foo*

ECS values. Look at what actually landed in the ECS fields, and adjust the normalization if the values don’t make sense:

GET arkime_sessions3-*/_search
{"size":0,"query":{"prefix":{"event.dataset":"foo"}},
 "aggs":{"action":{"terms":{"field":"event.action","size":50}},
         "result":{"terms":{"field":"event.result","size":50}},
         "proto":{"terms":{"field":"network.protocol","size":10}},
         "version":{"terms":{"field":"network.protocol_version","size":10}},
         "category":{"terms":{"field":"event.category","size":10}}}}

Dates. Use min and max aggregations on each date field. Dates in 1970, or far in the future, mean the units are wrong.

Stable IDs. Reprocess the same captures without deleting anything. The document counts shouldn’t change.

If you change a fingerprint and reprocess, delete the parser’s existing documents first (e.g., POST arkime_sessions3-*/_delete_by_query with the same prefix query). Otherwise the documents indexed under the old IDs stay, and everything is counted twice.

Arkime and behavior checks

System tests

Malcolm’s system tests run against real captures, and new parsers should be added to them:

  1. Submit a pull request to Malcolm-Test-Artifacts adding a capture under pcap/protocols/, ideally one capture per protocol (e.g., pcap/protocols/FOO.pcap).
  2. Submit a pull request to Malcolm-Test adding the capture’s path to UPLOAD_ARTIFACTS and the parser’s datasets to EXPECTED_DATASETS, in src/maltest/tests/test_ot_protocols.py for OT/ICS protocols or src/maltest/tests/test_common_protocols.py for others.

See malcolm-test: Malcolm System Tests for how to run them.

10. Dashboards

Build a dashboard for the protocol, export it into dashboards/dashboards/ as described in Adding new visualizations and dashboards, and add a link to it in the navigation pane shared by all dashboards (dashboards/dashboards/navigation.md). Protocol dashboards go on the ICS/IoT Protocols or Common Protocols line, in alphabetical order, separated by ` ● `:

[Foo](#/dashboard/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

11. Report problems upstream

Integrating a parser often turns up problems in the package itself: documentation that doesn’t match the code, fields whose units vary between rows, one field carrying unrelated values, or very slow startup under ZAM. Report them to the package’s maintainers with the evidence (the code location, the log rows and the captures that show it). If Malcolm needs a workaround in the meantime, add a comment next to it linking to the upstream issue, so it can be removed once the package is fixed.

Checklist