Logo

A powerful, easily deployable network traffic analysis tool suite for network security monitoring

Quick Start

Documentation

Components

Supported Protocols

Configuring

Arkime

Dashboards

Hedgehog Linux

Contribution Guide

Adding new log fields

Although OpenSearch is a NoSQL database and as-such is “unstructured” and “schemaless,” adding a new data source field requires it be defined in several places in order for it to show up and be usable throughout Malcolm. Minimally, this involves three files:

All three must list the same fields. When possible, it is recommended to use (or at least take inspiration from) the Elastic Common Schema (ECS) Reference when deciding how to define new field names.

If new log fields are coming from a Zeek script where they are defined in a record as &log fields, the script scripts/zeek_script_to_malcolm_boilerplate.py generates the parts of all three files. See Adding a new Zeek parser for the whole process.

OpenSearch index templates

Each file in dashboards/templates/composable/component/ is imported as a component template named custom_<filename> (by dashboards/scripts/shared-object-creation.sh). The index templates list the components they’re built from in composed_of:

Add fields to the component that matches their source:

A new component file is only needed for a new data source. When you add one, also add custom_<filename> to composed_of in the index template that should use it.

Within a component, fields are nested under template.mappings.properties, e.g., zeek.foo.request_id is under zeek, then foo, then request_id. Keep each level in alphabetical order. The types used are keyword for strings, long or integer for integers, float for floating-point values and date for timestamps. A long is the safe choice for Zeek’s int and count, which are both 64-bit: if a value doesn’t fit a field’s mapping, OpenSearch rejects the whole document, not just that field. Fields mapped as date must contain dates when they’re indexed (see 1400_zeek_convert.conf for Zeek logs).

The index templates limit the number of fields an index can have (mapping.total_fields.limit), counting the fields from every component they’re composed of. Keep an eye on it when adding a large number of fields.

Malcolm compares a hash of all the template files at startup and re-imports them if anything has changed. Changed mappings only apply to indices created after the import, so test new fields in a fresh index.

Arkime

Field definitions

Fields are defined in the [custom-fields] section of arkime/etc/config.ini, one per line:

zeek.foo.request_id=db:zeek.foo.request_id;group:zeek_foo;kind:integer;viewerOnly:true;friendly:request_id;help:request_id

The definitions for each log go together, under a comment naming the log.

Session views

The [custom-views] section defines the sections of Arkime’s session detail panel, one per log, sorted by name:

o_zeek_foo_detail=require:zeek.foo_detail;title:Zeek foo_detail.log;fields:zeek.foo.request_id,zeek.foo_detail.item_index,zeek.foo_detail.value

Groups and views are independent: a view shows whatever fields it lists, whatever group they belong to.

WISE value actions

Add every field name to the allFields array in arkime/wise/source.zeeklogs.js, keeping the array in sorted order:

      "zeek.foo.request_id",
      "zeek.foo_detail.item_index",