Logo

A powerful, easily deployable network traffic analysis tool suite for network security monitoring

Quick Start

Documentation

Components

Supported Protocols

Configuring

Arkime

Dashboards

Hedgehog Linux

Contribution Guide

Linux host system configuration

For most Linux distributions, Malcolm’s install.py script will perform the steps listed in this document – installing Docker, configuring the Linux kernel and other OS parameters, etc. – automatically. See the Installation example using Ubuntu 24.04 LTS](ubuntu-install-example.md#UIOpts) for an example. However, this document is provided as a reference for users who would prefer to do it manually

Operating system configuration

The host system (i.e., the one running Docker or Podman) must be configured for the best possible OpenSearch performance. Here are a few suggestions for Linux hosts (these may vary from distribution to distribution):

# the maximum number of open file handles
fs.file-max=2097152

# increase maximums for inotify watches
fs.inotify.max_user_watches=131072
fs.inotify.max_queued_events=131072
fs.inotify.max_user_instances=512

# decrease "swappiness" (swapping out runtime memory vs. dropping pages)
vm.swappiness=1

# the maximum number of memory map areas a process may have
vm.max_map_count=524288

# decrease "swappiness" (swapping out runtime memory vs. dropping pages)
vm.swappiness=1

# the % of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio=5

# maximum % of dirty system memory before committing everything
vm.dirty_ratio=10

# virtual memory accounting mode: always overcommit, never check
vm.overcommit_memory=1
# maximum number of TCP retransmissions
net.ipv4.tcp_retries2=5
# the maximum number of open file handles
* soft nofile 65535
* hard nofile 65535
# do not limit the size of memory that can be locked
* soft memlock unlimited
* hard memlock unlimited

OR the file /etc/systemd/system.conf.d/limits.conf containing:

[Manager]
# the maximum number of open file handles
DefaultLimitNOFILE=65535:65535
# do not limit the size of memory that can be locked
DefaultLimitMEMLOCK=infinity
# change disk read-adhead value (# of blocks)
blockdev --setra 512 /dev/sda

After making allthese changes, do a reboot for good measure!

Docker

Installing Docker

Docker installation instructions vary slightly by distribution. Please follow the links below to docker.com to find the instructions specific to your distribution:

After installing Docker, because Malcolm should be run as a non-root user, add your user to the docker group with something like:

$ sudo usermod -aG docker yourusername

Following this, either reboot or log out, then log back in.

Docker starts automatically on DEB-based distributions. On RPM-based distributions, users must start Docker manually or enable it using the appropriate systemctl or service command(s).

You can test Docker by running docker info, or (assuming you have internet access), docker run --rm hello-world.

Installing docker compose

Please follow this link on docker.com for instructions on installing the Docker Compose plugin.

Podman

Malcolm can run on Podman as a rootless alternative to Docker. The same Malcolm runtime scripts (e.g., ./scripts/start, ./scripts/stop, etc.) are used whether using Docker or Podman. When running Malcolm with Podman, podman compose is used as a wrapper around an external compose provider (such as docker-compose), which in turn uses the Podman back end to run and orchestrate containers. It is recommended to use the docker-compose compose provider rather than podman-compose since it is the original implementation of the Compose specification and is widely used on the supported platforms and because there are known issues with using the podman-compose provider to start Malcolm.

It should be noted that if rootless Podman is used, Malcolm itself cannot perform traffic capture on local network interfaces, although it can accept network traffic metadata forwarded from a a network sensor appliance.

Podman installation example

Although Malcolm can use Podman, Malcolm’s install.py script does not attempt to install or configure Podman because the procedure differs between distributions. In most cases, Podman can be installed from distributions’ default package repositories using apt/apt-get, yum, dnf, etc.. Podman v5.6.0 or higher is recommended due to fixes relating to user namespace management. Some third-party repositories provide more up-to-date packages for Podman and its dependencies.

The following process of installing and configuring Podman is for example’s sake only, and may not represent what’s required on each Linux distribution. This example uses a fresh instance of the Ubuntu 24.04 LTS cloud image as its basis and the third-party alvistack repository as the package source for Podman, but the steps will be similar on other distributions.

$ sudo apt-get update
…
Fetched 38.6 MB in 8s (4901 kB/s)

$ apt-cache policy podman
podman:
  Installed: (none)
  Candidate: 4.9.3+ds1-1ubuntu0.2
  …

$ echo 'deb [signed-by=/etc/apt/trusted.gpg.d/home_alvistack.gpg] http://download.opensuse.org/repositories/home:/alvistack/xUbuntu_24.04/ /' \
  | sudo tee /etc/apt/sources.list.d/home:alvistack.list >/dev/null

$ curl -fsSL https://download.opensuse.org/repositories/home:alvistack/xUbuntu_24.04/Release.key \
  | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/home_alvistack.gpg >/dev/null

$ sudo tee /etc/apt/preferences.d/99-home_alvistack >/dev/null <<'EOT'
Package: *
Pin: origin download.opensuse.org
Pin-Priority: 1

Package: buildah catatonit conmon containernetworking containernetworking-plugins containers-common cri-o-runc crun libcharon-standard-plugins libslirp0 passt podman podman-aardvark-dns podman-netavark python3-podman-compose slirp4netns
Pin: origin download.opensuse.org
Pin-Priority: 1001
EOT

$ sudo apt-get update
…
Get:5 http://download.opensuse.org/repositories/home:/alvistack/xUbuntu_24.04  InRelease [1551 B]
Get:6 http://downloadcontentcdn.opensuse.org/repositories/home:/alvistack/xUbuntu_24.04  Packages [170 kB]
Fetched 172 kB in 3s (61.8 kB/s)
Reading package lists... Done

$ apt-cache policy podman
podman:
  Installed: (none)
  Candidate: 100:5.6.2-1
  …

$ sudo apt-get install -y \
    buildah \
    catatonit \
    crun \
    fuse-overlayfs \
    passt \
    podman \
    podman-aardvark-dns \
    podman-netavark \
    slirp4netns \
    uidmap
…
Setting up podman (100:5.6.2-1) ...
…

$ grep -q unprivileged_userns_clone /etc/sysctl.d/* || \
    sudo tee -a /etc/sysctl.d/99-userns.conf >/dev/null <<'EOT'
# allow unprivileged user namespaces
kernel.unprivileged_userns_clone=1
EOT

$ grep -q ip_unprivileged_port_start /etc/sysctl.d/* || \
    sudo tee -a /etc/sysctl.d/99-lowport.conf >/dev/null <<'EOT'
# allow lower unprivileged port bind
net.ipv4.ip_unprivileged_port_start=443
EOT

$ sudo mkdir -p /etc/modprobe.d && \
    echo "options overlay metacopy=off redirect_dir=off" \
    | sudo tee /etc/modprobe.d/podman.conf >/dev/null

$ [[ -d /etc/systemd/system ]] && \
    sudo mkdir -p /etc/systemd/system/user@.service.d && \
    echo -e "[Service]\\nDelegate=cpu cpuset io memory pids" \
    | sudo tee /etc/systemd/system/user@.service.d/delegate.conf >/dev/null

$ sudo touch /etc/subuid && sudo touch /etc/subgid

$ grep --quiet johndoe /etc/subuid || sudo usermod --add-subuids 200000-265535 johndoe

$ grep --quiet johndoe /etc/subgid || sudo usermod --add-subgids 200000-265535 johndoe

$ sudo loginctl enable-linger johndoe

$ sudo usermod -a -G systemd-journal johndoe

$ mkdir -p /run/user/1000/podman

$ systemctl --user enable --now podman.service

$ mkdir -p ~/.docker/cli-plugins && \
    curl -fsSL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 \
        -o ~/.docker/cli-plugins/docker-compose && \
    chmod 755 ~/.docker/cli-plugins/docker-compose

$ mkdir -p ~/.config/containers && \
    tee -a ~/.config/containers/containers.conf >/dev/null <<'EOT'
[engine]

runtime="crun"
compose_warning_logs=false

[containers]

default_ulimits = [
  "memlock=9223372036854775807:9223372036854775807",
  "nofile=65535:65535",
  "nproc=262143:524287"
]

[network]

default_subnet_pools = [
  {"base" = "172.27.0.0/16", "size" = 24},
]
EOT
$ git clone https://github.com/idaholab/Malcolm
Cloning into 'Malcolm'...
remote: Enumerating objects: 57191, done.
remote: Counting objects: 100% (872/872), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 57191 (delta 813), reused 790 (delta 772), pack-reused 56319 (from 2)
Receiving objects: 100% (57191/57191), 237.41 MiB | 4.05 MiB/s, done.
Resolving deltas: 100% (42544/42544), done.

$ sudo apt-get install -y \
    python3-pip \
    python3-ruamel.yaml \
    python3-dotenv \
    python3-dialog \
    dialog
…

$ sudo ./scripts/install.py --skip-splash
--- Malcolm Configuration Menu ---
Select an item number to configure, or an action:
├── 1. Container Runtime (current: docker)
…
--- Actions ---
  s. Save and Continue Installation
  w. Where Is...? (search for settings)
  x. Exit Installer
---------------------------------

Enter item number or action: 1
Container Runtime (current: docker)
1: docker
2: podman
3: kubernetes
Enter choice number (docker): 2
--- Malcolm Configuration Menu ---
Select an item number to configure, or an action:
├── 1. Container Runtime (current: podman)
…
--- Actions ---
  s. Save and Continue Installation
  w. Where Is...? (search for settings)
  x. Exit Installer
---------------------------------

Enter item number or action: s
--- Malcolm Installation Options ---
Select an item number to configure, or an action:
└── 1. Automatically Apply System Tweaks (current: Yes)

--- Actions ---
  s. Save and Continue
  x. Exit Installer

Enter item number or action: s
============================================================
FINAL CONFIGURATION SUMMARY
============================================================
Configuration Only                                : No
Auto Apply System Tweaks                          : Yes
Configuration Directory                           : /home/johndoe/Malcolm/config
Container Runtime                                 : podman
Run Profile                                       : malcolm
Process UID/GID                                   : 1000/1000
Container Restart Policy                          : No
Container Network                                 : default
Default Storage Locations                         : Yes
HTTPS/SSL                                         : Yes
Node Name                                         : ubuntu-noble-191
============================================================

Proceed with Malcolm installation using the above configuration? (y / N): y
…
[2025-11-05 18:17:34] (SUCCESS) [INSTALLER]: Installation completed successfully
$ sudo reboot

…

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.8.0-85-generic root=UUID=52645fa4-47a3-4335-8817-2637487e1980 ro systemd.unified_cgroup_hierarchy=1 cgroup_enable=memory swapaccount=1 cgroup.memory=nokmem console=tty1 console=ttyS0

$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 128240
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 65535
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 262144
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

$ cat /etc/sysctl.d/99* | grep -v '^#' | cut -d= -f1 | xargs -r -l sysctl
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.use_tempaddr = 0
fs.file-max = 2097152
fs.inotify.max_user_watches = 131072
fs.inotify.max_queued_events = 131072
fs.inotify.max_user_instances = 512
vm.max_map_count = 524288
vm.swappiness = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.overcommit_memory = 1
net.core.somaxconn = 65535
net.ipv4.tcp_retries2 = 5
net.ipv4.ip_unprivileged_port_start = 443
kernel.unprivileged_userns_clone = 1

$ podman info
host:
  arch: amd64
  buildahVersion: 1.41.5
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_100:2.1.13-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: e21e7c85b7637e622f21c57675bf1154fc8b1866'
  cpuUtilization:
    idlePercent: 97.65
    systemPercent: 1.28
    userPercent: 1.07
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2048
  hostname: ubuntu-noble-126
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.0-85-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 33095901184
  memTotal: 33655021568
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: podman-aardvark-dns_100:1.16.0-1_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: podman-netavark_100:1.16.1-1_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun_100:1.24-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.24
      commit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_100:0.0+20250919.623dbf6f-1_amd64
    version: |
      pasta 0.0+20250919.623dbf6f
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_100:1.3.3-1_amd64
    version: |-
      slirp4netns version 1.3.3
      commit: unknown
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 1m 40.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/johndoe/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/johndoe/.local/share/containers/storage
  graphRootAllocated: 102888095744
  graphRootUsed: 2843721728
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/johndoe/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.25.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.2

$ podman compose version
Docker Compose version v2.40.3

At this point, users may:

Note that the first time Podman Malcolm starts the Malcolm containers it may take a several minutes for Podman to internally remap the user namespaces. This brief penalty is only applicable during startup, not during runtime. During this initial startup process, users monitoring top may see high CPU and disk I/O from the podman, exe or storage-chown-by-maps processes.

Once Malcolm has started, the ./scripts/status and/or podman ps commands will show its containers running under Podman:

$ ./scripts/status
NAME                                   IMAGE                                                COMMAND                  SERVICE             CREATED         STATUS                   PORTS
malcolm-pipeline-api-1                 ghcr.io/idaholab/malcolm/api:26.01.0                 "/usr/bin/tini -- /u…"   api                 8 minutes ago   Up 8 minutes (healthy)   5000/tcp
malcolm-pipeline-arkime-1              ghcr.io/idaholab/malcolm/arkime:26.01.0              "/usr/bin/tini -- /u…"   arkime              8 minutes ago   Up 8 minutes (healthy)   8000/tcp, 8005/tcp, 8081/tcp
malcolm-pipeline-arkime-live-1         ghcr.io/idaholab/malcolm/arkime:26.01.0              "/usr/bin/tini -- /u…"   arkime-live         8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-dashboards-1          ghcr.io/idaholab/malcolm/dashboards:26.01.0          "/usr/bin/tini -- /u…"   dashboards          8 minutes ago   Up 8 minutes (healthy)   5601/tcp
malcolm-pipeline-dashboards-helper-1   ghcr.io/idaholab/malcolm/dashboards-helper:26.01.0   "/usr/bin/tini -- /u…"   dashboards-helper   8 minutes ago   Up 8 minutes (healthy)   28991/tcp
malcolm-pipeline-filebeat-1            ghcr.io/idaholab/malcolm/filebeat-oss:26.01.0        "/usr/bin/tini -- /u…"   filebeat            8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-filescan-1            ghcr.io/idaholab/malcolm/filescan:26.01.0            "/usr/bin/tini -- /u…"   filescan            8 minutes ago   Up 8 minutes (healthy)   8001/tcp, 8006/tcp
malcolm-pipeline-freq-1                ghcr.io/idaholab/malcolm/freq:26.01.0                "/usr/bin/tini -- /u…"   freq                8 minutes ago   Up 8 minutes (healthy)   10004/tcp
malcolm-pipeline-htadmin-1             ghcr.io/idaholab/malcolm/htadmin:26.01.0             "/usr/bin/tini -- /u…"   htadmin             8 minutes ago   Up 8 minutes (healthy)   80/tcp
malcolm-pipeline-keycloak-1            ghcr.io/idaholab/malcolm/keycloak:26.01.0            "/usr/bin/tini -- /u…"   keycloak            8 minutes ago   Up 8 minutes (healthy)   8080/tcp, 8443/tcp, 9000/tcp
malcolm-pipeline-logstash-1            ghcr.io/idaholab/malcolm/logstash-oss:26.01.0        "/usr/bin/tini -- /u…"   logstash            8 minutes ago   Up 8 minutes (healthy)   5044/tcp, 9001/tcp, 9600/tcp
malcolm-pipeline-netbox-1              ghcr.io/idaholab/malcolm/netbox:26.01.0              "/usr/bin/tini -- /u…"   netbox              8 minutes ago   Up 8 minutes (healthy)   9001/tcp
malcolm-pipeline-nginx-proxy-1         ghcr.io/idaholab/malcolm/nginx-proxy:26.01.0         "/sbin/tini -- /usr/…"   nginx-proxy         8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-opensearch-1          ghcr.io/idaholab/malcolm/opensearch:26.01.0          "/usr/bin/tini -- /u…"   opensearch          8 minutes ago   Up 8 minutes (healthy)   9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp
malcolm-pipeline-pcap-capture-1        ghcr.io/idaholab/malcolm/pcap-capture:26.01.0        "/usr/bin/tini -- /u…"   pcap-capture        8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-pcap-monitor-1        ghcr.io/idaholab/malcolm/pcap-monitor:26.01.0        "/usr/bin/tini -- /u…"   pcap-monitor        8 minutes ago   Up 8 minutes (healthy)   30441/tcp
malcolm-pipeline-postgres-1            ghcr.io/idaholab/malcolm/postgresql:26.01.0          "/sbin/tini -- /usr/…"   postgres            8 minutes ago   Up 8 minutes (healthy)   5432/tcp
malcolm-pipeline-redis-1               ghcr.io/idaholab/malcolm/redis:26.01.0               "/sbin/tini -- /usr/…"   redis               8 minutes ago   Up 8 minutes (healthy)   6379/tcp
malcolm-pipeline-redis-cache-1         ghcr.io/idaholab/malcolm/redis:26.01.0               "/sbin/tini -- /usr/…"   redis-cache         8 minutes ago   Up 8 minutes (healthy)   6379/tcp
malcolm-pipeline-strelka-backend-1     ghcr.io/idaholab/malcolm/strelka-backend:26.01.0     "/usr/bin/tini -- /u…"   strelka-backend     8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-strelka-frontend-1    ghcr.io/idaholab/malcolm/strelka-frontend:26.01.0    "/sbin/tini -- /usr/…"   strelka-frontend    8 minutes ago   Up 8 minutes (healthy)   57314/tcp
malcolm-pipeline-strelka-manager-1     ghcr.io/idaholab/malcolm/strelka-manager:26.01.0     "/sbin/tini -- /usr/…"   strelka-manager     8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-suricata-1            ghcr.io/idaholab/malcolm/suricata:26.01.0            "/usr/bin/tini -- /u…"   suricata            8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-suricata-live-1       ghcr.io/idaholab/malcolm/suricata:26.01.0            "/usr/bin/tini -- /u…"   suricata-live       8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-upload-1              ghcr.io/idaholab/malcolm/file-upload:26.01.0         "/usr/bin/tini -- /u…"   upload              8 minutes ago   Up 8 minutes (healthy)   22/tcp, 80/tcp
malcolm-pipeline-zeek-1                ghcr.io/idaholab/malcolm/zeek:26.01.0                "/usr/bin/tini -- /u…"   zeek                8 minutes ago   Up 8 minutes (healthy)
malcolm-pipeline-zeek-live-1           ghcr.io/idaholab/malcolm/zeek:26.01.0                "/usr/bin/tini -- /u…"   zeek-live           8 minutes ago   Up 8 minutes (healthy)