HowTo: monitoring with Checkmk from scratch

Checkmk gives you a single board across servers, network, services and applications, with automatic discovery and alerts. This guide takes you from install to distributed monitoring, step by step — and why each piece works.

Install and create a site

Checkmk works on isolated sites managed with omd (Open Monitoring Distribution). You can run several sites on one server.

apt install ./check-mk-raw-2.3.0p*_0.bookworm_amd64.deb   # Raw = free
omd create monitor                 # create the site
omd config monitor set ADMIN_MAIL alerts@example.com
omd start monitor
omd status monitor                 # apache, core, rrdcached etc.
# initial 'cmkadmin' password is shown at create; change it in the UI

The agent: install + TLS registration

By default, Checkmk works in pull mode: the server connects to the agent on port 6556 (TLS) and pulls the data every cycle — the agent doesn't send it. (Push exists only for hosts the server can't reach.) The new agent (cmk-agent-ctl) registers encrypted and no longer exposes a plaintext port. Download the package from Setup > Agents.

dpkg -i check-mk-agent_2.3.0p*.deb
cmk-agent-ctl register --hostname web01 \
  --server checkmk.example.com --site monitor \
  --user automation --password '***'
cmk-agent-ctl status               # TLS: connection established
systemctl status check-mk-agent.socket

Add the host + discovery

Add the host (UI or API), run discovery and activate. Checkmk finds disks, interfaces and services by itself.

cmk -vI web01          # inventory/discovery: which services the host has
cmk -II web01          # full re-discovery (incl. vanished services)
cmk -O                 # activate changes (reload core)
cmk -D web01           # dump: how the host is configured

SNMP for switch, firewall, UPS

Network gear has no agent — you read it via SNMP. Set the community/v3 credentials on the host and run discovery.

# in host properties: SNMP community / SNMPv3 credentials
cmk -v --snmpwalk sw01             # check it responds
cmk -vI sw01                       # ports, temperature, PSU, traffic
# rules: 'Interfaces' for traffic thresholds, 'Temperature' etc.

Rules, folders and thresholds

Checkmk's power is in rules: you apply thresholds to groups of hosts (folders), not manually per host. E.g. disk and CPU thresholds on the whole /clients folder.

# Setup > Host & Service Parameters > Filesystems (used space)
#   WARN 80% / CRIT 90%, applied to the /clients folder
# Setup > ... > CPU utilization, Memory, Uptime
# 'Periodic service discovery' — auto-discovers new services daily

Notifications with escalation

Alerts via email, Telegram, webhook — with rules (who, when, which severity) and escalation if not acknowledged.

# Setup > Notifications > Add rule
#  - contact group: NOC team
#  - conditions: CRIT + host in the /clients folder
#  - method: email + Telegram (HTML script)
#  - escalation: repeat every 30 min if not ACKed
# test: 'Analyse notification' on a service in the UI

Distributed monitoring

For many locations: a central site and remote sites that collect locally and forward to central. Useful when monitoring several clients from a single NOC.

# on central: Setup > Distributed Monitoring > Add connection
#   -> connect the remote site (livestatus TCP/TLS on 6557)
# the remote runs its own core; central aggregates the views
omd config remote1 set LIVESTATUS_TCP on
omd restart remote1

Local checks + API

For any custom metric you write a local check (a script returning status + value). And everything you do in the UI can be done via the REST API.

# local check on the agent: /usr/lib/check_mk_agent/local/backup_age
#!/bin/bash
AGE=$(( ($(date +%s) - $(stat -c %Y /backup/last)) / 3600 ))
if [ $AGE -gt 26 ]; then echo "2 BackupAge age=$AGE last backup too old"; else echo "0 BackupAge age=$AGE ok"; fi
# API: add host + activate
curl -X POST '.../api/1.0/domain-types/host_config/collections/all' -H 'Authorization: Bearer automation ***' ...