web-log/container/site/content/posts/server-monitor/index.md

+++ categories = ["software"] tags = ["automation","grafana","influxdb","telegraf"] date = 2025-05-16T08:00:00-05:00 description = "" draft = false slug = "server-monitor" title = "📈 Server Monitoring with Grafana" author = "nicholas" +++

In this post I will develop a server monitoring dashboard that will show me the status of my server at a glance. The goal here is to replicate the functionality of a simple command-line "real-time" system information monitor e.g. top, htop, etc. via a web interface.

{{< image src="images/btop.png" caption="Btop UI" >}}

• A web-based monitoring dashboard, as opposed to e.g. htop, will be useful to me because it will allow me to access this dashboard from virtually any OS, anywhere at any time. No more ssh-ing into my machine and running all manner of monitoring-related commands.
• The web interface will also make integrating with other web-based dashboards trivial, which is something I plan to do later so that I can show my server stats alongside things like calendars, reminders, etc. in a master dashboard.
• The tools I plan to use will allow me to configure alerts that will notify me if certain metrics go beyond threshold values, e.g. disk space, CPU utilization > 90%, etc.

Project Scope

There are a few pieces of information I want immediately visible when I visit my dashboard:

• disk temperatures
• fan speeds
• CPU temperatures
• CPU utilization
• processes & threads

This is the basic set of host-level metrics that I care about at the moment, and once I am able to monitor these I will consider the project done. I will be using tools that will make additional metrics easy to add later, e.g. network traffic, container-level metrics, etc.

Tools

Here is a diagram of the tools I will use and how they relate to each other.

graph TD
  subgraph "🖧 Host Machine"
    A[📊 Host metrics] -->|📥 Collect| B[Telegraf]
    B[📦 Telegraf] -->|📊 Write| C[InfluxDB]
    D[Grafana] -->|📨 Query|C[📦 InfluxDB]
    C[📦 InfluxDB] -->|📈 Time-series data| D[Grafana]
  end

  D[📦 Grafana] -->|🌐 Web Browser| E[👤 User]

Stuck with InfluxDB v2

Unfortunately, I cannot run the latest version of Influxdb: Influxdb:3. My CPU does not support the required AVX instruction set, so the best I can do is Influxdb:2. I run into this same limitation each time I try to run a modern version (>=5) of MongoDB, which I use for other projects. Nevertheless, this older version will work fine, although I will be stuck writing my data queries in Flux instead of SQL. I really hate that.

Configuration

Directory Structure

Since I will want to be able to easily edit a telegraf.conf configuration file from the host machine, I will bind-mount a telegraf subdirectory within the root monitor directory. This will contain my telegraf.conf. Also, I will create a .env file to store secret values to pass to my docker-compose.yml file. The directory structure will look like this:

📁monitor/
├── docker-compose.yml
├── .env
├── 📁telegraf/
    └── telegraf.conf

Telegraf

Telegraf is configured in telegraf.conf. I provide my configuration below. It contains many more input plugins than are being used, but I wanted to cover my bases.

The two most important sections of this configuration file are the agent and the output sections.

• [agent] - defines agent-level settings for the Telegraf daemon.
  • hostname = "server" sets the host tag for all metrics collected by Telegraf.
• [[outputs.influxdb_v2]] - defines the output plugin, or where Telegraf sends the collected metrics. Targets InfluxDB v2.
  • bucket = "telegraf" defines the InfluxDB v2 bucket where metrics will be written.

Relevant to the example query I use below is the [[inputs.smart]] plugin. This plugin uses smartctl (smartmontools) to query disk health and attributes from the devices listed in devices = [ "/dev/sda","/dev/sdb","/dev/sdc","/dev/sdd","/dev/sde","/dev/sdf" ]. Since I added the line attributes = true, I will have access to individual SMART attributes e.g. temperature, reallocated sectors, power on hours, etc.

[agent]
  interval = "10s"
  round_interval = true
  hostname = "server"

[[outputs.influxdb_v2]]
  urls = ["http://server:3997"]
  token = "<token>"
  organization = "ward"
  bucket = "telegraf"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = true

[[inputs.diskio]]
  devices = ["sda", "sdb", "sdc", "sdd", "sde", "sdf"]

[[inputs.procstat]]
  pattern = ".*"
  pid_finder = "native"

[[inputs.smart]]
  path_smartctl = "/usr/sbin/smartctl"
  use_sudo = true
  devices = [ "/dev/sda","/dev/sdb","/dev/sdc","/dev/sdd","/dev/sde","/dev/sdf" ]
  attributes = true

[[inputs.mem]]
[[inputs.disk]]
[[inputs.net]]
[[inputs.netstat]]
[[inputs.system]]
[[inputs.sensors]]
[[inputs.kernel]]
[[inputs.swap]]
[[inputs.interrupts]]
[[inputs.linux_sysctl_fs]]
[[inputs.processes]]
[[inputs.zfs]]

Docker image

I am not using the official Telegraf image, but instead a third-party customized variant golift/telegraf. This is because I needed to monitor HDD temperatures, and wanted to use smartctl (smartmontools) to do it. Telegraf can do this, but using it inside a container makes this difficult.

The repository README explains the issue:

Provides a telegraf docker image with added tools for monitoring disks, sensors and IPMI. This exists because the base telegraf Docker image makes it difficult to monitor some system metrics.

Re-creates the official Telegraf docker container with the following tools added:

• smartctl (smartmontools)
• ipmitool
• nvme-cli
• sensors (lm-sensors)
• mtr (mtr-tiny)
• sudo

env

GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=<password>

GRAFANA_PORT=3999
INFLUXDB_PORT=3997
TELEGRAF_PORT=3996

DOCKER_INFLUXDB_INIT_USERNAME=admin
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=<token>
DOCKER_INFLUXDB_INIT_PASSWORD=<password>
DOCKER_INFLUXDB_INIT_ORG=ward

Docker Compose

volumes:
  grafana_data:
  influxdb_data:


networks:
  monitor_net:
    name: monitor_net
    driver: bridge

services:
  influxdb:
    image: influxdb:2
    container_name: influxdb

    ports:
      - "${INFLUXDB_PORT}:8086"
    restart: unless-stopped
    volumes:
      - influxdb_data:/var/lib/influxdb2
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=${DOCKER_INFLUXDB_INIT_USERNAME}
      - DOCKER_INFLUXDB_INIT_PASSWORD=${DOCKER_INFLUXDB_INIT_PASSWORD}
      - DOCKER_INFLUXDB_INIT_ORG=${DOCKER_INFLUXDB_INIT_ORG}
      - DOCKER_INFLUXDB_INIT_BUCKET=telegraf
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}
    networks:
      - monitor_net

  telegraf:
    image: golift/telegraf
    privileged: true
    container_name: telegraf
    restart: unless-stopped
    volumes:
      - ./telegraf:/etc/telegraf:ro
      - /:/hostfs:ro
    environment:
      - HOST_ETC=/hostfs/etc
      - HOST_PROC=/hostfs/proc
      - HOST_SYS=/hostfs/sys
      - HOST_VAR=/hostfs/var
      - HOST_RUN=/hostfs/run
      - HOST_MOUNT_PREFIX=/hostfs
    ports:
      - "${TELEGRAF_PORT}:9273"
    networks:
      - monitor_net

  grafana:
    image: grafana/grafana-oss
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
      #- GF_SERVER_ROOT_URL=http://grafana.uuard.com
    ports:
      - "${GRAFANA_PORT}:3000"
    volumes:
      - 'grafana_data:/var/lib/grafana'
    healthcheck:
      test: [ "CMD", "wget", "--spider", "http://localhost:3000/login" ]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - monitor_net

Grafana Config & Examples

The first two steps of configuring data collection(via Telegraf) and writing (via InfluxDB) are done. Now I need to fetch, process, and display the data to my liking (via Grafana). This mostly involves writing ugly Flux queries and selecting a visualization to represent the data.

HDD Temperature

This query gives me the temperature (°C) of each HDD in my machine:

from(bucket: "${datasource}")
  |> range(start: -1m)
  |> filter(fn: (r) =>
    r.host == "${host}" and
    r._measurement == "smart_device" and
    r._field == "temp_c")
  |> group(columns: ["device"])
  |> last()
  |> yield(name: "temperature")

In this query, there are two variables used: datasource and host. These were both defined in telegraf.conf.

[agent]
  ...
  hostname = "server"

[[outputs.influxdb_v2]]
  ...
  bucket = "telegraf"

The query returns the latest (last) temp_c values per device, for a single host, from the last 60 seconds of SMART metrics. I selected the gauge visualization to represent this.

CPU Core Utilization

This query gives me the CPU utilization per core (100% minus IDLE).

from(bucket: "${datasource}")
  |> range(start: -1m)
  
  |> filter(fn: (r) =>
    r.host == "${host}" and
    r._measurement == "cpu" and
    r._field == "usage_idle" and
    r.cpu != "cpu-total"
  )
  |> group(columns: ["cpu"])
  |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)
  |> map(fn: (r) => ({ r with _value: 100.0 - r._value }))
  |> yield(name: "core_total_usage")

Result

Here is what this looks like alongside the rest of the metrics I configured:

{{< image src="images/grafana-live-dashboard.png" caption="Grafana Dashboard" >}}

Each of the visualizations in this image are created with Flux queries similar to the example above. I don't bother writing each one out here. Instead, I attach a JSON file that can be imported into Grafana. It contains among many other details all of the queries used in the image.

Done.