🗺️ Prometheus Exporter Network Map

This document describes how Prometheus exporters are deployed in this repository: which host each exporter runs on, which host or service it collects data from, which Prometheus server scrapes it, and how Grafana consumes the result.

It is a reference map, not a runbook. For adding new targets, see:

• adding-node-exporter-to-an-inventory.md
• adding-nginx-exporter-to-an-inventory.md
• adding-postgres-exporter-to-an-inventory.md
• adding-blackbox-targets.md

🧭 Network Map

grafana-0  (Grafana)
   │  datasource: http://prometheus-0:9090
   v
prometheus-0  (Prometheus :9090) ───> Alertmanager (prometheus-0)
   │
   ├── scrape 5s ───────> job: node ─ node_exporter :9200
   │                        ansible-0, dns-0/1, rproxy-0/1/2, pg-0 … pg-4,
   │                        plex-0, tautulli-0, minecraft-1, redmine-0,
   │                        semaphore-0, vpn-0, sonarr-0, radarr-0,
   │                        lidarr-0, sabnzbd-0, books-0
   │
   ├── scrape 5s ───────> job: node ─ pve-0/1/2  node_exporter :9100
   │
   ├── scrape 30s /pve ─> job: pve_exporter ─ pve-0/1/2 :9221
   │                        └── reads ───> own node's PVE API (target=localhost)
   │
   ├── scrape 5s ───────> job: nginx_exporter ─ rproxy-0/1/2 :9113
   │                        └── reads ───> local nginx 127.0.0.1:9114/stub_status
   │
   ├── scrape 5s ───────> job: postgres_exporter ─ pg-0 … pg-4 :9187
   │                        └── queries ─> local PostgreSQL 127.0.0.1:5432
   │
   └── /probe?target=… ─> blackbox_exporter :9115 (runs on prometheus-0)
                            └── HTTP probe ─> ombi, books, lab, lidarr, radarr,
                                              sonarr, sabnzbd, semaphore,
                                              demo.ecube.one, lazy, tautulli

🧾 Network Map Summary

Each row is one deployed exporter instance: where it runs, what it reads, and where the metrics go.

📊 Exporter Summary

Every exporter except Blackbox runs on the host it measures, so all collection is local and Prometheus pulls over the network.

🔍 Notable Behaviors

Proxmox exporter is per-node, not per-cluster

pve_exporter runs on each Proxmox node but scrapes that node’s own local API rather than a single cluster endpoint. The same cluster-wide API token works on all nodes. Prometheus passes module and target parameters to /pve and uses a 30 second interval.

Blackbox is centralized

blackbox_exporter runs on prometheus-0 itself. Prometheus lists the probe URLs as job targets, then relabel_configs rewrite __address__ back to the local blackbox listener so the probe is executed from prometheus-0.

Probe targets are defined in inventory/prometheus/group_vars/all/main.yml under prometheus_setup_blackbox_targets.

Scrape targets are merged, not replaced

roles/prometheus_setup/tasks/exporters.yml reads the existing prometheus.yml from the Prometheus host, seeds a per-job target map keyed by the instance label, and then merges targets derived from the current inventory on top.

This means targets registered by a previous run against a different inventory stay in place. Running playbooks/prometheus/deploy_prometheus_exporters.yml against a single inventory does not remove hosts from other inventories.

To remove a stale target, edit the rendered configuration on the Prometheus host or remove the host and re-render, since inventory alone cannot delete a previously merged entry.

Grafana

Grafana is provisioned with a single default Prometheus datasource pointing at the first host in the prometheus group:

grafana_setup_prometheus_datasource_url: "http://{{ global_ip_addresses[groups['prometheus'][0]] }}:9090"

Dashboards in roles/grafana_setup/files/dashboards/ reference that datasource by the prometheus UID and cover node, nginx, PostgreSQL, Proxmox, web service, and reverse proxy health views.

🧱 Where Things Are Defined