📊 Nginx Reverse Proxy Monitoring

This documentation covers the setup and deployment of Nginx reverse proxy monitoring using Prometheus and the nginx-prometheus-exporter. All deployment is managed through Ansible.

Overview
Architecture
Components
Deployment
Configuration
Ansible Playbook
Scraping Metrics
Verification
Troubleshooting

Overview

The Nginx reverse proxy (rproxy) monitoring solution provides real-time visibility into reverse proxy performance and health. It collects metrics such as:

Active connections and connection states
Request rates and response codes
Request/response throughput
Cache hit/miss ratios
Upstream server health

The monitoring stack consists of:

nginx-prometheus-exporter — Scrapes Nginx status and exports metrics in Prometheus format
Local Nginx status endpoint — Provides Nginx statistics via stub_status
Prometheus server — Collects and stores metrics from exporters
Grafana dashboards — Visualizes Nginx metrics

Architecture

Component Interaction

┌─────────────────────────────────────────────────────────────┐
│ Reverse Proxy Host (rproxy)                                 │
│                                                             │
│  ┌──────────────────┐          ┌──────────────────────┐     │
│  │ Nginx Reverse    │          │ Local Nginx Status   │     │
│  │ Proxy (port 80)  ├─────-────┤ Endpoint (port 9114) │     │
│  └──────────────────┘          └──────────────────────┘     │
│           │                              ▲                  │
│           │                              │                  │
│  ┌────────┴──────────────────────────────┴──────────┐       │
│  │ nginx-prometheus-exporter (port 9113)            │       │
│  │ - Connects to localhost:9114/stub_status         │       │
│  │ - Exports Prometheus metrics on /metrics         │       │
│  └───────────────────────┬──────────────────────────┘       │
│                          │                                  │
└──────────────────────────┼──────────────────────────────────┘
                           │
                    [Metrics TCP 9113]
                           │
                           ▼
        ┌──────────────────────────────────┐
        │ Prometheus Server                │
        │ (port 9090)                      │
        │                                  │
        │ Scrapes nginx_exporter targets   │
        │ - Every 5 seconds                │
        └──────────────────────────────────┘
                           │
                           ▼
        ┌──────────────────────────────────┐
        │ Grafana                          │
        │ (port 3000)                      │
        │                                  │
        │ Visualizes Nginx metrics         │
        └──────────────────────────────────┘

Data Flow

Nginx reverse proxy logs connection and request statistics
Local Nginx status vhost exposes /stub_status endpoint
nginx-prometheus-exporter polls the status endpoint every second
Exporter converts text metrics to Prometheus format
Prometheus scrapes exporter on port 9113 every 5 seconds
Metrics are stored in Prometheus time-series database
Grafana queries Prometheus and displays dashboards

Components

1. nginx-prometheus-exporter

The nginx-prometheus-exporter bridges Nginx status data and Prometheus.

Role: roles/nginx_prometheus_exporter_setup

Key Features:

Converts Nginx metrics to Prometheus format
Runs as dedicated systemd service
Listens on port 9113 by default
Scrapes internal Nginx status endpoint
Minimal CPU/memory footprint

Binary Details:

Version: 1.5.1
Location: /usr/local/bin/nginx-prometheus-exporter
User/Group: nginx-prometheus-exporter:nginx-prometheus-exporter
Service: nginx-prometheus-exporter.service

2. Local Nginx Status Vhost

A dedicated Nginx virtual host exposes statistics via the stub_status module.

Configuration:

Server name: nginx-prometheus-exporter-status
Listen address: 127.0.0.1:9114
Status path: /stub_status
Access: Localhost only (security)

Nginx Config Template: roles/nginx_prometheus_exporter_setup/templates/nginx-prometheus-exporter-status.conf.j2

3. Prometheus Configuration

Prometheus is configured to scrape nginx-prometheus-exporter targets.

Scrape Job:

Job name: nginx_exporter
Scrape interval: 5 seconds
Metrics path: /metrics
Port: 9113

Prometheus Config Template: roles/prometheus_setup/templates/prometheus.yml.j2

Deployment

Prerequisites

Ansible 2.9 or later
Ubuntu 18.04 LTS or later on rproxy hosts
Nginx already installed via nginx_setup role
Prometheus server accessible for configuration updates

Deployment Chain

The deploy_rproxy.yml playbook deploys the complete rproxy stack in this order:

roles:
  - global                           # Set up global variables and facts
  - nginx_setup                      # Install and configure Nginx
  - nginx_prometheus_exporter_setup  # Deploy exporter
  - rproxy_setup                     # Configure reverse proxy sites

How to Deploy

1. Run the deployment playbook:

ansible-playbook \
  -k \
  -i inventory/rproxy/inventory.ini \
  playbooks/rproxy/deploy_rproxy.yml

2. Verify deployment:

ansible rproxy -i inventory/rproxy/inventory.ini -m systemd \
  -a "name=nginx-prometheus-exporter state=started"

3. Test metrics endpoint:

curl http://rproxy-host:9113/metrics

Configuration

Inventory Configuration

Nginx exporter targets are defined in Prometheus inventory.

File: inventory/prometheus/group_vars/all/main.yml

Example Configuration:

prometheus_setup_nginx_exporter_targets:
  - target: "rproxy-0:9113"
    labels:
      instance: "rproxy-0"
      site: "reverse-proxy"
  - target: "rproxy-1:9113"
    labels:
      instance: "rproxy-1"
      site: "reverse-proxy"

Role Variables

Role: roles/nginx_prometheus_exporter_setup/defaults/main/main.yml

Key configurable variables:

Variable	Default	Purpose
`nginx_prometheus_exporter_setup_version`	`1.5.1`	Exporter version to deploy
`nginx_prometheus_exporter_setup_listen_address`	`0.0.0.0:9113`	Exporter listen address and port
`nginx_prometheus_exporter_setup_telemetry_path`	`/metrics`	Prometheus metrics endpoint path
`nginx_prometheus_exporter_setup_scrape_uri`	`http://127.0.0.1:9114/stub_status`	Nginx status endpoint URI
`nginx_prometheus_exporter_setup_user`	`nginx-prometheus-exporter`	Service user
`nginx_prometheus_exporter_setup_nginx_status_listen_address`	`127.0.0.1:9114`	Local status vhost listen address

Override these in inventory group_vars if needed:

nginx_prometheus_exporter_setup_version: "1.6.0"
nginx_prometheus_exporter_setup_listen_address: "localhost:9113"

Ansible Playbook

Main Playbook: deploy_rproxy.yml

The primary deployment playbook for reverse proxies.

File: playbooks/rproxy/deploy_rproxy.yml

Content:

---
# Purpose: Sets up Reverse Proxy on rproxy hosts.

- name: Deploy Reverse Proxy
  hosts: rproxy
  become: true
  gather_facts: true
  roles:
    - global
    - nginx_setup
    - nginx_prometheus_exporter_setup
    - rproxy_setup

Role Execution

1. global role

Gathers system facts
Sets up global variables (IP addresses, ports, etc.)

2. nginx_setup role

Installs Nginx
Configures base Nginx settings

3. nginx_prometheus_exporter_setup role

Creates systemd user and group
Downloads and installs exporter binary
Creates local status vhost config
Enables status vhost in Nginx
Creates systemd service file
Starts exporter service

4. rproxy_setup role

Creates reverse proxy sites
Applies SSL certificates
Configures upstream targets
Reloads Nginx with all configs

Role Tasks

File: roles/nginx_prometheus_exporter_setup/tasks/main.yml

Key tasks:

Create service user and group
Download and extract exporter binary
Create symlink for easy binary access
Set proper ownership and permissions
Template local Nginx status vhost config
Enable status vhost in Nginx
Validate Nginx configuration
Template systemd service file
Enable and start service

Scraping Metrics

Prometheus Scrape Configuration

Prometheus is configured via roles/prometheus_setup/templates/prometheus.yml.j2

Nginx exporter scrape job:

{% if prometheus_setup_nginx_exporter_targets | default([]) | length > 0 %}
  - job_name: 'nginx_exporter'
    scrape_interval: 5s

    static_configs:
{% for target in prometheus_setup_nginx_exporter_targets %}
      - targets: ['{{ target.target }}']
{% if target.labels is defined %}
        labels:
{% for key, value in target.labels.items() %}
          {{ key }}: '{{ value }}'
{% endfor %}
{% endif %}
{% endfor %}
{% endif %}

How Targets are Passed to Prometheus

Inventory defines targets in inventory/prometheus/group_vars/all/main.yml:

prometheus_setup_nginx_exporter_targets:
  - target: "rproxy-0:9113"
    labels:
      instance: "rproxy-0"

Prometheus playbook deploys with targets:

ansible-playbook \
  -i inventory/prometheus/inventory.ini \
  playbooks/prometheus/deploy_prometheus.yml

Prometheus configuration template renders targets using Jinja2 loops
Prometheus service reloads and begins scraping targets

Exported Metrics

The nginx-prometheus-exporter exports these metric families:

Metric	Type	Description
`nginx_up`	Gauge	Whether Nginx is running (1) or down (0)
`nginx_requests_total`	Counter	Total requests processed
`nginx_connections_active`	Gauge	Active connections
`nginx_connections_reading`	Gauge	Connections reading requests
`nginx_connections_writing`	Gauge	Connections writing responses
`nginx_connections_waiting`	Gauge	Idle waiting connections
`nginx_connections_accepted_total`	Counter	Total accepted connections
`nginx_connections_handled_total`	Counter	Total handled connections

Example query to check exporter status:

curl http://rproxy-host:9113/metrics | grep nginx_up

Verification

1. Check Service Status

On rproxy host:

systemctl status nginx-prometheus-exporter

Expected output:

● nginx-prometheus-exporter.service - Nginx Prometheus Exporter
     Loaded: loaded (/etc/systemd/system/nginx-prometheus-exporter.service; enabled; vendor preset: enabled)
     Active: active (running)

2. Verify Metrics Endpoint

curl -s http://localhost:9113/metrics | head -20

Expected output (sample):

# HELP nginx_up Whether the Nginx server is up
# TYPE nginx_up gauge
nginx_up 1
# HELP nginx_requests_total Total number of requests
# TYPE nginx_requests_total counter
nginx_requests_total 1234567
# HELP nginx_connections_active Active connections
# TYPE nginx_connections_active gauge
nginx_connections_active 42

3. Verify Local Status Endpoint

curl -s http://127.0.0.1:9114/stub_status

Expected output:

Active connections: 42
server accepts handled requests
 1234567 1234567 1234567
Reading: 2 Writing: 5 Waiting: 35

4. Verify Nginx Status Vhost

curl -I http://127.0.0.1:9114/

Expected output:

HTTP/1.1 200 OK
Server: nginx/1.18.0
Content-Type: text/plain

5. Check Prometheus Targets

Access Prometheus web UI: http://prometheus-host:9090/targets

Look for nginx_exporter job. Expected state: UP

6. Query Prometheus

In Prometheus query interface, run:

nginx_up

Expected result: Returns 1 for each exporter target

Troubleshooting

Issue: Service Fails to Start

Symptom: systemctl status nginx-prometheus-exporter shows failed state

Check logs:

journalctl -u nginx-prometheus-exporter -n 50

Common causes:

Port already in use — Change nginx_prometheus_exporter_setup_listen_address in inventory
Binary download failed — Check internet connectivity and GitHub releases availability

Permissions issue — Ensure service user exists with correct permissions:

getent passwd nginx-prometheus-exporter
ls -la /usr/local/bin/nginx-prometheus-exporter

Issue: Cannot Connect to Metrics Endpoint

Symptom: curl: (7) Failed to connect to localhost port 9113

Check if service is running:

netstat -tlnp | grep 9113

Check service status:

systemctl status nginx-prometheus-exporter

Manual test of exporter:

sudo -u nginx-prometheus-exporter /usr/local/bin/nginx-prometheus-exporter \
  -nginx.scrape-uri=http://127.0.0.1:9114/stub_status

Issue: Metrics Show 0 or Missing Data

Symptom: Prometheus scrapes successfully but metrics are zeros or incomplete

Check local Nginx status endpoint:

curl -s http://127.0.0.1:9114/stub_status

If no output, check:

Nginx status vhost enabled:

ls -la /etc/nginx/sites-enabled/nginx-prometheus-exporter-status.conf

Nginx configuration valid:
```
nginx -t
```
Nginx reloaded after status vhost setup:
```
systemctl reload nginx
```

Issue: Prometheus Shows Target as DOWN

Symptom: Prometheus targets page shows nginx_exporter job as RED (DOWN)

Check exporter is accessible:

curl -v http://rproxy-host:9113/metrics

Check Prometheus logs:

journalctl -u prometheus -n 50

Verify firewall rules:

sudo ufw status | grep 9113

If blocked, allow port:

sudo ufw allow 9113/tcp

Issue: High CPU Usage from Exporter

Symptom: nginx-prometheus-exporter process consuming excessive CPU

Check exporter logs for errors:

journalctl -u nginx-prometheus-exporter -f

Reduce scrape frequency in Prometheus config:

scrape_interval: 15s  # Increase from 5s to 15s

Then redeploy Prometheus.

Issue: Exporter Exits Immediately After Start

Symptom: Service starts then stops within seconds

Check for startup errors:

/usr/local/bin/nginx-prometheus-exporter \
  -nginx.scrape-uri=http://127.0.0.1:9114/stub_status

Common causes:

Invalid scrape URI — Verify stub_status endpoint is accessible

Binary incompatibility — Check CPU architecture:

file /usr/local/bin/nginx-prometheus-exporter
uname -m

Missing dependencies — Check required libraries:
```
ldd /usr/local/bin/nginx-prometheus-exporter
```

Summary

The Nginx Prometheus monitoring solution provides comprehensive visibility into reverse proxy performance through:

Automated deployment via Ansible playbooks
Real-time metrics from nginx-prometheus-exporter
Centralized storage in Prometheus
Dashboard visualization in Grafana

Use this documentation to deploy, configure, and troubleshoot Nginx monitoring.

📊 Nginx Reverse Proxy Monitoring

Table of Contents

Overview

Architecture

Component Interaction

Data Flow

Components

1. nginx-prometheus-exporter

2. Local Nginx Status Vhost

3. Prometheus Configuration

Deployment

Prerequisites

Deployment Chain

How to Deploy

Configuration

Inventory Configuration

Role Variables

Ansible Playbook

Main Playbook: deploy_rproxy.yml

Role Execution

Role Tasks

Scraping Metrics

Prometheus Scrape Configuration

How Targets are Passed to Prometheus

Exported Metrics

Verification

1. Check Service Status

2. Verify Metrics Endpoint

3. Verify Local Status Endpoint

4. Verify Nginx Status Vhost

5. Check Prometheus Targets

6. Query Prometheus

Troubleshooting

Issue: Service Fails to Start

Issue: Cannot Connect to Metrics Endpoint

Issue: Metrics Show 0 or Missing Data

Issue: Prometheus Shows Target as DOWN

Issue: High CPU Usage from Exporter

Issue: Exporter Exits Immediately After Start

Summary