Prometheus Installation and Setup

What is Prometheus?

Prometheus is a monitoring tool. Every few seconds it visits each server, reads its numbers (CPU, memory, disk, requests…) and saves them with a timestamp. You can then search those numbers, draw graphs and get alerts when something looks wrong.

  • Scraping. Prometheus pulls the numbers itself, by reading a web page on each target.
  • Exporters. Small programs that put a system's numbers on that web page: node_exporter for the VM, mysqld_exporter for MySQL, the nginx exporter for nginx.
  • PromQL. The query language you use to ask questions, like "is every target up?".

The lab

The expense app runs on three VMs (frontend, backend and MySQL) behind an ALB. A separate monitoring VM runs Prometheus and watches all of them.

You open the Prometheus UI on port 9090 of the monitoring VM. Prometheus pulls metrics every 15 seconds from node_exporter (9100) on the frontend, backend and MySQL VMs, from the nginx exporter (9113) on frontend, and from mysqld_exporter (9104) on MySQL. Browser (you) your IP only :9090 monitoring VM Prometheus 3.16.0 port 9090 scrapes every 15s GET frontend VM (nginx) node_exporter :9100 nginx exporter :9113 backend VM node_exporter :9100 MySQL VM node_exporter :9100 mysqld_exporter :9104

The plan: install everything by hand first so you understand every file, then automate it in the next version.

Order of work:

  1. Install Prometheus with its default config. It watches only itself (this page, sections 1–3).
  2. Install the exporters: node_exporter, mysqld_exporter and the nginx exporter.
  3. Come back and add the exporters as targets (section 4).
Install onPortVersionDownload sourceRuns as user
monitoring VM90903.16.0prometheus.io/downloadprometheus

Where the files go

  • Program: a versioned folder like /opt/prometheus-3.16.0/.
  • Symlink: /opt/prometheus always points to the current version. systemd uses this path.
  • Config: /etc/prometheus/.
  • Service file: /etc/systemd/system/.

To upgrade, you unpack the new version, point the symlink at it and restart. Before you start, check the download page: if there is a newer version, change only the VERSION= line.


1. Before you start

1.1 Check the CPU type

Prometheus is built separately for each kind of CPU, so first check which one your VM has:

uname -m
OutputDownload the file with
x86_64 (t3.micro)linux-amd64
aarch64 (t4g / Graviton)linux-arm64

Never download darwin: that is the macOS build. If you do, systemd fails with Exec format error.

1.2 Become root and work in /opt

sudo -i
cd /opt

Everything is downloaded and unpacked straight into /opt. The download file is deleted afterwards, so /opt only keeps the versioned folders and the symlinks.


2. Install Prometheus (monitoring VM only)

Prometheus starts with its default config, which only watches itself. The exporters are added later, in section 4.

2.1 Download and unpack

cd /opt
VERSION=3.16.0
curl -fLO https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-${VERSION}.linux-amd64.tar.gz
tar -xzf prometheus-${VERSION}.linux-amd64.tar.gz

Check that the file name says linux-amd64. A darwin file is the macOS build and fails with Exec format error.

2.2 Rename, symlink and clean up

Give the folder a simple versioned name, point /opt/prometheus at it, and delete the download:

mv /opt/prometheus-${VERSION}.linux-amd64 /opt/prometheus-${VERSION}
rm -f /opt/prometheus-${VERSION}.linux-amd64.tar.gz
ln -sfn /opt/prometheus-${VERSION} /opt/prometheus
ls /opt/prometheus/
/opt/prometheus/prometheus --version

The folder holds prometheus (the server), promtool (checks config files for mistakes) and a sample prometheus.yml.

2.3 Create the user and folders

Prometheus runs as its own system user, with no login shell. The first line only creates the user if it doesn't exist yet.

id prometheus || useradd --system --no-create-home --shell /sbin/nologin prometheus
mkdir -p /etc/prometheus /var/lib/prometheus
cp /opt/prometheus/prometheus.yml /etc/prometheus/prometheus.yml
chown -R prometheus:prometheus /opt/prometheus-${VERSION} /etc/prometheus /var/lib/prometheus
PathHolds
/opt/prometheusOnly the programs (replaced on upgrade)
/etc/prometheus/prometheus.ymlThe live config: the only file you edit
/var/lib/prometheusThe collected metrics (kept during upgrades)

Why copy the config to /etc? An upgrade replaces the /opt/prometheus-<version> folder, so any edits there would be left behind in the old folder. And the service only reads /etc/prometheus/prometheus.yml, so editing /opt/prometheus/prometheus.yml changes nothing.

2.4 Look at the default config

cat /etc/prometheus/prometheus.yml

It has one job, prometheus, which scrapes localhost:9090 (Prometheus itself). Leave it as it is for now.

2.5 Create the systemd service

The service file makes Prometheus start on boot and restart if it crashes. Note that ExecStart uses the symlink, so it never changes when you upgrade.

cat > /etc/systemd/system/prometheus.service <<'UNIT'
[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.listen-address=:9090 \
  --web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
UNIT
FlagMeaning
--config.fileWhich config file is live
--storage.tsdb.pathWhere metrics are stored on disk
--storage.tsdb.retention.time=15dDelete data older than 15 days
--web.listen-address=:9090The port for the web UI and API
--web.enable-lifecycleLets curl -X POST .../-/reload reload the config without a restart

2.6 Check the config, then start

/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml
systemctl daemon-reload
systemctl enable --now prometheus
systemctl status prometheus --no-pager

If the status isn't active (running), read the error before changing anything:

journalctl -u prometheus -n 50 --no-pager

2.7 Verify

ss -tlnp | grep 9090
curl -s http://localhost:9090/-/healthy
curl -s 'http://localhost:9090/api/v1/query?query=up'
  • ss shows Prometheus listening on port 9090.
  • /-/healthy answers when the server is healthy.
  • The last command runs the query up through the API.

3. Open the port in AWS

To reach the web UI from your browser, allow port 9090 in the monitoring VM's security group, but only from your IP:

Security groupInbound portSource
monitoring SG9090 (Prometheus UI)your IP only

Open http://<monitoring-public-ip>:9090. Status → Target health shows exactly one target: prometheus, UP.

Stop here and install the exporters (node_exporter, mysqld_exporter, nginx exporter). Come back to section 4 once all of them are running.


4. Add the exporters as targets

Do this only when every exporter answers curl from the monitoring VM. Until now, Prometheus has only watched itself.

4.1 Back up the default config

cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.default

4.2 Write the new config

The config is written with a heredoc instead of pasting into vim, because pasting can break YAML indentation.

cat > /etc/prometheus/prometheus.yml <<'YAML'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["frontend.daws90s.shop:9100"]
        labels:
          tier: "frontend"
      - targets: ["backend.daws90s.shop:9100"]
        labels:
          tier: "backend"
      - targets: ["mysql.daws90s.shop:9100"]
        labels:
          tier: "mysql"

  - job_name: "nginx"
    static_configs:
      - targets: ["frontend.daws90s.shop:9113"]
        labels:
          tier: "frontend"

  - job_name: "mysql"
    static_configs:
      - targets: ["mysql.daws90s.shop:9104"]
        labels:
          tier: "mysql"
YAML
chown prometheus:prometheus /etc/prometheus/prometheus.yml
JobTargetsWhat it scrapes
prometheuslocalhost:9090Prometheus itself (the default job)
node3 VMs on 9100node_exporter, with a tier label per VM
nginxfrontend:9113nginx-prometheus-exporter
mysqlmysql:9104mysqld_exporter

The tier label is added to every metric from that target, so you can filter by it later, like up{tier="mysql"}.

4.3 Check and reload

/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload
  • The reload works because Prometheus runs with --web.enable-lifecycle. Without it, use systemctl reload prometheus or systemctl restart prometheus.
  • If promtool reports an error, fix it first. A reload with a broken config is rejected and Prometheus keeps the old one. Check journalctl -u prometheus -n 20.

5. Verify end to end

In the UI (http://<monitoring-ip>:9090):

  1. Status → Target health: every target shows UP.
  2. Run these queries in the Query tab:
QueryExpected
up1 for every target
up{job="node"}3 series, one per tier
mysql_up1
nginx_up1
node_load1a small number per VM
rate(nginx_http_requests_total[5m])more than 0 after you open the app in a browser

6. Troubleshooting

SymptomLikely causeCheck
Exec format error in journalctlWrong build (darwin or arm64)The file name must contain linux-amd64; download again
status=203/EXECWrong path in ExecStart, or the program isn't executablels -l /opt/<tool>/, check the symlink
status=217/USERThe service user doesn't existid <user>
bind: address already in useAnother program is using the portss -tlnp | grep <port>
Target DOWN, context deadline exceededSecurity group or VPC blocks the portcurl from the monitoring VM hangs
Target DOWN, connection refusedExporter not running on that VMsystemctl status <service> on that VM
Reload rejectedYAML error in the config/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml
Config change has no effectEdited the wrong filesystemctl cat prometheus | grep config.file

Rule: read the error first. Don't add -q or -s to commands while debugging, because they hide the messages you need.


7. Upgrading later

This is where the symlink pays off. Unpack the new version next to the old one, point the symlink at it and restart:

cd /opt
VERSION=<new-version>
# download + extract + rename to /opt/prometheus-${VERSION}, delete tarball (as in 2.1–2.2)
chown -R prometheus:prometheus /opt/prometheus-${VERSION}
ln -sfn /opt/prometheus-${VERSION} /opt/prometheus
systemctl restart prometheus
  • Rollback: point the symlink back to the old folder and restart. No changes to the service file.
  • The config in /etc/prometheus and the data in /var/lib/prometheus are not touched by an upgrade.
← Back to portfolio