What is Prometheus?
Prometheus is a monitoring tool. Every few seconds it visits each server, reads its numbers (CPU, memory, disk, requests…) and saves them with a timestamp. You can then search those numbers, draw graphs and get alerts when something looks wrong.
- Scraping. Prometheus pulls the numbers itself, by reading a web page on each target.
- Exporters. Small programs that put a system's numbers on that web page: node_exporter for the VM, mysqld_exporter for MySQL, the nginx exporter for nginx.
- PromQL. The query language you use to ask questions, like "is every target up?".
The lab
The expense app runs on three VMs (frontend, backend and MySQL) behind an ALB. A separate monitoring VM runs Prometheus and watches all of them.
The plan: install everything by hand first so you understand every file, then automate it in the next version.
Order of work:
- Install Prometheus with its default config. It watches only itself (this page, sections 1–3).
- Install the exporters: node_exporter, mysqld_exporter and the nginx exporter.
- Come back and add the exporters as targets (section 4).
| Install on | Port | Version | Download source | Runs as user |
|---|---|---|---|---|
| monitoring VM | 9090 | 3.16.0 | prometheus.io/download | prometheus |
Where the files go
- Program: a versioned folder like
/opt/prometheus-3.16.0/. - Symlink:
/opt/prometheusalways points to the current version. systemd uses this path. - Config:
/etc/prometheus/. - Service file:
/etc/systemd/system/.
To upgrade, you unpack the new version, point the symlink at it and restart. Before you start, check the download page: if there is a newer version, change only the VERSION= line.
1. Before you start
1.1 Check the CPU type
Prometheus is built separately for each kind of CPU, so first check which one your VM has:
uname -m
| Output | Download the file with |
|---|---|
x86_64 (t3.micro) | linux-amd64 |
aarch64 (t4g / Graviton) | linux-arm64 |
Never download darwin: that is the macOS build. If you do, systemd fails with Exec format error.
1.2 Become root and work in /opt
sudo -i
cd /opt
Everything is downloaded and unpacked straight into /opt. The download file is deleted afterwards, so /opt only keeps the versioned folders and the symlinks.
2. Install Prometheus (monitoring VM only)
Prometheus starts with its default config, which only watches itself. The exporters are added later, in section 4.
2.1 Download and unpack
cd /opt
VERSION=3.16.0
curl -fLO https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-${VERSION}.linux-amd64.tar.gz
tar -xzf prometheus-${VERSION}.linux-amd64.tar.gz
Check that the file name says linux-amd64. A darwin file is the macOS build and fails with Exec format error.
2.2 Rename, symlink and clean up
Give the folder a simple versioned name, point /opt/prometheus at it, and delete the download:
mv /opt/prometheus-${VERSION}.linux-amd64 /opt/prometheus-${VERSION}
rm -f /opt/prometheus-${VERSION}.linux-amd64.tar.gz
ln -sfn /opt/prometheus-${VERSION} /opt/prometheus
ls /opt/prometheus/
/opt/prometheus/prometheus --version
The folder holds prometheus (the server), promtool (checks config files for mistakes) and a sample prometheus.yml.
2.3 Create the user and folders
Prometheus runs as its own system user, with no login shell. The first line only creates the user if it doesn't exist yet.
id prometheus || useradd --system --no-create-home --shell /sbin/nologin prometheus
mkdir -p /etc/prometheus /var/lib/prometheus
cp /opt/prometheus/prometheus.yml /etc/prometheus/prometheus.yml
chown -R prometheus:prometheus /opt/prometheus-${VERSION} /etc/prometheus /var/lib/prometheus
| Path | Holds |
|---|---|
/opt/prometheus | Only the programs (replaced on upgrade) |
/etc/prometheus/prometheus.yml | The live config: the only file you edit |
/var/lib/prometheus | The collected metrics (kept during upgrades) |
Why copy the config to /etc? An upgrade replaces the /opt/prometheus-<version> folder, so any edits there would be left behind in the old folder. And the service only reads /etc/prometheus/prometheus.yml, so editing /opt/prometheus/prometheus.yml changes nothing.
2.4 Look at the default config
cat /etc/prometheus/prometheus.yml
It has one job, prometheus, which scrapes localhost:9090 (Prometheus itself). Leave it as it is for now.
2.5 Create the systemd service
The service file makes Prometheus start on boot and restart if it crashes. Note that ExecStart uses the symlink, so it never changes when you upgrade.
cat > /etc/systemd/system/prometheus.service <<'UNIT'
[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=15d \
--web.listen-address=:9090 \
--web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
UNIT
| Flag | Meaning |
|---|---|
--config.file | Which config file is live |
--storage.tsdb.path | Where metrics are stored on disk |
--storage.tsdb.retention.time=15d | Delete data older than 15 days |
--web.listen-address=:9090 | The port for the web UI and API |
--web.enable-lifecycle | Lets curl -X POST .../-/reload reload the config without a restart |
2.6 Check the config, then start
/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml
systemctl daemon-reload
systemctl enable --now prometheus
systemctl status prometheus --no-pager
If the status isn't active (running), read the error before changing anything:
journalctl -u prometheus -n 50 --no-pager
2.7 Verify
ss -tlnp | grep 9090
curl -s http://localhost:9090/-/healthy
curl -s 'http://localhost:9090/api/v1/query?query=up'
ssshows Prometheus listening on port 9090./-/healthyanswers when the server is healthy.- The last command runs the query
upthrough the API.
3. Open the port in AWS
To reach the web UI from your browser, allow port 9090 in the monitoring VM's security group, but only from your IP:
| Security group | Inbound port | Source |
|---|---|---|
| monitoring SG | 9090 (Prometheus UI) | your IP only |
Open http://<monitoring-public-ip>:9090. Status → Target health shows exactly one target: prometheus, UP.
Stop here and install the exporters (node_exporter, mysqld_exporter, nginx exporter). Come back to section 4 once all of them are running.
4. Add the exporters as targets
Do this only when every exporter answers curl from the monitoring VM. Until now, Prometheus has only watched itself.
4.1 Back up the default config
cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.default
4.2 Write the new config
The config is written with a heredoc instead of pasting into vim, because pasting can break YAML indentation.
cat > /etc/prometheus/prometheus.yml <<'YAML'
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node"
static_configs:
- targets: ["frontend.daws90s.shop:9100"]
labels:
tier: "frontend"
- targets: ["backend.daws90s.shop:9100"]
labels:
tier: "backend"
- targets: ["mysql.daws90s.shop:9100"]
labels:
tier: "mysql"
- job_name: "nginx"
static_configs:
- targets: ["frontend.daws90s.shop:9113"]
labels:
tier: "frontend"
- job_name: "mysql"
static_configs:
- targets: ["mysql.daws90s.shop:9104"]
labels:
tier: "mysql"
YAML
chown prometheus:prometheus /etc/prometheus/prometheus.yml
| Job | Targets | What it scrapes |
|---|---|---|
| prometheus | localhost:9090 | Prometheus itself (the default job) |
| node | 3 VMs on 9100 | node_exporter, with a tier label per VM |
| nginx | frontend:9113 | nginx-prometheus-exporter |
| mysql | mysql:9104 | mysqld_exporter |
The tier label is added to every metric from that target, so you can filter by it later, like up{tier="mysql"}.
4.3 Check and reload
/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload
- The reload works because Prometheus runs with
--web.enable-lifecycle. Without it, usesystemctl reload prometheusorsystemctl restart prometheus. - If
promtoolreports an error, fix it first. A reload with a broken config is rejected and Prometheus keeps the old one. Checkjournalctl -u prometheus -n 20.
5. Verify end to end
In the UI (http://<monitoring-ip>:9090):
- Status → Target health: every target shows UP.
- Run these queries in the Query tab:
| Query | Expected |
|---|---|
up | 1 for every target |
up{job="node"} | 3 series, one per tier |
mysql_up | 1 |
nginx_up | 1 |
node_load1 | a small number per VM |
rate(nginx_http_requests_total[5m]) | more than 0 after you open the app in a browser |
6. Troubleshooting
| Symptom | Likely cause | Check |
|---|---|---|
Exec format error in journalctl | Wrong build (darwin or arm64) | The file name must contain linux-amd64; download again |
status=203/EXEC | Wrong path in ExecStart, or the program isn't executable | ls -l /opt/<tool>/, check the symlink |
status=217/USER | The service user doesn't exist | id <user> |
bind: address already in use | Another program is using the port | ss -tlnp | grep <port> |
Target DOWN, context deadline exceeded | Security group or VPC blocks the port | curl from the monitoring VM hangs |
Target DOWN, connection refused | Exporter not running on that VM | systemctl status <service> on that VM |
| Reload rejected | YAML error in the config | /opt/prometheus/promtool check config /etc/prometheus/prometheus.yml |
| Config change has no effect | Edited the wrong file | systemctl cat prometheus | grep config.file |
Rule: read the error first. Don't add -q or -s to commands while debugging, because they hide the messages you need.
7. Upgrading later
This is where the symlink pays off. Unpack the new version next to the old one, point the symlink at it and restart:
cd /opt
VERSION=<new-version>
# download + extract + rename to /opt/prometheus-${VERSION}, delete tarball (as in 2.1–2.2)
chown -R prometheus:prometheus /opt/prometheus-${VERSION}
ln -sfn /opt/prometheus-${VERSION} /opt/prometheus
systemctl restart prometheus
- Rollback: point the symlink back to the old folder and restart. No changes to the service file.
- The config in
/etc/prometheusand the data in/var/lib/prometheusare not touched by an upgrade.