A reference design for watching a set of services that run on Amazon ECS. It is the starting point we adapt for a client, not a finished product.
The problem
Teams often have plenty of data and no clear picture. Dashboards nobody opens, alerts that fire all day, and a pager that people learn to ignore. The goal here is a small set of views and alerts that people trust.
How it works
- Each service exposes a
/metricsendpoint. - Prometheus runs as an ECS service and scrapes every service it can discover.
- Alertmanager receives alerts from Prometheus, groups them and routes them.
- Alerts that need a person go to SNS, which reaches whoever is on call.
- Grafana reads from Prometheus and shows the dashboards.
Decision log
| Decision | Why |
|---|---|
| Run Prometheus on ECS | The team already runs ECS, so there is one fewer platform to learn. |
| Persistent storage for Prometheus data | A restarted task should not lose its history. |
| Alert on symptoms first | Users feel slow pages and errors. Causes go on dashboards, not pagers. |
| One runbook page per alert | Whoever is paged knows what it means and what to try first. |
| Send CloudWatch and Prometheus alerts to the same place | One on-call path, not two. |
Rough cost shape
We do not quote numbers here, because they depend on how many services you run and how long you keep data. The main drivers are:
- The compute for Prometheus and Grafana, which is small and steady.
- Storage for metrics, which grows with the number of series and the retention period.
- Data transfer, if services sit in several Availability Zones.
Confirm any estimate in the AWS pricing calculator before you commit.
Things we check on every build
- Alerts are reviewed after the first month, and noisy ones are removed.
- Dashboards each answer a named question.
- Access to Grafana goes through your existing identity provider where possible.