Monitoring Kubernetes with Prometheus: A Practical Guide
August 5, 2026•CloudCops
The first time this usually shows up, it's a pager at an ugly hour. The cluster looked healthy the day before, the deployment was fine, and then a node starts evicting pods, a rollout loops, or the API server gets sluggish enough that everything downstream feels broken. If the monitoring stack only tells you that “Prometheus is up,” you're already in the blind spot.
Monitoring Kubernetes with Prometheus works when it sees the cluster the way Kubernetes behaves in production, not the way a dashboard screenshot makes it look. That means watching the node, the orchestration layer, and the control plane together, then wiring the alerts so they point to a real operational problem instead of another noisy chart. In practice, the gap isn't the install, it's the assumptions you make after the install.
Why Your Monitoring Stack Is the Blind Spot
The first bad incident is usually the one that exposes the whole monitoring posture. A team ships a workload, scales it out, and assumes the stack is covered because Prometheus and Grafana are installed. Then a node runs short on disk, pods start getting evicted, and the only signal anyone has is a downstream application error that appears much later. That's not observability, it's a delayed confession.
Kubernetes makes this failure mode easy to create because the thing you're watching is constantly changing. Pods disappear, nodes get replaced, and the control plane becomes part of the system you need to trust. Prometheus becomes useful only when it scrapes multiple layers of telemetry, not when it stares at container health in isolation. The operational reality is that a single-pane dashboard can look polished and still miss the condition that caused the outage.
Practical rule: if your alert can't distinguish node pressure from workload pressure, it's too vague to be useful.
The useful mental model is simple. Node-level signals tell you whether the infrastructure has room left. Orchestration-layer signals tell you whether Kubernetes is scheduling, rolling out, and reporting object state the way you expect. Control-plane metrics tell you whether the cluster itself is lagging, stalling, or becoming the bottleneck. Prometheus's Kubernetes monitoring pattern is built around scraping exactly those three layers, and that architecture matters because it lets operators correlate saturation with workload behavior instead of watching container health alone. Sysdig's Kubernetes monitoring guide lays out that layered pattern clearly.
A lot of teams discover this late, after they've built dashboards around one layer and then wondered why the page arrived too late. The better pattern is to treat the monitoring stack as part of the platform, not a bolt-on. If you already have a rough operational baseline and want a second pass on common mistakes, this Kubernetes monitoring best practices guide is a useful companion.
The Three-Layer Architecture Behind Effective Kubernetes Monitoring
Prometheus works on Kubernetes because Kubernetes produces state in layers, and each layer answers a different question. Node metrics tell you whether the worker machines are under pressure. Orchestration metrics tell you whether Kubernetes is placing and tracking workloads correctly. Control-plane metrics tell you whether the cluster's brain is slowing down. If you flatten those layers into one generic “health” view, you lose the signal that matters when the incident starts.
Node metrics answer capacity questions
Node-level telemetry covers CPU, memory, disk, and load on each worker. These are the signals that tell you whether the infrastructure is about to stop carrying the workload. They don't tell you whether your application is healthy, but they do tell you whether the platform can keep up. Prometheus is commonly paired with node-oriented exporters so those signals stay visible even when pods churn.
Orchestration metrics answer state questions
The orchestration layer is where deployment state, resource requests, scheduling, and API server latency live. kube-state-metrics becomes essential here, because it exposes Kubernetes object state as Prometheus metrics without needing application code changes. In practice, that's often the difference between seeing a rollout go bad and guessing after the fact. The Grafana Kubernetes cluster monitoring dashboard reflects that layered approach, and it's a good reminder that cluster monitoring has to include object state, not just node charts.
Control-plane metrics answer platform questions
The scheduler, controller manager, and DNS service all have their own behavior, and they can become the source of failure even when workloads look normal. Kubernetes' metrics reference confirms that component metrics are exposed over HTTP in Prometheus format and can be scraped directly. That matters because a stuck controller can look like a workload bug until you inspect the control plane itself. In production, teams often use dashboards that combine cluster CPU, memory, filesystem, network I/O, and pod or container usage so those layers sit side by side instead of in separate islands. CloudOrbis Inc. monitoring tools is a sensible example of a monitoring context where that sort of operational visibility matters, especially when a platform spans more than one environment.
A dashboard is only honest when it can tell you whether the problem is the node, the rollout, or the cluster itself.
Installing the kube-prometheus-stack with Helm
The stack many reach for first is kube-prometheus-stack, because it bundles Prometheus, Alertmanager, Grafana, the Prometheus Operator, and kube-state-metrics into one release. That packaging is valuable, but the actual production decision is not the Helm chart itself. It's whether Prometheus has the permissions and namespace visibility it needs to discover targets across the cluster. Without that, you get an install that looks complete and behaves blind.
A solid workflow starts with a dedicated monitoring namespace, then the Helm repo, then the stack install. The important bit is granting cluster-wide read-only RBAC before you trust any scrape results. Prometheus needs discovery permissions across Kubernetes resources to find targets reliably, and the usual failure mode is a missing ClusterRole or ClusterRoleBinding that leaves whole namespaces invisible. The Prometheus Helm chart guide is a useful reference if you want the chart mechanics without the fluff.
A representative install flow looks like this:
- Create the namespace.
- Add the Prometheus community Helm repo.
- Install
kube-prometheus-stackinto that namespace. - Confirm the chart created the expected operators, exporters, and dashboards.
- Check that
ServiceMonitorobjects are being picked up.
The verification matters more than the install command. If targets don't show up in the Prometheus UI, that's usually not a Grafana problem. It's often a discovery problem, and discovery problems usually come from namespace-limited visibility or missing RBAC. Teams also get tripped up by exposing Prometheus directly, when a safer internal service or port-forwarding path is enough for validation. The practical habit is to check whether the pods and services are healthy before you blame the queries.
The Cyber Command guide on deploying to AKS using DevOps pipelines fits naturally here if your platform is already wired around CI/CD. It helps put the monitoring release into the same delivery discipline as the rest of the cluster, which is where it belongs.
Once the release lands, the video below is a useful reference for the hands-on install flow and the usual validation steps.
After that, check the Prometheus targets page and the ServiceMonitor objects together. If one exists without the other, the stack is installed but not operational.
Configuring Scraping Targets and Writing PromQL Queries
Once the stack is running, the work starts. Kubernetes does not give you a tidy list of stable services to monitor. It gives you ephemeral pods, shifting endpoints, and labels that only matter if scrape configuration keeps up. That is why ServiceMonitor CRDs matter more than static scrape configs in production, they let Prometheus follow services as they move.
kube-state-metrics is the anchor for object state, and it is often scraped on port 8080. node-exporter handles per-node signals. cAdvisor gives you container-level resource usage. Custom exporters fill the gaps for application-specific metrics when the built-in views are not enough. The point is not to scrape everything, it is to scrape the signals that help you make the next decision during an incident. For a practical setup guide, CloudCops' kube-state-metrics guide is useful when you are wiring object-state metrics for the first time.
PromQL patterns that earn their keep
The thresholds that matter in production are not decorative. PromQL examples often track CPU above 80%, memory above 85% of limit, pod restarts above 5 in 1 hour, node disk below 10% free, and API server latency above 1 second at p99. Those values reflect a mature operational baseline, not dashboard theater, and they are the kinds of thresholds that let Alertmanager page for symptoms that point to a real failure mode. Sysdig's Kubernetes monitoring article is a clear public reference for those practical patterns.
A few query habits matter more than fancy PromQL tricks:
- Track request and limit context: raw CPU use is less helpful than CPU compared to request or limit.
- Watch restart behavior over time: a single restart does not mean much, repeated restarts do.
- Separate node pressure from pod pressure: those are different incident classes.
- Prefer p99 for latency-sensitive components: averages hide ugly tails.
- Route only actionable alerts: Alertmanager should tell someone what broke, not just that a metric crossed a line.
Alertmanager becomes useful only when you keep the route tree sane. Group related alerts, keep warning and critical paths separate, and avoid turning every scrape anomaly into a page. I have seen more damage from alert fatigue than from missing dashboards, because teams stop trusting the notifications long before the next real incident.
If you want a practical baseline, start with the cluster-state alerts first, then add service-specific rules. That sequencing keeps you from drowning in custom metrics before the platform-level issues are covered.
The Cyndra article on turning dashboard insights into actions fits this work too, because the hardest part is not seeing a chart, it is making the chart change an operator's next step.
Grafana Dashboards and Production Hardening
Prometheus stores the data, but Grafana is where operators live during the incident. The default dashboards in kube-prometheus-stack are useful, yet they're only a starting point. Production dashboards need to reflect the three telemetry layers, because that's how people debug a cluster at 3 AM. A dashboard that mixes pod restarts, node pressure, and API server latency is far more actionable than a wall of charts grouped by exporter.
The best dashboards I've seen don't try to be complete. They try to answer three questions fast. Is the node healthy, is the workload rolling correctly, and is the control plane responding normally? Once those are visible, the rest becomes inspection, not discovery. The Cyndra article on turning dashboard insights into actions is relevant here because the hardest part isn't seeing a chart, it's making the chart change an operator's next step.
Production monitoring checklist
| Area | Decision | Common Mistake |
|---|---|---|
| Storage | Use long-term retention strategy or remote storage when history matters | Keeping everything on the local Prometheus volume forever |
| High availability | Add Thanos or another HA pattern when single-instance Prometheus is too fragile | Treating one Prometheus pod as a resilient platform |
| Access control | Keep the Prometheus API behind RBAC and internal access paths | Exposing the UI directly to the internet |
| Network boundaries | Restrict scrape and UI access with network policies | Letting every namespace talk to everything |
| Retention and queries | Tune retention and query timeouts to match cluster size | Leaving defaults in place and hoping the storage keeps up |
| Metrics design | Watch for high-cardinality labels before they pile up | Adding labels that explode time-series count |
Those hardening choices separate a demo from a platform. Thanos is the common answer when teams need long-term storage or a more resilient Prometheus layer, and remote_write to a managed time-series backend is another viable path when you want to push storage concerns elsewhere. Either way, the decision comes down to how long you need historical data and how much operational burden you want to carry in-cluster.
Security needs the same discipline. RBAC on the Prometheus API, internal service exposure, and network policies around scrape access are not optional once the stack becomes part of production. Cost control matters too, because retention and cardinality both turn into real storage pressure if nobody watches them. The monitoring platform should reduce incident time, not create a second cluster to babysit.
Troubleshooting Playbook and Key Takeaways
When the stack misbehaves, the fastest path is symptom-first diagnosis. Scrape failures, missing metrics, noisy alerts, and empty dashboards each point to different layers of the setup. A common mistake is jumping straight to Grafana when the problem is really target discovery, permissions, or a broken exporter.
A fast diagnostic path
- Scrape failures: Check
ServiceMonitorlabels, confirm the namespace selector is correct, verify RBAC, and make sure Prometheus can reach the endpoint from its own pod. - Missing metrics: Confirm kube-state-metrics is running and exposing the expected paths on port 8080, then verify any custom exporter has a matching
ServiceMonitorselector. - Alert fatigue: Review recording rules, look for noisy label combinations, and tighten the evaluation strategy before adding more notifications.
- Dashboard gaps: Confirm the Grafana datasource points at the right Prometheus service URL, then verify dashboard variables match the namespace and pod labels in your cluster.
The exact failure pattern matters because each one usually has a different root cause. A scrape failure often means registration or access, not a broken application. Missing metrics often means the exporter isn't where you think it is. An empty dashboard sometimes means the datasource is fine but the variables are filtering everything out.
If you only remember one thing, remember this. A working install is not the same thing as a monitoring platform.
The practical takeaways are consistent across clusters. Architecture comes before installation. RBAC comes before scraping. ServiceMonitor beats static configs in a dynamic cluster. Threshold baselines beat arbitrary dashboards. Hardening comes before production traffic. Those five habits are what keep Prometheus useful after the honeymoon phase ends.
Monitoring Kubernetes with Prometheus works when it's treated like production infrastructure, not like a checkbox. If you want help turning a brittle stack into something that reduces pages, CloudCops GmbH designs and runs cloud-native observability platforms with Prometheus, Grafana, Thanos, and Kubernetes in the loop. Visit CloudCops GmbH if you want to compare your current setup against a production-grade operating model.
Ready to scale your cloud infrastructure?
Let's discuss how CloudCops can help you build secure, scalable, and modern DevOps workflows. Schedule a free discovery call today.
Continue Reading
Cloud Service Monitoring: From Alerts to Observability
Master cloud service monitoring. This guide explains telemetry, observability patterns, modern tooling like Prometheus, and how to lower MTTD/MTTR.
Performance Benchmarking: A Cloud-Native Playbook
A step-by-step guide to performance benchmarking for cloud-native platforms. Learn to define goals, select KPIs, automate tests in CI, and analyze results.
What Is Continuous Monitoring: Cloud Security 2026
What is continuous monitoring in cloud-native environments? Understand its critical role in security & compliance, and how to implement it with modern tools.