A Guide to Prometheus for Kubernetes

If you're running workloads on Kubernetes, you need a monitoring solution that was born in the same cloud-native world. That solution is Prometheus. Using Prometheus for Kubernetes monitoring isn't just a popular choice; it has become the industry standard for a reason. It’s built to handle the dynamic, often chaotic nature of containerized environments where pods and services appear and disappear constantly.

Why Prometheus Is the Gold Standard for Kubernetes

It’s no accident that Prometheus and Kubernetes work so well together. Both projects originated in the Cloud Native Computing Foundation (CNCF) and share a common DNA. They are core components of a modern stack, and understanding how to properly use them is a crucial part of choosing the right tech stack.

Prometheus wasn't just adapted for Kubernetes; it feels like it was designed for it from day one. Unlike legacy monitoring tools that wait for agents to push data, Prometheus uses a pull-based model. It actively reaches out and scrapes metrics from its targets. This approach is a perfect match for a Kubernetes cluster, where you can't rely on static configurations for services that might not exist in five minutes.

Native Integration and Automatic Discovery

The real magic of running Prometheus on Kubernetes is its native service discovery. Prometheus can talk directly to the Kubernetes API to get a live inventory of services, pods, and endpoints, then automatically start scraping them. This completely gets rid of the manual, error-prone configuration that made monitoring a nightmare in older infrastructures.

This automatic discovery is a game-changer for operations teams. You no longer need to update a configuration file every time a new microservice is deployed; Prometheus handles it for you, ensuring nothing is missed.

This is all made possible by a few key Kubernetes Custom Resource Definitions (CRDs) that the Prometheus Operator introduces:

• ServiceMonitors: These CRDs are the workhorse. They tell Prometheus how to find a Kubernetes Service and scrape the metrics from all the pods behind it. You define it once, and as the service scales, Prometheus keeps up.
• PodMonitors: These work just like ServiceMonitors but target pods directly using label selectors. This is incredibly useful for scraping metrics from headless services, daemonsets, or any pod that isn't fronted by a traditional service.
• Exporters: Components like kube-state-metrics are the critical translators. They watch the Kubernetes API and turn the state of objects—like deployments, nodes, and pods—into metrics that Prometheus can actually scrape and understand.

This tight integration gives you a real-time, high-fidelity view of your cluster's health, from individual pod resource usage all the way up to the status of your control plane components. It shifts monitoring from a reactive chore to a proactive discipline, letting engineers spot trouble long before it can cause an outage.

Before you can start monitoring anything, you're at a fork in the road. How you deploy Prometheus on Kubernetes is a foundational choice that will dictate your operational overhead, control, and long-term scalability. This isn't just about installation; it's about committing to a specific operational model.

Your choice really boils down to three distinct paths: a manual do-it-yourself (DIY) build, using the community-standard Helm chart, or handing it all off to a managed service. Each has real trade-offs that cater to different teams, budgets, and levels of Kubernetes expertise.

Comparing Prometheus Deployment Methods on Kubernetes

Choosing the right path sets the foundation for your entire observability strategy. The table below breaks down the common methods, helping you map your team's needs to the right approach.

This decision impacts everything from initial setup complexity to your ability to scale effectively down the line. It's worth taking the time to get it right.

The De Facto Standard: kube-prometheus-stack

For most teams, the conversation starts and ends with the kube-prometheus-stack Helm chart. There's a reason it's the default choice: it bundles Prometheus, Alertmanager, Grafana, and crucial exporters into a single, cohesive package that just works.

This isn't just about saving time on the initial install. The real value is that it solves about 80% of the integration headaches out of the box. Think about all the moving parts:

• Configuring RBAC permissions
• Wiring up service discovery
• Setting up Grafana data sources
• Deploying node exporters and kube-state-metrics

The chart handles all of this, giving you a production-ready monitoring stack in minutes. You still get extensive control through its values.yaml file, but you're not starting from a blank slate.

For any new project, starting with the kube-prometheus-stack is a no-brainer. It lets you focus on monitoring your actual applications instead of spending weeks building the monitoring platform itself.

The Other Paths: DIY and Managed Services

On one end of the spectrum is the manual, do-it-yourself approach. This gives you absolute control, but it's a massive undertaking. You're responsible for wiring every component, managing every configuration file, and handling every update. This path is rarely the right one unless you have a dedicated platform team and a very specific, non-standard requirement.

On the other end are managed Prometheus services. These platforms offer a completely hands-off experience by handling scaling, long-term storage, and maintenance for you. This is a compelling option for large organizations concerned with uptime SLAs, compliance, and reducing operational headcount. The trade-off is what you'd expect: less control, potential for vendor lock-in, and cost.

A small, agile team will almost certainly favor the velocity of the Helm chart. A large, regulated enterprise might find the operational simplicity and support of a managed solution worth the price tag. Your decision should come down to your team's Kubernetes proficiency, your budget, and how much time you can realistically dedicate to maintaining your monitoring infrastructure.

Deploying with the Kube-Prometheus-Stack Helm Chart

When it comes to deploying Prometheus on Kubernetes, the kube-prometheus-stack Helm chart isn't just an option; it's the standard for a reason. It goes way beyond a vanilla Prometheus install by bundling Grafana, Alertmanager, and critical exporters into a single, cohesive package. This approach saves you from the nightmare of manually integrating all these components, letting you focus on monitoring your apps instead of building the monitoring platform itself.

While the default settings get you a running stack in minutes, they won't get you a production-ready one. That’s where your values.yaml file comes in. A mature monitoring stack is the bedrock of operational excellence—we've seen organizations report a 40-50% reduction in mean time to detect (MTTD) and a 35% decrease in mean time to resolve (MTTR) after getting this right. You can see a great breakdown of how observability drives these numbers in this insightful KubeCon talk.

Key Configurations for a Production Setup

Your first two priorities for any real-world deployment are simple: data persistence and security. Without them, you’ll lose all your metrics and dashboards the first time a Prometheus or Grafana pod gets rescheduled.

A stateless monitoring system is only useful for a demo. In the real world, you need historical data to debug incidents and analyze trends. Enabling persistence for Prometheus and Grafana is non-negotiable.

These are the values.yaml settings you absolutely need to configure from day one:

• Prometheus Persistence: Enable a PersistentVolumeClaim to ensure your time-series data survives pod restarts. While you're at it, set a reasonable data retention policy, like 30d, to keep your disk usage under control.
• Grafana Persistence: You need to do the same for Grafana. Enabling persistence here ensures your dashboards, data sources, and user settings aren't wiped out on the next restart.
• Grafana Admin Password: Never leave the default admin password. This is a huge security risk. Change it immediately, ideally by referencing a Kubernetes secret or setting the adminPassword value.
• Alertmanager Configuration: At a minimum, set up a basic Alertmanager configuration with a default receiver. Even a simple route to a Slack channel is a massive improvement, ensuring your team actually gets notified when things break.

For a more granular look at the specific values.yaml settings, you can check out our detailed guide on the Prometheus Helm chart. Getting these fundamentals right from the start gives you a stable foundation. This chart-based approach delivers the best of both worlds: the speed of a pre-packaged solution and the deep configuration control needed for a serious Prometheus setup on Kubernetes.

Tapping into Application Metrics with ServiceMonitors and PodMonitors

Getting cluster metrics is essential, but it's just the starting point. The real value of Prometheus on Kubernetes emerges when you start pulling metrics directly from your own applications. This is the moment Prometheus graduates from an infrastructure tool to a core component of your entire development lifecycle.

Thankfully, the Prometheus Operator makes this transition seamless. It extends the Kubernetes API with Custom Resource Definitions (CRDs) that let you declaratively define what to monitor. The two most important CRDs you'll work with are ServiceMonitor and PodMonitor. These simple YAML objects tell Prometheus exactly where to find your applications and how to scrape them, all managed right alongside your application code in Git.

ServiceMonitor vs. PodMonitor: Which One to Use?

So, how do you decide between a ServiceMonitor and a PodMonitor? The choice boils down to how your application is exposed inside the Kubernetes cluster.

• ServiceMonitor: This is your default, go-to option for most standard deployments. It targets a Kubernetes Service and automatically discovers all the pods behind it. If you have a regular deployment fronted by a service that load-balances traffic, this is what you'll use 90% of the time.

• PodMonitor: This CRD skips the Service layer entirely and targets pods directly using label selectors. It’s the perfect tool for headless services, StatefulSets (like databases or message queues), or any other scenario where you need to scrape pods individually without a traditional service endpoint.

This declarative approach is a massive win for GitOps. Developers can now own the monitoring configuration for their own services. They just commit a ServiceMonitor manifest with their application code, and the Prometheus Operator automatically configures the scraping. It’s a self-service model for observability.

Here’s a practical example. This ServiceMonitor targets our payment-api in the prod namespace, telling Prometheus to scrape the web-metrics port on each pod every 30 seconds.

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: payment-api-monitor namespace: monitoring labels: release: prometheus # This must match the selector on your Prometheus Operator spec: selector: matchLabels: app: payment-api namespaceSelector: matchNames: - prod endpoints:

• port: web-metrics interval: 30s

This simple resource is all it takes. Prometheus became the de facto standard for a reason; industry surveys from firms like Apptio show that over 70% of organizations rely on it for their Kubernetes observability. By using these CRDs, you empower your teams to instrument their own services and gain visibility in minutes.

Once you have those metrics flowing, you can start building powerful queries. To get started, check out our collection of useful Prometheus queries for analyzing application performance.

Scaling Prometheus with Thanos for Long-Term Storage

A single Prometheus instance is a fantastic starting point, but it has its limits. Sooner or later, you'll hit a wall. Local storage isn't built for infinite data retention, and you can't get a unified query view across multiple Kubernetes clusters. This is the next stage of maturity for any monitoring strategy: moving beyond a single instance to a globally scalable, highly available metrics system.

That’s precisely the problem Thanos was built to solve. It extends your existing Prometheus setup, transforming it into a distributed powerhouse without forcing your team to abandon the tools they already know. Instead of replacing Prometheus, Thanos sits on top of it, adding the long-term storage and query capabilities you need to grow.

The Thanos Architecture Explained

So, how does it work? Thanos cleverly breaks the problem down into a few key components that work in concert. Grasping these pieces is the key to understanding how it scales Prometheus so effectively.

• Sidecar: This is a container that runs right alongside each of your Prometheus pods. Its main job is to watch for new metric blocks and upload them to a central, cost-effective object storage bucket (like AWS S3 or Google Cloud Storage). This is the magic that unlocks long-term data retention, getting you past the local disk limits of a single pod.

• Querier: The Thanos Querier acts as a single, global query endpoint. It’s smart enough to pull data from two places: the real-time metrics from each Prometheus instance's Sidecar and the historical data sitting in your object storage. This gives your team one place to run queries that can span your entire infrastructure, past and present.

• Store Gateway: This component is the bridge to your history. It makes all the metric data in your object storage bucket available to the Querier, allowing you to efficiently run queries against months or even years of historical data.

The diagram below shows how these monitoring components fit into a modern, declarative Kubernetes ecosystem.

This visual drives home the point that modern monitoring is managed declaratively, often through GitOps, using CRDs like ServiceMonitor to automatically discover and scrape application metrics. For a platform team, adopting a tool like Thanos becomes the next logical step. It's how you meet strict compliance requirements for data retention while scaling your monitoring capabilities as the business grows.

To see how this works in a complex, real-world environment, check out our case study on scaling monitoring with Thanos.

Common Questions from the Field

As teams move Prometheus into production on Kubernetes, the same questions pop up time and time again. Theory is one thing; making it work reliably under pressure is another. Let's tackle some of the most common points of confusion we see with teams on the ground.

ServiceMonitor vs. PodMonitor: Which One Should I Use?

This is easily the most frequent question, but the answer is usually straightforward. Your choice depends on how you want Prometheus to find your application's metrics endpoints.

• A ServiceMonitor is your default, go-to choice. It tells Prometheus to discover pods through a standard Kubernetes Service. For the vast majority of stateless applications, this is the pattern you want. It's simple and it just works.

• A PodMonitor is for the exceptions. You use it when you need to bypass the Service abstraction and scrape pods directly based on their labels. This is perfect for headless services, StatefulSets where you need to target specific instances, or any other scenario where a traditional, load-balanced Service isn't part of the picture.

The rule of thumb we follow is this: always start with a ServiceMonitor. Only reach for a PodMonitor when you have a specific, documented reason why a Service-based discovery won't work for your use case.

Can a Single Prometheus Server Monitor Multiple Clusters?

Technically, you can bend it to do this. But you absolutely shouldn't. It's considered a major anti-pattern in the community for good reason.

Prometheus is designed and optimized to run inside the cluster it's monitoring. This colocation makes service discovery simple and reliable. Trying to monitor a remote cluster introduces network dependencies, security complexities, and a single point of failure that undermines the entire reason you run a resilient Kubernetes platform.

For a unified view across multiple clusters, the correct architecture is a federated system like Thanos. Each cluster runs its own local Prometheus instance, and Thanos provides a global query layer on top. This gives you that aggregated, multi-cluster dashboard without sacrificing the reliability of the local monitoring setup.

The Prometheus ecosystem is incredibly mature for Kubernetes, with over 150 official exporters and integrations available. When you pair it with Grafana, which 85% of users do for visualization, you get a powerful, production-ready observability stack. You can read more about how these tools work together to build dashboards that provide real operational insight.

At CloudCops GmbH, we design and build production-ready observability platforms using Prometheus, Grafana, and Thanos. If you're looking to get full visibility into your cloud-native environments, learn how we can help.

Continue Reading

Read Prometheus Helm Chart: A Production-Ready Guide

May 7, 2026

Prometheus Helm Chart: A Production-Ready Guide

Deploy the Prometheus Helm chart like a pro. Our guide covers production-ready installation, values.yaml tuning, ServiceMonitors, HA, and GitOps best practices.

prometheus helm chart

+4

C

CloudCops

Read How to Improve MTTR: A Cloud-Native Guide 2026

Jun 24, 2026

How to Improve MTTR: A Cloud-Native Guide 2026

Learn how to improve MTTR with a practical, cloud-native guide covering OpenTelemetry, automated remediation, GitOps, & blameless culture.

how to improve mttr

+4

C

CloudCops

Read Mean Time to Recovery: A Guide for Cloud-Native Teams

Jun 17, 2026

Mean Time to Recovery: A Guide for Cloud-Native Teams

Learn to calculate, measure, and reduce Mean Time to Recovery (MTTR) in cloud-native systems. Our guide covers DORA metrics, SLOs, and actionable techniques.

mean time to recovery

+4

C

CloudCops