Documentation Grafana Cloud Observe and act Monitor infrastructure Kubernetes Monitoring Configure Kubernetes Monitoring Configure with Helm chart Configure with wizard

Configuration steps for Kubernetes Monitoring with Helm chart

On the main navigation of Grafana Cloud, navigate to the Cluster configuration tab under Kubernetes Monitoring. To configure with the Helm chart wizard, the wizard guides you through the following steps:

1. Backend and distribution: Install the required recording and alert rules, and choose your Cluster’s platform.
2. Access token: Provide the telemetry collector with an access policy token so you can send metrics and logs to Grafana Cloud.
3. Choose setup: Choose a preconfigured deployment of the Alloy collector that meets your Cluster’s scaling and resilience needs.
4. Monitoring:
   • Choose the telemetry you want to collect for metrics, logs, and events.
   • Decide whether to have services running in your Cluster automatically discovered so you can instrument them from Grafana Cloud.
5. Deployment:
   • Enter a name for your cluster and the namespace where the Alloy collector should be deployed.
   • Copy and paste the Helm command into your terminal.

The wizard turns these choices into Helm values and a deployment command. To learn how the chart organizes features, collectors, presets, and destinations, refer to Helm chart overview.

Before you begin

Make sure you have met the prerequisites required for these configuration steps.

To deploy Kubernetes Monitoring with the Helm chart, you need:

• A Grafana Cloud account
• The Admin role to install alerts
• To have completed the activation of the app
• A Kubernetes Cluster, environment, or fleet you want to monitor
• The kubectl and Helm command-line tools

• To check version compatibility for the features you plan to enable:
  • kube-state-metrics uses client-go to communicate with Clusters. For Kubernetes client-go version compatibility, refer to kube-state-metrics.
  • Cost metrics with OpenCost require Kubernetes 1.8 or later.
  • Visualizations for storage require Helm chart release v1.5.1 or later.

Backend and distribution

Install the required recording and alert rules into your Grafana Cloud stack, and choose the platform for your Cluster.

• A recording rule pre-computes query results and saves them as new metrics. Dashboards and alert rules then read the saved results instead of running expensive queries each time, which makes queries and alert evaluation faster. Recording rules add new time series to your stack rather than reducing existing ones. The workload views in Kubernetes Monitoring read from the metrics that the recording rules produce.
• An alert rule defines the conditions that fire an alert.

1. Under Backend Installation, click Install to install the required, preconfigured alert rules and recording rules to your Grafana Cloud stack. After they’re installed, the status shows Alert rules and recording rules installed.

2. Under Select your Kubernetes platform, choose your platform:

   • Kubernetes, for EKS on EC2, GKE, and similar distributions
   • Azure AKS
   • GKE Autopilot
   • EKS on Fargate
   • IBM Cloud
   • OpenShift

3. Click Next.

Access token

Choose how the Helm chart authenticates to Grafana Cloud. Grafana Alloy, the telemetry collector that the Helm chart deploys on your Cluster, needs an access policy token to send metrics and logs to Grafana Cloud.

You can supply that token in one of three ways: create a new access policy token, use an existing token, or point the chart at a Kubernetes Secret in your Cluster that already holds a token.

To create a new token:

1. Click Create a new token.

2. In the Access policy token name box, enter a name. This name helps you view and revoke the token in Grafana.com later.

3. In the Expiration date box, select or enter an expiration in days, or keep No expiry.

   The token uses the predefined scopes metrics:read and set:alloy-data-write. A scope defines which permissions a token has. For more information, refer to Scopes.

4. Click Create token.

To use an existing token:

1. Click Use an existing token.

2. Paste your token into the box.

To use a stored Kubernetes Secret:

1. Click Use a stored Kubernetes Secret.

2. In the Secret name box, enter the name of the Secret that already exists in your Cluster.

3. In the Namespace box, keep the installation namespace or enter the namespace that holds the Secret.

4. Optionally, expand the advanced options to change the Username key and Password key if your Secret doesn’t use the default keys username and password.

   The Secret must hold your Grafana Cloud instance ID as the username and an access policy token as the password, with the metrics:read and set:alloy-data-write scopes. To find your instance ID, refer to Find instance endpoints. The configuration page provides a kubectl command that includes your instance ID, which you can copy to create the Secret. Because the chart reads the token from the Secret at runtime, your token isn’t stored in Grafana and doesn’t appear in the generated Helm command.

5. Click Next.

Choose setup

On the Choose setup step at the Choose your Alloy deployment screen, select a preconfigured deployment of the Alloy collector based on the scale and resilience your Cluster needs. A collector is an instance of Grafana Alloy that gathers telemetry from your Cluster and sends it to Grafana Cloud. In the deployment options offered by the wizard, collectors run on every Node (a DaemonSet). Every option deploys collectors.

• A clustered backend is a second Alloy workload (a StatefulSet) that runs on your own Nodes and collects data that exists once per Cluster rather than per Node, such as Cluster events and state metrics. (It’s unrelated to the Backend Installation in the first step, which installs recording and alert rules into your Grafana Cloud stack.)
• Autoscaling lets a deployment handle more load without a reinstall. Buffering to persistent storage (a PVC) lets the backend hold data on disk so it survives restarts.

Use the following table to determine whether to choose the Scalable, Standard, or Lightweight Alloy deployment.

Click Next.

Monitoring

On the Monitoring step at Remote monitoring and service discovery, choose the telemetry that Kubernetes Monitoring collects from your Cluster, and whether to discover the services running in it.

Fleet Management lets you manage Alloy collectors from Grafana Cloud. The Deployment step’s generated deployment code connects your collectors to Fleet Management automatically, so you can change their remote settings without a Helm upgrade.

1. Choose the telemetry to collect for Kubernetes monitoring. Enable Kubernetes monitoring is required, so you can’t deselect it, but you can choose what it collects. Click the menu icon next to Enable Kubernetes monitoring and select Advanced metrics tuning or Advanced logs and events tuning. Switch each type of telemetry on or off, then click Save. For details on each option, refer to Metrics and Logs and events.

   You can change these choices later, but how depends on the type. Metrics settings apply locally on your Cluster, so changing them means updating your Helm values and redeploying the chart. You manage logs and events settings through Fleet Management, so you can change them without redeploying.

2. Decide whether to select Auto-discover services on my cluster. This option finds the services running in your Cluster so you can instrument them from Grafana Cloud.

   Instrumentation is what makes a service send application telemetry. You can add instrumentation code to the service yourself, or instrument it without code changes through Grafana Beyla. Either way, Grafana Cloud has to know a service exists before it can offer to instrument it. That’s what auto-discovery does.

   Auto-discovery feeds Instrumentation Hub, a Grafana Cloud tool in public preview. Instrumentation Hub lists the discovered services and gives you one-click instrumentation for each one.

   Select Auto-discover services on my cluster if you want to instrument services from Grafana Cloud. The wizard creates a configuration pipeline in Fleet Management. Instrumentation Hub uses that pipeline to discover services. Discovery alone collects no application telemetry. Telemetry starts only after you instrument a service, and discovery doesn’t override instrumentation you already have.

   Skip this option if you don’t plan to instrument services from Grafana Cloud. Click the menu icon next to Auto-discover services on my cluster, select Managed discovery pipeline settings, then choose Not recommended. The wizard doesn’t create a pipeline, Instrumentation Hub won’t list services from your Cluster, and one-click instrumentation isn’t available. You can still configure instrumentation in your Helm values at any time.

   If a discovery pipeline from an earlier setup already exists, choosing Not recommended leaves it in place. To remove it, delete it in Fleet Management.

Metrics

To choose which metrics are collected, click the menu icon next to Enable Kubernetes monitoring and select Advanced metrics tuning. Metrics run directly on your Cluster. These settings are configured locally and aren’t managed through Grafana Cloud.

Switch these metrics options on or off, then click Save.

• Cluster metrics: Collects CPU, memory, network, and storage metrics from Cluster Nodes and Pods.
  What is collectedWhy enable?

  Node resource usage; metrics about Pod health; Persistent volume usage; and Deployment, StatefulSet, and DaemonSet status.

  Essential for Cluster health monitoring; powers the Kubernetes Overview dashboard; tracks resource utilization and capacity planning; and detects Pod crashes, OOM kills, and so on.
• Cost metrics (costMetrics): Collects resource costs per namespace and workload to identify expensive Pods and optimize spending. Uses OpenCost.
  What is collectedWhy enable?Private registry settings

  Resource cost attribution by namespace, workload, and Pod; CPU and memory cost breakdowns; Cloud provider pricing data integration; and cost efficiency metrics.

  Shows cost data; helps identify expensive workloads; enables FinOps and cost optimization; and tracks spending trends over time.

  If your environment pulls container images from a private registry, you must configure OpenCost’s image source after you deploy. Without this setting, OpenCost Pods become stuck in ImagePullBackOff and no cost metrics are collected. Refer to Configure OpenCost for a private image registry.
• Energy metrics (hostMetrics.energyMetrics): Measures power consumption per Node and workload for carbon footprint tracking. Uses Kepler.
• Pod & Service metrics by annotation: Scrapes metrics from Pods and Services that have the k8s.grafana.com/scrape annotation.
  What does this do?Why enable?Why disable?

  Automatically discovers running pods and services; dynamically scrapes Prometheus metrics from annotated Pods; detects new workloads without manual configuration; and uses Kubernetes annotations to find metrics endpoints.

  Enable for: zero-configuration metrics collection; automatically monitor new applications as they deploy; support microservices architectures with dynamic scaling; and find application-specific metrics beyond system metrics.

  Disable when: You want to explicitly define every scrape target; you’re concerned about discovering unintended metrics endpoints; your cluster has very strict network policies.
• Prometheus Operator objects: Use this option if Prometheus Operator is already running in your Cluster and you have ServiceMonitor, PodMonitor, or Probe objects. It adds those objects as scrape targets.
  What does this do?Why enable?Why disable?

  Discovers and monitors Prometheus Operator custom resource definitions (CRDs). ServiceMonitor defines how to scrape metrics from Kubernetes services, PodMonitor defines how to scrape metrics from Pods, and Probe defines blackbox probing of endpoints.

  Enable when: you’re already using Prometheus Operator in your Cluster; you have existing ServiceMonitor/PodMonitor definitions; you want to use your existing Prometheus configurations; you want to enable migration from Prometheus Operator to Grafana Alloy.

  Disable when: You’re not using Prometheus Operator Objects; you prefer using the native configuration of Grafana Alloy.

Logs and Events

To choose which logs and events are collected, click the menu icon next to Enable Kubernetes monitoring and select Advanced logs and events tuning. These options are deployed and controlled through Grafana Cloud, so you can update and scale instrumentation without changing your Cluster configuration.

Switch these logs and events options on or off, then click Save.

• Cluster events: Collects Kubernetes events, including Pod failures, scheduling events, and resource state changes.
  What is collectedWhy enable?

  Generated from: the scheduler (assigning Pods to Nodes); kubelet (managing Pods on Nodes); the controller manager (handling scaling, deployments, and so on); the API server (processing requests).

  Troubleshooting scheduling & deployment issues; see why a Pod isn’t starting (no Nodes with enough memory); tracking resource lifecycle changes (Pod created → scheduled → pulled → started → ready → terminated); detecting transient or recurring failures (repeated image pull errors, failed probes, or node taints); auditing Cluster activity (to identify which controller or user triggered changes).
• Node logs: Collects kubelet and system logs from worker Nodes running systemd, including journald logs.
  What is collectedWhy enable?

  Logs from: the operating system (kernel logs, systemd, network drivers); Kubernetes node agents (like kubelet); container runtimes (Docker, containerd); system daemons (journald, syslog).

  Diagnosing node instability (memory exhaustion, CPU throttling, or disk space issues); debugging scheduling or startup failures (when Pods can’t start, the issue may be at the Node level); investigating network or storage problems; determining driver or volume mount failures; auditing system changes (Node reboots, kubelet restarts, or OS updates).
• Pod logs: Collects application logs from running Pods. Available as podLogsViaLoki (Loki pipeline) or podLogsViaOpenTelemetry (native OTLP format, the OpenTelemetry protocol) in the Helm chart values.
  What is collectedWhy enable?

  Standard output (stdout) and standard error (stderr) Pod logs from the processes in the container, such as initialization messages, API request logs, warnings or errors, and application-specific information.

  Enable for: debugging issues, such as when an application crashes or behaves unexpectedly, logs reveal what went wrong; monitoring behavior, such as tracking normal operational messages (startup confirmation, API requests, or job completion); auditing events, to view logs that show what actions were taken by your app or scripts running inside containers; gaining performance insight, such as tracing slow operations or bottlenecks using timestamps and log levels.
• Kubernetes manifests (kubernetesManifests): Collects Kubernetes resource manifests and changes as logs, using k8s-manifest-tail. Provides the data behind the View Manifest and Changes tabs on workloads and Pods.
  What is collectedWhy enable?

  Full resource manifests for Pods and workloads; creation, update, and deletion events for those resources; and the version history that powers the manifest diff viewer.

  When switched, you can compare a manifest against previous versions in a side-by-side diff; on Deployment detail page of k8s-manifest-tail, you can view a highlight on the graphs in the Optimization section to see if a deployment change correlates with a change in behavior.

After you finish choosing the telemetry, click Next.

Deployment

1. In the Cluster name box, enter a unique, human-readable name for this Cluster. Use only lowercase letters, numbers, and hyphens, start with a letter, and end with a letter or number (up to 63 characters). This name appears in dashboards, alerts, and Cluster selectors.

2. In the Namespace box, enter the namespace where you want to install Grafana Alloy.

3. Optionally, under Custom labels, add key and value labels, also known as attributes, to help you search for, identify, and deploy pipelines in Fleet Management. Click Add more to add another label. These labels are added to every collector’s remoteConfig.extraAttributes map in the generated deployment code.

   The generated deployment code registers your collectors with Fleet Management automatically, so you can manage them from Grafana Cloud and change the logs and events settings later without a Helm upgrade.

4. If you don’t already have the Grafana Helm charts repository, add it:

   Bash Copy

   helm repo add grafana https://grafana.github.io/helm-charts

   If you already have it, update it:

   Bash Copy

   helm repo update

5. Under Deployment code to copy, copy the generated helm upgrade --install command and run it in your terminal. Make sure your kubectl context points to the Cluster you want to monitor.

   After the chart deploys and your Cluster starts sending kube-state-metrics data, billing begins based on your active host hours and active container hours.

Verify your installation

Confirm that the collectors are running and data is arriving in Grafana Cloud:

1. Click Test connection to verify that the collectors install and send data to Grafana Cloud.

2. Click See cluster status to view the status of data collection. Data appears as the collectors start scraping and sending it to Grafana Cloud. This view shows the health of the different sources of metrics, Pod logs, and Cluster events, as well as any applicable version numbers.

This completes the configuration wizard.

Manage your installation

After you complete the wizard, use the following tasks to adjust, troubleshoot, and extend your installation.

Configure OpenCost for a private image registry

If you enabled cost metrics and your environment pulls container images from a private registry, configure OpenCost’s image source. Without this setting, OpenCost Pods become stuck in ImagePullBackOff and no cost metrics are collected.

The helm upgrade --install command that the wizard generated contains your Helm values. If you don’t have a values.yaml file, retrieve your current values first. Refer to Retrieve Helm values.

1. Add this to your values.yaml file:

   YAML Copy

   telemetryServices:
     opencost:
       deploy: true
       opencost:
         exporter:
           image:
             registry: my.private.registry.example.com

2. Deploy or upgrade the chart:

   Bash Copy

   helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring \
     --values values.yaml

For other private registry options, refer to Private image registry.

Troubleshoot

Refer to Troubleshooting for any issues that occur after configuration.

Optionally install any integrations

You can use Grafana integrations to monitor the health and status of services and applications running in your Kubernetes clusters.

To install a Kubernetes integration to begin scraping metrics:

1. From the main menu, navigate to Connections, and filter for Kubernetes.

2. Select the integration for the service you want to monitor.

3. Follow the instructions on the screen to copy and use the configuration snippet and install the integration.

4. After installing an integration, redeploy the configuration using the method you originally used.

Retrieve Helm values

If you installed Kubernetes Monitoring with the Helm CLI, you can retrieve the values for your configuration by using the helm get values command.