--- title: "Manage your configuration | Grafana Cloud documentation" description: "How Grafana Kubernetes Monitoring works and how to manage your configuration" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Manage your Kubernetes Monitoring configuration Kubernetes Monitoring gathers metrics, logs, and events, and calculates costs for your infrastructure. It also provides recording rules, alerting rules, and allowlists. ## How Kubernetes Monitoring works Kubernetes Monitoring uses the following to provide its data and visualizations. ### cAdvisor [cAdvisor](https://github.com/google/cadvisor) (one per Node) is present on each Node in your Cluster, and emits container resource usage metrics such as CPU usage, memory usage, and disk usage. Alloy collects these metrics and sends them to Grafana Cloud. ### Cluster events Kubernetes Cluster controllers emit information about events concerning the lifecycle of Pods, deployments, and Nodes within the Cluster. Alloy pulls these Cluster events using the Kubernetes API server and converts them into log lines, then sends them to Grafana Cloud logs. ### Grafana Alloy [Grafana Alloy](/docs/grafana-cloud/send-data/alloy/introduction/): - Collects all metrics, Cluster events, and Pod logs - Receives traces pushed from applications on Clusters - Sends the data to Grafana Cloud ### kube-state-metrics Kubernetes Monitoring uses [kube-state-metrics service](https://github.com/kubernetes/kube-state-metrics) (one replica, by default) to enable you to see the links between Cluster, Node, Pod, and container. The kube-state-metrics service listens to Kubernetes API server events, and generates Prometheus metrics that document the state of your Cluster’s objects. Over a thousand different metrics provide the status, capacity, and health of individual containers, Pods, deployments, and other resources. kube-state-metrics: - Generates metrics without modification - Listens to the Kubernetes API server (not present on each Node) - Provides metrics on the state of objects in your Cluster (Pods, Deployments, DaemonSets) The Kubernetes Monitoring Cluster navigation feature requires the following metrics: - `kube_namespace_status_phase` - `container_cpu_usage_seconds_total` - `kube_pod_status_phase` - `kube_pod_start_time` - `kube_pod_container_status_restarts_total` - `kube_pod_container_info` - `kube_pod_container_status_waiting_reason` - `kube_daemonset.+` - `kube_replicaset.+` - `kube_statefulset.+` - `kube_job.+` - `kube_node.+` - `kube_cluster.+` - `node_cpu_seconds_total` - `node_memory_MemAvailable_bytes` - `node_filesystem_size_bytes` - `node_namespace_pod_container` - `container_memory_working_set_bytes` ### kubelet [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#:~:text=The%20kubelet%20is%20the%20primary,object%20that%20describes%20a%20pod) (one per Node): - Is the primary `Node agent` present on each Node in the Cluster - Emits metrics specific to the `kubelet` process like `kubelet_running_pods` and `kubelet_running_container_count` - Ensures containers are running - Provides metrics on Pods and their containers Grafana Alloy collects these metrics and sends them to Grafana Cloud. ### Kubernetes mixins Kubernetes Monitoring is heavily indebted to the open source [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin) project, from which the recording and alerting rules are derived. Grafana Labs continue to contribute bug fixes and new features upstream. ### Node Exporter The Prometheus exporter [node-exporter](https://github.com/prometheus/node_exporter) runs as a DaemonSet on the Cluster to: - Gather metrics on hardware and OS for Linux Nodes in the Cluster - Emit Prometheus metrics for the health and state of the Nodes in your Cluster Grafana Alloy collects these metrics and sends them to Grafana Cloud. ### OpenCost Kubernetes Monitoring uses the combination of [OpenCost](https://opencost.io/) and Grafana to allow you to monitor and manage costs related to your Kubernetes Cluster. For more details, refer to [Manage costs](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/). ### Node logs These are logs written by host processes, not by anything running inside a container. Some Kubernetes nodes run system services as systemd units: kubelet (schedules and manages pods), containerd or docker (the container runtime), and kernel/network subsystems. When these components log errors, crashes, or state changes, those entries go to journald on the host. ### Pod logs Alloy pulls Pod logs from the workloads running within containers, and sends them to Loki. > Note > > Log entries must be sent to a Loki data source with `cluster`, `namespace`, and `pod` labels. ### Traces Traces generated by applications within the Cluster are pushed to Grafana Alloy. For the endpoint addresses where traces (and other application telemetry) can be pushed, refer to [Client endpoint configuration](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/collector-reference/#client-endpoint-configuration). ### Windows Exporter When monitoring Windows Nodes, the configuration installs the [`windows-exporter` DaemonSet](https://github.com/prometheus-community/windows_exporter) to ensure metrics are available for scraping. ## Recording rules [Recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) calculate in advance any mathematical expressions that are frequently needed or take a lot of computation, and then save the result as a new set of time series. This decreases the time for query and calculation. > Note > > Recording rules may emit time series with the same metric name, but different labels. To modify these programmatically, refer to [Set up Alerting for Cloud](/docs/grafana-cloud/alerting-and-irm/alerting/set-up/). Kubernetes Monitoring includes the following recording rules to speed up queries and the evaluation of [alerting rules](#alerting-rules). `apiserver_request:availability30d` `apiserver_request:availability30d` `apiserver_request:availability30d` `apiserver_request:burnrate1d` `apiserver_request:burnrate1d` `apiserver_request:burnrate1h` `apiserver_request:burnrate1h` `apiserver_request:burnrate2h` `apiserver_request:burnrate2h` `apiserver_request:burnrate30m` `apiserver_request:burnrate30m` `apiserver_request:burnrate3d` `apiserver_request:burnrate3d` `apiserver_request:burnrate5m` `apiserver_request:burnrate5m` `apiserver_request:burnrate6h` `apiserver_request:burnrate6h` `cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits` `cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests` `cluster:namespace:pod_memory:active:kube_pod_container_resource_limits` `cluster:namespace:pod_memory:active:kube_pod_container_resource_requests` `cluster_quantile:apiserver_request_sli_duration_seconds:histogram_quantile` `cluster_quantile:apiserver_request_sli_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile` `cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile` `cluster_verb_scope:apiserver_request_sli_duration_seconds_count:increase1h` `cluster_verb_scope:apiserver_request_sli_duration_seconds_count:increase30d` `cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase1h` `cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase30d` `code:apiserver_request_total:increase30d` `code:apiserver_request_total:increase30d` `code_resource:apiserver_request_total:rate5m` `code_resource:apiserver_request_total:rate5m` `code_verb:apiserver_request_total:increase1h` `code_verb:apiserver_request_total:increase1h` `code_verb:apiserver_request_total:increase1h` `code_verb:apiserver_request_total:increase1h` `code_verb:apiserver_request_total:increase30d` `container_cpu_usage_seconds_total` `container_memory_rss` `container_memory_working_set_bytes` `namespace_cpu:kube_pod_container_resource_limits:sum` `namespace_cpu:kube_pod_container_resource_requests:sum` `namespace_memory:kube_pod_container_resource_limits:sum` `namespace_memory:kube_pod_container_resource_requests:sum` `namespace_workload_pod:kube_pod_owner:relabel` `namespace_workload_pod:kube_pod_owner:relabel` `namespace_workload_pod:kube_pod_owner:relabel` `namespace_workload_pod:kube_pod_owner:relabel` `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` `node_namespace_pod_container:container_memory_cache` `node_namespace_pod_container:container_memory_rss` `node_namespace_pod_container:container_memory_swap` `node_namespace_pod_container:container_memory_working_set_bytes` `node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile` `node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile` `node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile` ## Alerting rules Kubernetes Monitoring comes with preconfigured alerting rules to alert on conditions such as “Pods crash looping” and “Pods getting stuck in not ready”. The following alerting rules create alerts to notify you when issues arise with your Clusters and their workloads. To learn more, refer to the upstream Kubernetes-Mixin’s [Kubernetes Alert Runbooks](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/runbook.md#kubernetes-alert-runbooks) page. To update programmatically the alerting rule links to point your own runbooks in these preconfigured alerts, use a tool like [cortex-tools](https://github.com/grafana/cortex-tools) or [grizzly](https://github.com/grafana/grizzly). ### kubelet alerting rules - `KubeNodeNotReady` - `KubeNodeReadinessFlapping` - `KubeNodeUnreachable` - `KubeletClientCertificateExpiration` - 7 day expiration - `KubeletClientCertificateExpiration` - 1 day expiration - `KubeletDown` - `KubeletPlegDurationHigh` - `KubeletPodStartUpLatencyHigh` - `KubeletServerCertificateExpiration` - 7 day expiration - `KubeletServerCertificateExpiration` - 1 day expiration - `KubeletClientCertificateRenewalErrors` - `KubeletServerCertificateRenewalErrors` - `KubeletTooManyPods` ### Kubernetes alerting rules - `KubeContainerWaiting` - `KubeDaemonSetMisScheduled` - `KubeDaemonSetNotScheduled` - `KubeDaemonSetRolloutStuck` - `KubeDeploymentGenerationMismatch` - `KubeDeploymentReplicasMismatch` - `KubeDeploymentRolloutStuck` - `KubeHpaMaxedOut` - `KubeHpaReplicasMismatch` - `KubeJobFailed` - `KubeJobNotCompleted` - `KubePodCrashLooping` - `KubePodNotReady` - `KubeStatefulSetGenerationMismatch` - `KubeStatefulSetReplicasMismatch` - `KubeStatefulSetUpdateNotRolledOut` ### Kubernetes API alerting rules - `KubeAggregatedAPIDown` - `KubeAggregatedAPIErrors` - `KubeAPIDown` - `KubeAPIErrorBudgetBurn` - `KubeAPIErrorBudgetBurn` - `KubeAPIErrorBudgetBurn` - `KubeAPIErrorBudgetBurn` - `KubeAPITerminatedRequests` - `KubeClientCertificateExpiration` - less than 7 days - `KubeClientCertificateExpiration` - less than 1 day ### Kubernetes resource usage alerting rules - `CPUThrottlingHigh` - `KubeCPUOvercommit` - `KubeCPUQuotaOvercommit` - `KubeMemoryOvercommit` - `KubeMemoryQuotaOvercommit` - `KubeQuotaAlmostFull` - `KubeQuotaExceeded` - `KubeQuotaFullyUsed` ### Kubernetes storage alerting rules - `KubePersistentVolumeErrors` - `KubePersistentVolumeFillingUp` - 1 hour - `KubePersistentVolumeFillingUp` - 1 minute - `KubePersistentVolumeInodesFillingUp` - 1 hour - `KubePersistentVolumeInodesFillingUp` - 1 minute ### Kubernetes system alerting rules - `KubeClientErrors` - `KubeVersionMismatch` ## Metrics management and control If your account is based on [billable series](#billable-series), there are ways to control and manage metrics. ### Identify unnecessary or duplicate metrics To identify unnecessary or duplicate metrics generated from a Cluster, you can: - Use the **Cardinality** page to discover on a Cluster-by-Cluster basis where all your active series are coming from. From the main menu, click **Configuration** and then the **Cardinality** tab. [Cardinality page within the app](/media/docs/grafana-cloud/k8s/cardinality-page.png) - Analyze current metrics usage and associated costs from the [billing and usage dashboard](/docs/grafana-cloud/platform/cost-management-and-billing/understand-usage-cost/) located in your Grafana instance. ### Analyze usage For techniques to analyze usage, refer to [Analyze Prometheus metrics costs](/docs/grafana-cloud/cost-management-and-billing/analyze-costs/metrics-costs/prometheus-metrics-costs/). ### Reduce usage Use and refine an allowlist to reduce metrics to only those you want to receive. An allowlist is a set of metrics and labels that you want to gather while all others are dropped. Most metric sources apply a default allowlist that keeps the minimal set of metrics Kubernetes Monitoring needs, because `metricsTuning.useDefaultAllowList` defaults to `true`. The control plane sources are the exception: `apiServer`, `kubeControllerManager`, `kubeDNS`, `kubeProxy`, and `kubeScheduler` have no default allowlist and keep everything they scrape. To adjust a source, set `metricsTuning.includeMetrics` to keep additional metrics and `metricsTuning.excludeMetrics` to drop metrics you don’t need. Both accept regular expressions. On a source that has a default allowlist, set `metricsTuning.useDefaultAllowList` to `false` to stop filtering it. When keeping and dropping aren’t enough, use `extraMetricProcessingRules`, which takes Alloy relabeling rules. For the `keep` and `drop` actions and the other fields a rule accepts, refer to [`rule` block](/docs/grafana-cloud/send-data/alloy/reference/components/prometheus/prometheus.relabel/#rule). Relabeling works the same way it does in Prometheus, so [Relabeling rule fields](https://training.promlabs.com/training/relabeling/writing-relabeling-rules/relabeling-rule-fields/) is useful background on what each field does. The examples there use Prometheus YAML, so write the rules themselves in Alloy syntax. These keys belong to the individual metric source, under the feature that collects it, such as `clusterMetrics.kubelet` or `hostMetrics.linuxHosts`. They’re configured locally on your Cluster rather than from Grafana Cloud. For examples, refer to [Processing and labeling](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/customize-helm-chart/#processing-and-labeling) and [Metrics tuning](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/metrics-tuning/README.md) in the Helm chart repository. To learn more about how dropping metrics affects cost, refer to [Relabel Prometheus metrics to reduce usage](/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/client-side-filtering/#relabel-prometheus-metrics-to-reduce-usage). ### Billable series If your account does not use [host-hours pricing](/docs/grafana-cloud/platform/pricing-and-usage/kubernetes-monitoring/), pricing is based on [billable series](/pricing/). To learn more about this pricing model, refer to [Active series and DPM](/docs/grafana-cloud/platform/pricing-and-usage/metrics/). Default telemetry data collection (also called [active series](/docs/grafana-cloud/platform/pricing-and-usage/metrics/#active-series)) varies depending on your Kubernetes Cluster size (number of Nodes) and running workloads (number of Pods, containers, Deployments, etc.). Active series usage scales with both. Each additional Node adds series from the collectors and exporters that run on every Node, and each additional Pod or container adds series describing Cluster state. Targets beyond the defaults, such as application metrics, API server metrics, and scheduler metrics, add more series on top. Because usage also varies with your Cloud provider and Kubernetes version, measure your own Cluster rather than working from a general estimate. To see where your active series come from, use the **Cardinality** page described in [Metrics management and control](#metrics-management-and-control). ## Logs management Control and manage logs by: - Only collecting logs from Pods in certain namespaces - Dropping logs based on content For more on analyzing, customizing, and deduplicating logs, refer to [Logs in Explore](/docs/grafana/latest/explore/logs-integration/). ### Limit logs to only Pods from certain namespaces By default, Kubernetes Monitoring gathers logs from Pods in every namespace. However, you may only want or need logs from Pods in certain namespaces. Set the namespace list in your values file. On the Pod logs feature you use, such as `podLogsViaLoki`, the `namespaces` key limits collection to the namespaces you list, and `excludeNamespaces` drops the namespaces you list. For example, the following gathers logs only from the `production` and `staging` namespaces: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml podLogsViaLoki: namespaces: - production - staging ``` When namespace and label matching aren’t enough, set `extraDiscoveryRules` on the same feature, which takes Alloy relabeling rules. Refer to [Discovery](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/customize-helm-chart/#discovery). ### Drop logs based on content Similarly to filtering to specific namespaces, you can use [Loki processing rules](/docs/grafana-cloud/send-data/alloy/reference/components/loki/loki.process/) to further process and optionally drop log lines. Set them with `extraLogProcessingStages` on the Pod logs feature you use. For example, this processing stage drops any log lines that contain the word debug: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml podLogsViaLoki: extraLogProcessingStages: |- stage.drop { expression = ".*(debug|DEBUG).*" } ``` ## Manage Assistant preferences The **Manage app** tab on the Kubernetes Monitoring **Configuration** page includes preferences for Grafana Assistant. ### Auto-run assistant health checks The **Auto-run assistant health checks** toggle controls whether [Assistant health checks](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/triage-your-infrastructure/use-assistant-health-checks/) run automatically on every Kubernetes detail page. - When the toggle is on, Grafana Assistant executes Prometheus and Loki queries in the background on every Cluster, namespace, workload, Pod, container, and Node detail page, then displays the health check message. - When the toggle is off, the check doesn’t run automatically. Refresh it manually from the banner, or open the Assistant from a panel to ask about a page. Turn off the auto health check to reduce background query load and Assistant usage. The toggle is saved in your browser’s local storage, so it’s a per-user, per-browser preference that doesn’t affect teammates or other browsers. [**Auto-run assistant health checks** toggle on the **Manage app** tab](/media/docs/grafana-cloud/k8s/auto-run-assistant-health-checks.png) ## Deactivate Kubernetes Monitoring To deactivate Kubernetes Monitoring, refer to [Deactivate Kubernetes Monitoring](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/activate/#deactivate-kubernetes-monitoring). ## Get support To open a [support ticket](/docs/grafana-cloud/security-and-account-management/support/), navigate to your [Grafana Cloud Portal](/docs/grafana-cloud/security-and-account-management/cloud-portal/), and click **Open a Support Ticket**.