Monitor on Grafana Labs

Monitor GEM using Grafana Cloud

Thu, 01 May 2025 09:53:19 +0000

Monitor GEM using Grafana Cloud

Note
Monitoring with Grafana Cloud is required for all GEM installations. Refer to Monitor the health of your system for more information. Professional services will assist you in configuring monitoring for your GEM installation.

Grafana Labs champions the value of observability for your software applications and infrastructure. This extends to the Database solutions offered as part of the Grafana Enterprise Stack: namely Grafana Enterprise Metrics, Logs, and Traces. Successful operation of these databases requires comprehensive monitoring and alerting. This allows the teams managing these Databases to swiftly identify issues and diagnose their root causes. Grafana Labs itself maintains robust observability for the Grafana Enterprise Metrics, Logs, and Traces Databases it operates to power Grafana Cloud, our hosted observability-as-a-service offering.

Our recommended approach for monitoring your Grafana Enterprise Databases is to use Grafana Cloud. This approach is the easiest, most robust way to observe your database in accordance with Grafana Labs best practices and is included free of charge with your purchase of a Grafana Enterprise Database.

How it works

Each Grafana Enterprise Database is instrumented for observability by default. Each Database component exposes metrics on a scrapable Prometheus compatible /metrics endpoint and emits logs and traces.

The Helm chart for each Enterprise Database includes an option for meta monitoring. When enabled, the Helm chart deploys Grafana Alloy alongside the Enterprise Database. Alloy is configured to collect metrics and logs from all database components and apply additional metadata, such as extra labels to indicate where the metrics or logs were scraped from. Alloy ONLY collects logs and metrics relevant to the internal system state of the Grafana Enterprise Databases; no other information is collected.

From there, Grafana Alloy forwards the telemetry data to your Grafana Cloud account, where it can be stored and queried to help you understand the health of your Grafana Enterprise Database. You can access this account at any time, and have complete control over who has access to it using Grafana Cloud’s identity and access management functionality. Availability and uptime adhere to the Grafana Cloud SLA.

Additionally, this Grafana Cloud account is preconfigured with Grafana Labs’ recommended dashboards for visualizing the telemetry data collected from your Grafana Enterprise Database. Should you choose, you can also create alerts on the data in this account to notify you when the telemetry data collected from your Grafana Enterprise Database is no longer within expected parameters. Grafana Labs can provide a recommended set of alerts to configure.

If you need to file a support escalation, you can choose to give the Grafana Labs support team access to the telemetry data in your Grafana Cloud account. Historically, Grafana Labs has found that faster access to telemetry data radically reduces the time needed to resolve escalations.

Note
You can also use self-monitoring with GEM.

Self-monitoring

Thu, 12 Jun 2025 13:03:13 +0000

Self monitoring

Caution
Starting with GEM version 2.16, self-monitoring is deprecated and set to be removed in a future release. As of May 2025, you must use metamonitoring. Refer to Monitor the health of your system.

Overview

Since version 1.4, Grafana Enterprise Metrics (GEM) includes the ability to directly record self-monitoring metrics to allow you to easily monitor the health and stability of GEM itself. The metrics GEM collects about itself are written to a built-in __system__ tenant. The metrics written can be queried as usual using tokens created under the built-in __system__ access policy. Since version 1.8, GEM directly records exemplars as part of self-monitoring metrics.

The way self-monitoring works ensures that any metrics available from GEM via /metrics endpoints are available directly in GEM without needing to be scraped by an external process. While these metrics would ordinarily need to be scraped using Prometheus or Grafana Alloy, with self-monitoring, they are available after the following setup.

This feature provides a simple, out-of-the-box way to monitor GEM itself with a minimum amount of configuration or extra dependencies. To get the maximum value of this feature, we recommend you install GEM’s Grafana plug-in, which automatically provisions a set of dashboards that use the self-monitoring metrics. The dashboards are in line with Grafana Labs’ best practices for understanding GEM system health. Self-monitoring is compatible with plugin versions >= 3.0.4 (which require Grafana 8). Grafana 7.5 users should use version 2.1.1.

Configuration

The sections below describe the steps needed to set up self monitoring.

Single binary mode

Self-monitoring is enabled by default - no action is necessary in single binary mode!

Microservices mode

In order to use self-monitoring in microservices mode, you’ll need a hostname that you can use to address the gRPC port (9095 by default) of each of the GEM distributors. This could be a load balancer that balances between each distributor, a DNS A record that includes IPs for each distributor, or a Kubernetes service that balances between each gRPC port of the distributor pods. For the purposes of this example, we’ll assume that you are using a Kubernetes service and GEM is running in a namespace called enterprise-metrics.

Add the following section to your GEM configuration file used by each GEM pod or process.

instrumentation:
  distributor_client:
    address: dns:///distributor.enterprise-metrics.svc.cluster.local:9095

Or, you can alternatively add the command line flag to the arguments passed to each GEM pod or process.

-instrumentation.distributor-client.address='dns:///distributor.enterprise-metrics.svc.cluster.local:9095'

What is described above will give you system health metrics about the entire GEM cluster. To better understand GEM behavior, you also want to understand resource usage at a per-tenant level. In order to get the self-monitoring metrics you need to understand this behavior (and populate the “Per Tenant Usage” dashboards provisioned by the GEM plugin), you must also deploy the overrides-exporter component.

Exemplars

Since GEM 1.8, self-monitoring has the ability to directly record exemplars. However, recording of the exemplars under the __system__ tenant is still controlled by the same limits applied to all other tenants. This means that recording of exemplars for the __system__ tenant is disabled by default (as it is for all tenants) and must be enabled using the runtime configuration file or enabled globally.

Since the __system__ tenant is built into GEM itself and immutable, limits for it (such as enabling exemplars) cannot be set using the Admin API. Instead, if you wish to emit exemplars for the __system__ tenant you must override the max_global_exemplars_per_user setting for the __system__ tenant using the runtime configuration file or enable exemplars globally.

Here is an example of using the runtime configuration file:

overrides:
  __system__:
    max_global_exemplars_per_user: 300000

Verification

After you’ve deployed the configuration changes above, you’ll need to verify that self-monitoring is working correctly. We’ll learn how to query the self-monitoring metrics later, but to verify they’re working we can check a simple counter incremented when self-monitoring metrics are emitted.

Pick a single pod or process that is part of your GEM cluster. For this example, we’ll assume that you have picked an ingester.

Make a curl request to the /metrics endpoint of the ingester.

$ curl -s 'http://ingester-01.example.com/metrics' | grep 'cortex_self_monitoring_pushes_total'
# HELP cortex_self_monitoring_pushes_total Number of successes pushing self-monitoring metrics
# TYPE cortex_self_monitoring_pushes_total counter
cortex_self_monitoring_pushes_total 15

NOTE If you are running GEM in a Kubernetes cluster, individual pods might not be directly accessible from outside the Kubernetes cluster. In this case you can make the request from another pod running in the Kubernetes cluster, or you can make use of the kubectl port-forward command.

If the metric above is 0 or doesn’t exist, check the logs for each GEM component looking for errors or warnings related to pushing metrics to a distributor.

Querying

In order to query self-monitoring metrics directly, you’ll need to create a token associated with the __system__ access policy. The steps below assume you have already done this and copied down the token. The following examples further assume that your GEM cluster is available at the host gem.example.com over HTTPS.

First, set the token as a variable to use for the subsequent commands.

$ export API_TOKEN="the long token string you copied"

Next, we’ll make a request to the Prometheus query endpoint of GEM looking for a particular metric. In this case, grafana_metrics_enterprise_build_info

$ curl -s -u "__system__:$API_TOKEN" "https://gem.example.com/prometheus/api/v1/query?query=grafana_metrics_enterprise_build_info" | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "grafana_metrics_enterprise_build_info",
          "branch": "gem-release-1.4",
          "goversion": "go1.16.3",
          "instance": "ingester-01:80",
          "revision": "ccd12b7a",
          "target": "ingester",
          "version": "v1.4.1"
        },
        "value": [
          1622833381.751,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "grafana_metrics_enterprise_build_info",
          "branch": "gem-release-1.4",
          "goversion": "go1.16.3",
          "instance": "distributor-01:80",
          "revision": "ccd12b7a",
          "target": "distributor",
          "version": "v1.4.1"
        },
        "value": [
          1622833381.751,
          "1"
        ]
      },

      <...snip...>
    ]
  }
}

As you can see, querying self-monitoring metrics with GEM is the same process as querying any other type of metrics.

Implementation

Though you don’t need to be familiar with how self-monitoring works at a technical level, it’s detailed below in the hopes that it’s useful.

Gathering

Self-monitoring metrics are gathered internally the same way metrics exposed via the /metrics endpoint are: they are registered with a Prometheus Registerer on application start up. The metrics are updated during the normal course of running the application and periodically ( every 15 seconds by default) flushed directly to a distributor. Any metric available from the /metrics endpoint of a GEM component will also be available in the self-monitoring system.

The metrics are written to the distributor over its gRPC interface. This allows the self-monitoring system control over the exact tenant the metrics are stored under. This enables it to cleanly separate system metrics (under the __system__ tenant) from user data.

Injected labels

Normally, when metrics are scraped by Prometheus, labels are automatically added by Prometheus that identify where the metrics came from. Since self-monitoring metrics are not scraped by any external system, labels are automatically added internally to help identify which component the metrics came from.

The following labels are added to metrics emitted by the self-monitoring system.

instance: this label is made up of the node or host name a component is running on in combination with the HTTP port used. For example a value for this label in a GEM cluster running on Kubernetes might be ingester-1:80 or querier-5bf6ddccd7-hzbtn:80.
target: this label is made of a comma separated list of the targets a GEM process is running as (ingester, querier, etc.) or all in single binary mode.

System tenant and access policy

In order to cleanly separate self-monitoring data from user data, GEM comes with a built-in __system__ tenant and __system__ access policy. All self-monitoring data is written to the __system__ tenant. The self-monitoring data may be queried using tokens associated with the__system__ access policy. Because these are built into GEM itself, they cannot be removed. However, writing self-monitoring metrics to the system tenant can be turned off using the flag -instrumentation.enabled=false or the associated configuration setting.

Recording rules

In order to use self-monitoring metrics to power associated self-monitoring dashboards, the GEM ruler also includes built-in recording rules. These recording rules perform aggregations of self-monitoring metrics they same way the ruler aggregates other metrics. Because these recording rules are built-in to GEM itself, they cannot be removed. However, they can be turned off using the same flag that enables or disables self-monitoring -instrumentation.enabled=false or the associated configuration setting.

Overhead

Self-monitoring metrics are stored in GEM itself. Like any other metrics, they consume space in object storage. When enabled in microservices mode, each GEM component (ingester, querier, etc) will emit approximately 2000 series per component. These series are emitted for each component and GEM duplicates them based on the replication factor in the ingesters.

To understand how many series will be written under the __system__ tenant as part of self-monitoring, you can use the following formula:

2000 * $NUMBER_OF_GEM_PROCESSES * $REPLICATION_FACTOR

Since these series are written to GEM in a similar way to other series, they’ll be deduplicated by the compactor in object storage to reduce space required. To understand how many series will end up in object storage via the __system__ tenant, you can use the following formula:

2000 * $NUMBER_OF_GEM_PROCESSES

Monitor Grafana Mimir

Tue, 22 Apr 2025 16:36:44 +0000

Monitor Grafana Mimir

The following topics guide you in preparing your environment to display dashboards that you can use to monitor Grafana Mimir.

Monitor compactor health

Tue, 22 Apr 2025 16:43:47 +0000

Monitor compactor health

Grafana Enterprise Metrics emits several metrics related to compactor health. The following queries are useful to get a high-level view of compactor activity. For users with self-monitoring enabled, please see the GEM system monitoring / compactor dashboard, which includes panels built from these queries.

Successful compactor jobs run per hour

sum(increase(cortex_compactor_runs_completed_total[1h]))

This value should be relatively stable when viewed over a long enough period of time, for example hours or days.

Failed compactor jobs run per hour

sum(increase(cortex_compactor_runs_failed_total[1h]))

Note: Restarting the compactor process will interrupt in process compaction jobs. This will increase the value of cortex_compactor_runs_failed_total, but it is not cause for concern as long as these restarts are expected. In the event of a compactor crash, this metric will not be incremented. Compactor process crash events should be monitored separately.

Number of blocks per tenant

sum by (user) (cortex_bucket_blocks_count - cortex_bucket_blocks_marked_for_deletion_count)

This value should be relatively stable over a long enough period of time, for example several days. If the compactor is lagging behind, it will increase over time.

Monitoring bucket index health

Before enabling the bucket index, the index health can be verified by monitoring the cortex_bucket_index_last_successful_update_timestamp_seconds metric. This metric tracks the last successful bucket index update per tenant. The following query can be used to determine the index age for each tenant:

time() - cortex_bucket_index_last_successful_update_timestamp_seconds

The maximum index age should generally line up with the value of the -compactor.cleanup-interval flag.

Note: Some jitter is added to the cleanup interval to prevent all compactor replicas from running at the same moment every time the interval elapses. Additionally, the cleanup takes some time to perform. Because of this, you may see the index age slightly older than the cleanup interval. This is not cause for concern. We recommend configuring an alerting threshold when the index age exceeds (2 * compactor.cleanup-interval) + 5 minutes.