Leadership election

Mon, 18 Aug 2025 18:19:44 +0000

Leadership election

The admin API allows you to create and manage resources in your Grafana Enterprise Metrics (GEM) cluster such as tenants, access policies, and tokens. When performing operations that modify resources such as updates or deletes, the admin API requires the use of resource version numbers to make sure the most recent version of resources are being operated on. An additional safety feature the admin API makes use of to ensure concurrent modifications don’t overwrite each other is leadership election. Admin API components elect a single component to perform all modification requests, making sure that simultaneous requests don’t overwrite each other. Details about how to configure or disable this feature are provided below.

Configuration

Leadership election is a recommended feature of the GEM admin API, and is therefore enabled by default. It can be configured or disabled under the admin_api section of the GEM configuration file. Leadership election uses the same mechanism as other GEM components to coordinate among themselves, the hash ring. This means you need to configure this instance of the ring the same way you configured the ring used by distributors and ingesters when setting up your GEM cluster.

An example of this configuration is provided below using the memberlist gossip ring.

admin_api:
  leader_election:
    ring:
      kvstore:
        store: memberlist
      tokens_observe_period: 0

Note that any supported backend (Etcd, Consul, or memberlist) can be used for leader election with the admin API, memberlist is only used as an example here. For simplicity, it is recommended to use whatever ring backend you are already making use of for other GEM components.

Implementation

Admin API leadership election does not require a quorum to operate like other uses of the ring in GEM components. This means it is not required to run an odd number of admin API components. Instead, it is sufficient to run two for redundancy.

Disabling Leadership Election

Leadership election can be disabled under the admin_api section of the GEM configuration file. This is not recommended and leadership election will be enabled permanently in a future release.

admin_api:
  leader_election:
    enabled: false

Audit logging

Mon, 18 Aug 2025 18:19:44 +0000

Admin API Audit logging

An audit log is available for actions on the Admin API path. This captures any HTTP requests against /admin/api/* and outputs them to the standard logger, tagged with level=audit for filtering.

The following fields may be present on each log line depending on the API endpoint called:

traceID: The jaeger-trace-id if configured.
requestURI: The URI of the incoming request.
httpMethod: The HTTP method of the incoming request.
remoteIPAddress: The IP address of the requesting client, either the address of the proxy in front of this admin API instance (if there is one), or the original source of the request if it connects directly to this admin API instance.
forwardedIPAddress: The forwarded IP address if X-Forwarded-For is set by a proxy.
requestBody: The HTTP body sent with the request.
httpStatus: The HTTP response code sent back to the client.
reason: If available, the reason for the specific httpStatus. For example, failed authentication.
authorization: The method used for authorization.
authFromCache: true if successful authentication came from a cache check, otherwise false.
tokenID: The token ID of the authenticated user, or the subject if using OAuth integration.
accessPolicyID: The access policy ID of the requesting token.
webauth-user: The username as set by the header configured by user_header_name.
X-Grafana-Org-Id and X-Grafana-User: The respective headers as sent by the requesting client. You can configure Grafana to send this information about the user interacting with the API.

Configuration

Audit logs are disabled by default. They can be configured under the admin_api section of the GEM configuration file.

For example:

admin_api:
  auditlogging:
    enabled: true

Request body logging

By default, when Admin API audit logging is enabled, the body of the HTTP request is also logged.

As the body could potentially be large, the size of the request body is limited by the value of the -admin-api.max-request-body-size-bytes flag (default 10MiB). If this limit is exceeded, the request will be rejected with an HTTP 413 response.

Alternatively, you can disable logging the request body with the -admin-api.log-request-body flag.

Example log output

level=audit ts=2023-05-26T12:07:03.859986132Z traceID=45a25b15f51938d0 requestURI=/admin/api/v3/tenants httpMethod=POST remoteIPAddress=172.20.0.1 requestBody="{\n  \"name\": \"acme\",\n  \"display_name\": \"Acme Co.\",\n  \"created_at\": \"2023-04-13T17:37:59.341728283Z\",\n  \"status\": \"active\",\n  \"cluster\": \"enterprise-metrics\",\n  \"limits\": {\n    \"ruler_max_rule_groups_per_tenant\": 1\n  }\n}" httpStatus=200 authorization=bearer authFromCache=false tokenID=myuser accessPolicyID=admin-ap

GEM Admin API on Grafana Labs

Leadership election