Integrate with Graphite using the proxy on Grafana Labs

Architecture overview

Tue, 22 Apr 2025 16:43:47 +0000

Architecture overview

This page provides an overview of the architecture of the Graphite proxy services and how they interact with the other GEM services.

Note that the “Graphite deployment” shown in the architecture diagram is optional, but without it the number of functions which are available in the Graphite query language is limited to a subset of what standard Graphite provides.

API

The Graphite proxy in GEM exposes the same write and read endpoints as Grafana Cloud Graphite. All endpoints have a /graphite/ prefix.

If using access policies, write endpoints require the metrics:write scope and read endpoints require the metrics:read scope.

Metrics ingestion

This section explains the components of a typical ingestion pipeline. The use of Carbon-relay-ng is optional, if the metrics producer has the ability to send its metrics in the right format via HTTP(S) directly to the Graphite write proxy then Carbon-relay-ng could be omitted, but since Carbon-relay-ng comes with lots of useful features such as buffering of failed writes it is recommended to use it.

Carbon-relay-ng

Carbon-relay-ng is a relay for carbon streams, to make it forward data to the Graphite proxy a gnet route needs to be configured and pointed at the Graphite ingest API (<GEM gateway URL>/graphite/metrics) on the Graphite write proxy. It comes with a wide variety of features, for a detailed documentation of them refer to Carbon-relay-ng. The official documentation also comes with an example configuration which demonstrates the usage a gnet route, refer to grafanaNet route.

Graphite write proxy

The Graphite write proxy accepts the incoming requests which Carbon-relay-ng is sending and translates the Graphite metrics into Prometheus metrics, then it forwards them to the GEM distributors. For more information about how the process works, refer to Graphite write proxy.

Metrics querying

This section explains the components that are used to query Graphite metrics.

Graphite Querier

The Graphite querier provides the Graphite render API, for more information about the API refer to the Graphite documentation. It fetches the data required to serve a query from GEM, performs the configured aggregations on the data at query time and runs the requested Graphite functions on the aggregated data. For more information about how the process works, refer to Graphite querier.

Grafana datasource

If you are using the Graphite datasource to query the Graphite proxy, use the URL <GEM gateway URL>/graphite.

Graphite

Open source Graphite can be used by the Graphite querier to off-load the Graphite function processing of query functions that are not implemented in its own native function library, that way all Graphite functions are available to use.

You can run the Graphite querier without providing it any open source Graphite deployment. However, without the Graphite deployment the set of query functions that are available in the Graphite query language will be limited to a reduced set of very commonly used functions. In order to gain function completeness a Graphite deployment is required.

Use the Graphite querier

Tue, 22 Apr 2025 16:43:47 +0000

Use the Graphite querier

The Graphite querier provides the Graphite querying API, for more information about the API refer to the Graphite documentation.

Supported Graphite functions

The Graphite querier comes with a set of built-in functions which tend to execute faster than the original Graphite implementation, we refer to these built-in functions as the “native query engine”. The native query engine only implements a set of very commonly used functions, many of the less commonly used functions are missing from it. Whenever the Graphite querier receives a query which includes a function that it doesn’t have in the native query engine it offloads the function processing to a stock Graphite deployment (if available). To learn more about how the Graphite querier offloads the function processing to a separate Graphite process, refer to Offloading the function processing to Graphite.

The native query engine is imported from an open source project called CarbonAPI, for more information about its function library and which functions are/aren’t implemented, refer to the CarbonAPI repository.

Query handling

The query handling endpoint accepts Graphite queries, it processes them in the following steps. This is a simplification which ignores the fact that some steps in the process are cached.

Parsing the query

The query gets parsed and based on the used functions a decision is made which query engine shall be used. Whenever all used functions are supported by the native query engine then this one is used, otherwise the Graphite function processing will be off-loaded to the external Graphite process.

The metric name patterns get extracted from the query and a Prometheus query gets generated to fetch the required data to serve the query from Grafana Mimir. To generate that Prometheus query the name mapping schemes get applied in reverse, for more information about the name mapping schema refer to Graphite write proxy.

Breaking the query up into sub-queries

If the original query is requesting a long time range, then it gets broken up into sub-queries. Each sub-query has a maximum time range of 1d by default, configurable via the flag graphite.querier.split-queries-by-interval, the time range boundaries are aligned to multiples of the split interval. The default sub-query length of 1d is derived from the fact that by default the blocks in the store have a length of 1d after compaction, that way each sub-query processes exactly one block.

Each sub-query gets assigned two points budgets defining how many data points it may produce, the two budgets are called soft budget and hard budget. The points budgets per sub-query get calculated based on the total query budgets configured via the flags graphite.querier.max-points-per-req-{soft,hard} which by default have the values 1000000 and 20000000, each sub-query gets an equal share which is proportionate to the total queried time range.

The soft budget defines to what number of points the results should optimally be aggregated, but it may be exceeded if it is not possible to stay within the soft budget due to the configured retentions in the storage-schemas. If the number of points can’t be reduced to fit within the hard budget then the query directly errors to prevent overloading of the query engine.

Note that the sub-queries are only responsible for fetching and processing the raw data in preparation for the query engine to run on top of the data. The query engine will later run on the combined sub-query results and not on each sub-query result separately.

Processing of the sub-queries

The sub-queries which resulted from breaking up the original query get processed concurrently by a pool of worker threads. The size of the work pool is configurable via the flag graphite.querier.query-handling-concurrency, with a default value of 8. Increasing the worker pool size can speed up the processing of queries with long time ranges because they will be processed at a higher concurrency as long as enough physical cores are available, but it will also increase the memory consumption.

Fetching the data

Each sub-query worker separately fetches the data which it requires to process its sub-query from Grafana Mimir. This means that if one Graphite query gets broken up into 5 sub-queries then Grafana Mimir sees 5 separate queries, so it is expected that the Grafana Mimir query rate might be higher than the Graphite query rate.

Choosing the target interval for each metric

Each metric will get aligned to an interval that is determined based on the tenant’s storage-schemas configuration. First the metrics get grouped by the schemas of which they match the pattern, these groups are called rollup groups because they always get rolled up (horizontally aggregated) together, then each rollup group’s target interval gets determined based on the queried time range and the schema configuration. For this the time range of the original Graphite query is used and not the time range of the sub-query. To learn more details about the process of determining the target interval, refer to Storage schemas.

After each rollup group has an assigned target interval the total number of points which this sub-query is going to generate gets calculated based on the number of metrics in each rollup group, their assigned target intervals, and the time range of the sub-query.

If the total number of points which will get generated by this sub-query exceeds the sub-query’s soft budget then the rollup group which currently has the lowest target interval gets bumped to the next retention of its matching storage schema. Since the retentions are always defined in increasing interval order, bumping a rollup group to the next retention results in less points being generated. This process gets repeated until either the total number of points generated by the sub-query fits in the soft budget or all rollup groups have already been bumped to their final retention.

If the total number of points which will get generated by this sub-query still exceeds the sub-query’s hard budget then an error gets returned and the query processing gets aborted.

Alignment and aggregation of the data

The data points of each metric now get aligned to the target interval of the rollup group which the metric is in. If there are data points with time stamps that are not a multiple of the target interval then they get aligned to multiples of the interval by shifting them to a lower time stamp. If there are multiple data points within one interval then they get aggregated together by the aggregation function chosen based on the storage-aggregation’s configuration of the tenant, the resulting value will be assigned with a timestamp that is a multiple of the target interval. To learn more about the storage-aggregation refer to Storage aggregations.

This process is cached, but the caching logic is omitted in this description for the sake of simplicity, to learn more about the caching logic refer to Caching.

Combining the sub-query results

Each sub-query returns a set of metrics that has been aligned and aggregated to the correct interval, now the results of the different sub-queries get combined into one set of metrics.

If a sub-query result contains a metric which is not present in another sub-query result then the gap is filled with NaN.

At this point it is possible that the same metric has been aggregated to different intervals in the different sub-queries, because if one sub-query resulted in a larger number of metrics than another then its retention might have been bumped to fit the number of points generated by the sub-query into its soft budget In this situation the sub-query results with the lower interval get aggregated to match the interval of the sub-query result with the higher interval, because the Graphite query engines require each metric to have a constant interval.

The result after combining the sub-query results is one set of metrics, where each metric is guaranteed to have a consistent interval and each metric has data points filling the entire queried time range because all gaps have been filled with NaN values.

Function processing

The combined sub-query results now get passed into the query engine that has been chosen in the beginning of the query handling process. This can either be the native query engine (preferred) or a an optional Graphite deployment running in a separate process (for function completeness). To learn more about how the Graphite querier offloads the function processing to a separate Graphite process, refer to Offloading the function processing to Graphite.

Returning to the user

The query engine returns a set of metrics which has been generated by running the Graphite functions specified in the query on the combined sub-query results, this set of metrics now gets returned to the user.

Illustration

This is an illustration of the above described query handling process:

Caching

The aggregation work which is performed as part of the query handling gets cached in order to minimize the latency of queries that request the same metrics with overlapping time ranges multiple times. The caching happens in chunks of data, where each chunk has a size of 1d by default, configurable via -graphite.querier.split-queries-by-interval. The boundaries of the cached chunks are always multiples of the chunk size in UTC, meaning that each chunk contains the data of one day from midnight to midnight in UTC by default. Partial chunks don’t get cached, they get regenerated at every query.

Imagine a Grafana dashboard querying a given set of metrics with a constant time range length of 3d applied relative to the current time.

The first query requests the time range 2021-01-10T13:25:00Z - 2021-01-12T13:25:00Z
- The sub-query result for the time range 2021-01-10T13:25:00Z - 2021-01-11T00:00:00Z gets generated but it can’t be cached because it is partial
- The sub-query result for the time range 2021-01-11T00:00:00Z - 2021-01-12T00:00:00Z gets generated and cached
- The sub-query result for the time range 2021-01-12T00:00:00Z - 2021-01-12T13:25:00Z gets generated but it can’t be cached because it is partial
The Grafana dashboard refreshes again 1min later
Now the new query is requesting the time range 2021-01-10T13:26:00Z - 2021-01-12T13:26:00Z
- The sub-query result for the time range 2021-01-10T13:26:00Z - 2021-01-11T00:00:00Z gets generated but it can’t be cached because it is partial
- The sub-query result for the time range 2021-01-11T00:00:00Z - 2021-01-12T00:00:00Z gets retrieved from the cache
- The sub-query result for the time range 2021-01-12T00:00:00Z - 2021-01-12T13:26:00Z gets generated but it can’t be cached because it is partial

This means the data fetching, the alignment and the aggregation of the data of the entire day 2021-01-11 is omitted. The caching is especially effective for queries that query for long time ranges. There are two caches involved in this, they are named metric name cache and aggregation cache.

Metric name cache

The metric name cache caches the resolution of metric name patterns that are used in the queries into lists of concrete metric names. Each entry in the metric name cache contains a list of metric names corresponding to a combination of the following attributes:

Metric name pattern
Time range

Aggregation cache

The aggregation cache caches the aligned and aggregated results of sub-queries on a per-metrics basis. Each entry in the aggregation cache contains a chunk of data corresponding to a combination of the following attributes:

Metric name
Time range with a length of 1d by default
Target interval
Aggregation function used

Cache usage

This illustrates the flow how these two caches are used in the query handling process.

Offloading the function processing to Graphite

The Graphite querier’s native query engine only implements an incomplete set of Graphite functions, whenever a query involves a Graphite function that is not implemented in the native query engine then the Graphite querier offloads the function processing to a Graphite deployment which runs in a separate process.

It does this by forwarding the original query to the Graphite process, just like a simple reverse HTTP Proxy. The Graphite process is configured to use the Graphite querier as a cluster server, because it doesn’t have the necessary data to serve the query locally it will then reach out to the Graphite querier to fetch the raw data via a separate call to the /graphite/render endpoint which queries only for the data without any Graphite functions in the query.

The Graphite querier will then handle this second query which it received just like any other query and it will return the queried data to the client, which in this case is the Graphite process. In the process of handling this query the Graphite querier will still perform all the necessary aggregations on the queried data and aggregate it based on the configured storage-schemas, storage-aggregations and the max-points-per-req-soft parameter.

The Graphite process then executes the functions specified in the query on the data it received from the Graphite querier and returns the result back to the Graphite querier by replying to the original request that the Graphite querier has forwarded to it.

The Graphite querier then proxies the response from the Graphite process back to the user.

We recommend using this image to deploy the Graphite process, because it allows the configuration of the CLUSTER_SERVERS via the env var GRAPHITE_CLUSTER_SERVERS:

docker.io/raintank/graphite-mt:8-7ebde4c60

A valid configuration example via the environment variables in a Kubernetes deployment looks like this, this assumes that the Graphite querier is reachable on the address graphite-querier on port 80:

        - name: GRAPHITE_CLUSTER_SERVERS
          value: graphite-querier:80/graphite

The chain of calls looks like this, where the graphite-querier might or might not be the same process:

client -> graphite-querier -> graphite -> graphite-querier -> GEM Querier

Querying the remote read API

For enhanced internal query performance, the Graphite querier performs the queries against the Prometheus remote read API. The only configuration required is the query address, as described in this configuration example:

    -graphite.querier.query-address: 'http://<current server-http-listen-address>:<current server-http-listen-port>',

Regarding graphite.querier.query-address we should make some clarifications:

The address should point to the Grafana Mimir query frontend to benefit from its capabilities.
If the Query Frontend or Grafana Mimir is configured with an http prefix, then you need to add it at the end of the query address to make it consistent. Otherwise, graphite won’t be able to query the right querier address.
If you are running the single binary installation then this address will be localhost and unless you have a specific http prefix then you can skip the -graphite.querier.query-address flag entirely.

Use the Graphite write proxy

Tue, 22 Apr 2025 16:43:47 +0000

Use the Graphite write proxy

The Graphite write proxy accepts the ingest requests sent by Carbon-relay-ng and then translates the incoming Graphite metrics into Prometheus metrics. Once the metrics have been translated into Prometheus they get forwarded to the GEM distributor, which handles them the same way as it would handle any other Prometheus remote_write requests. The Graphite to Prometheus metric translation differentiates between untagged Graphite metrics and tagged Graphite metrics, the name mapping scheme for the two looks as following:

Untagged Graphite metrics

Graphite metric: some.test.metric

In Prometheus: graphite_untagged{__n000__="some", __n001__="test", __n002__="metric"}

Tagged Graphite metrics

Graphite metric: some.test.metric;my_tag=my_value;another_tag=another_value

In Prometheus: graphite_tagged{name="some.test.metric", my_tag="my_value", another_tag="another_value"}

Writing metrics with the remote write API

For an enhanced internal write performance we’re introducing a new feature that will become the default write strategy in future releases. This is a recommended internal tweak for the graphite write proxy to use Grafana Mimir remote write API. To enable it, pass the following configuration:


-graphite.write-proxy.remote-write-enabled: 'true',
-graphite.write-proxy.write-endpoint: 'http://<distributor-address>:<distributor-port>/api/v1/push',
-graphite.write-proxy.skip-label-validation: 'true',

Regarding graphite.write-proxy.write-endpoint we should make some clarifications:

The address should point to the Grafana Mimir distributors.
If the there is a gateway in front of the distributors with a http prefix, then you need to add it at the end of the query address to make it consistent. Otherwise graphite won’t be able to write against the right endpoint.
If you are running the single binary installation then this address will be http://localhost:<local-installation-port>/api/v1/push.

Work with schemas

Tue, 22 Apr 2025 16:43:47 +0000

Work with schemas

The schemas define how data is aggregated before the Graphite query functions process the aggregated data. There are two types of schemas:

Storage-schemas define to what interval data should be aggregated and also up to which point in the past data should be available for querying.
Aggregation-schemas define what aggregation function should be used to aggregated the data.

Each of the two schemas has a global default which can be configured via the following flags:

-graphite.querier.schemas.default-storage-schemas-file
-graphite.querier.schemas.default-storage-aggregations-file

If the config flag -graphite.querier.schemas.enable-user-overrides is enabled then the global defaults can be overridden on a per-tenant basis.

Querying and updating a tenant’s schemas

Each of the two schema types has an API endpoint that can be used to query it and to update it, the endpoints are:

/graphite/config/storageSchemas
/graphite/config/storageAggregations

This is an example request to the Graphite querier which gets the current storage-schemas of the tenant 12345:

~$ curl -H 'X-Scope-OrgId: 12345' http://<graphite-querier>/graphite/config/storageSchemas
[default]
pattern    = .*
intervals  = 0:1s
retentions = 1m:5w

~$

This is an example request to the Graphite querier which updates the storage-schemas of the tenant 12345:

~$ cat storage-schemas.conf
[default]
pattern    = .*
intervals  = 0:1s
retentions = 1m:5w,2h:1y

~$ curl -H 'X-Scope-OrgId: 12345' --data-binary '@storage-schemas.conf' http://<graphite-querier>/graphite/config/storageSchemas
~$

The tenant schemas get stored in the object store bucket configured with the configuration flags prefixed by -graphite.querier.schemas.

Schema caching

When a query is received the Graphite querier needs to obtain the tenant’s schemas, in order to avoid getting the schema from the object store for every single query the schemas get cached in the process memory by default for 1h, this is configurable via -graphite.querier.schemas.schema-ttl.

Proactive cache refreshing

When the Graphite querier gets a schema from its local cache in order to handle a query it checks whether this cache entry is already expired and whether it is in the first or in the second half of its cache life time. If the entry is expired then it doesn’t get used and the latest schema gets fetched from the object store, the obtained schema is then used to handle the query and to populate the cache. If the entry is not expired but it is already in the second half of its cache life time then the cached value is used, but at the same time an asynchronous job is kicked off which fetches the latest schema from the object store and then populates the cache with it.

This pro-active cache refreshing mechanism prevents that queries ever get blocked on the fetching of the latest schema from the object store, assuming that the tenant submits queries relatively frequently. In the case where a tenant only submits queries very infrequently it is possible that their query handling gets blocked on the fetching of the latest schemas.

Schema structure and syntax

Each of the two schema types (storage-schemas/storage-aggregations) consists of a list of schemas where each schema has a pattern property containing a regular expression. When the Graphite querier looks up the schema for a given metric it loops over the list of schemas and matches their patterns against the given metric name, the first schema where the pattern matches is going to be used, this means that the order of the schemas within the configuration files matters.

It is common practice that the last schema in each file is named default and has the pattern .*, this is a catch-all which matches all metrics that haven’t matched any of the previous schemas. If there is no default schema and none of the defined schemas matches, then a hard-coded default is used.

Storage schemas

The format of the storage schemas is very similar to standard Graphite, with two additional parameters. For the documentation of the format refer to storage-schemas.conf.

Example:

[apache_busyWorkers]
pattern = ^servers\.www.*\.workers\.busyWorkers$
retentions = 15s:7d,1m:21d,15m:5y

This example defines that:

All metrics matching this regular expression will be available for querying for up to 5y
If a query only queries the most recent 7d of data then the data is available at an interval of 15s
If a query queries the most recent 21d of data then the data is available at an interval of 1m
If a query queries the most recent 5y of data then the data is available at an interval of 15m
If a query queries for more than 5y, it will only get data returned for the most recent 5y

Additions to the standard storage schemas

The Graphite querier supports the standard format of Graphite’s storage-schemas and storage-aggregation config files. In addition to the standard properties, it supports two more parameters in the storage-schemas config file which are specific to the Graphite querier.

Intervals

The intervals parameter can be used to define a minimum interval with which the data of a certain static time range must be returned. This is useful in cases where the retentions would potentially assign an interval to data that is lower than the real interval of the stored data, for example because the data in the store has been generated according to an older version of the storage-schema which has been updated in the meantime. In such a scenario the gaps between the stored points would get filled with math.NaN values in order to enforce the interval defined by the retentions, which can lead to unexpected query results.

The intervals parameter is a list of absolute time ranges, each range has an interval associated with it which is the minimum interval at which this data shall be returned. Each time range is defined by its beginning and it ends at the beginning of the following time range, the last time range is open ended into the future.

Example:

100:15s,200:30s,300:15s This string defines that:

The data in the time range 100-200 must be returned with an interval of at least 15s
The data in the time range 200-300 must be returned with an interval of at least 30s
The data in the time range 300-<unlimited> must be returned with an interval of at least 15s

Keep in mind that in Graphite it is a requirement that each metric which gets passed into the Graphite query engine has a constant interval, this means if a user would query the time range 100-500 then the highest minimum interval of that time range (30s) would effectively become the minimum interval for the entire queried range.

The minimum interval that has been determined according to the intervals parameter is then used in the selection of the retention interval at which the data shall be returned.

For example if the intervals parameter defines a minimum interval of 30s like in the above example and the defined retentions which could be used for a query are 15s:7d,15m:5y based on the time range then the Graphite querier will pick the second retention and return the data at an interval of 15m, because even though the first retention would be valid according to the queried time range its interval is lower than the minimum interval of 30s defined by intervals.

An example schema with the intervals parameter might look like this:

[default]
pattern = .*
intervals = 1594166400:30min,1625702400:15s
retentions = 15s:7d,15m:5y

Applying retentions relative to end of queried time range

The parameter relativeToQuery is an optional flag which can be added to a storage schema, when it is not defined then its default value is false.

If set to true then this flag causes that the defined retentions do not get applied relative to the current wall clock time, but instead they get applied relative to the end of the queried time range. This means that for example with a retentions settings of 15s:7d a time range of 7d can be queried and gets returned with an interval of 15s even if the query requests the time range from now()-1y-7d to now()-1y because in that situation the 7d retention would get applied relative to now()-1y.

An example schema with this parameter might look like this:

[default]
pattern = .*
relativeToQuery = true
retentions = 15s:7d,15m:5y

Storage aggregations

The format of the storage aggregations is exactly the same as in standard Graphite.

For the documentation of the format refer to storage-aggregation.conf.

A valid storage aggregations entry might look like this example:

[all_min]
pattern = \.min$
xFilesFactor = 0.1
aggregationMethod = min

Troubleshoot the Graphite proxy

Tue, 22 Apr 2025 16:43:47 +0000

Troubleshoot the Graphite proxy

You might run into some common issues when running the Graphite Proxy in a production environment. To learn how to react to each issue, keep reading.

Metrics ingestion

Common metrics ingestion issues such as high error rates and high latency.

High error rate on ingest API endpoint

The query that follows shows the error rate of the ingest requests. To run the query replace <job> with the string that corresponds to the Grafana Mimir gateway job or the Graphite write proxy job.

sum by (status_code) (
  rate(
    cortex_request_duration_seconds_bucket{job="<job>", route="graphite_metrics"}[5m]
  )
)

Check if the GEM distributors are returning errors to ingest requests submitted by the Graphite write proxy.

If so, these errors will be forwarded by the Graphite write proxy.

Check if Graphite write proxy instances are running out of memory.

If so, either increase their memory limits or increase the number of instances.

Look at the logs to find more specific errors.

High latency on ingest requests

The query that follows displays the latency of the ingest requests in percentiles. In the query, replace <job> with the string that corresponds to the Grafana Mimir gateway job or the Graphite write proxy job. Also, replace the <percentile> with the percentile that you want to see.

histogram_quantile(
  <percentile>,
  sum by (le) (
    rate(
      cortex_request_duration_seconds_bucket{job="<job>", route="graphite_metrics"}[5m]
    )
  )
)

Check if the increased latency comes from the GEM distributors to which the Graphite write proxy is sending its ingest requests.

If the distributor’s ingest latency is high, then this will also raise the ingest latency of the Graphite write proxy. Therefore, you should try to reduce the distributor latency.

Scale the Graphite write proxy to a higher number of replicas.

Doing so might reduce the latency in some situations, but it will not help if the distributors are slow at handling the ingest requests coming from the Graphite write proxy.

Metrics querying

This section covers issues related to metrics querying.

High error rate on queries

The error rate of queries can be seen with the following query. The <job> will need to be replaced either by the cortex-gw job or by the Graphite querier job.

sum by (status_code) (
  rate(
    cortex_request_duration_seconds_bucket{job="<job>", route="graphite_render"}[5m]
  )
)

Check if the GEM queriers are returning errors to queries submitted by the Graphite querier.

If so these errors will result in errors being returned to the user.

Check if Graphite querier instances are running out of memory. If so you have the following options:

Give the Graphite queriers more memory, either by increasing the number of instances or by raising the memory limit of each instance.

It is important to keep in mind that each query gets executed on a single Graphite querier instance. This means that if the rate of queries is low, but each of them consumes really a lot of memory, then increasing the number of Graphite querier instances might not help because each of them might already process only a single query at a time.

In this situation it would be better to give each instance of the Graphite querier more memory instead of increasing their number.

Reduce the concurrency at which sub-queries are processed in the Graphite querier. To adjust the concurrency you can update the value of the flag -graphite.querier.query-handling-concurrency.
Reduce the number of points to which each result gets aggregated. This might reduce the peak memory usage of the query handling process, but its effectiveness depends on the specific queries. To reduce the number of points which the Graphite querier generates you can use the limit defined via the flag -graphite.querier.max-points-per-req-soft.
Reject heavy queries. This will result in an error returned to the user if the submitted query was too heavy, but at least it can prevent out of memory conditions, which means that other queries can still be processed. To reject heavy queries you have the following options:
Reduce the number of points each query may produce at max, to do so you can use the flag -graphite.querier.max-points-per-req-hard.
Leverage the limits which GEM provides, such as max_fetched_series_per_query.

If a stock Graphite deployment is used for queries that can’t be handled by the Graphite querier, check if Graphite runs out of memory.

If so you can either increase its memory limit or the number of Graphite instances. Just like with the Graphite querier each query gets executed on a single Graphite instance, which means that if you already scaled their number to a level where no two queries get executed concurrently in the same instance then further increasing their number won’t help because the additional instances will be idle.

Look at the logs to find more specific errors.

High latency on queries

The latency on queries in percentiles can be seen with the following query.

histogram_quantile(
  <percentile>,
  sum by (le) (
    rate(
      cortex_request_duration_seconds_bucket{job="<job>", route="graphite_render"}[5m]
    )
  )
)

Check if the increased latency comes from the GEM querier which is used by the Graphite querier.

If the querier is slow then this will also raise the latency of the query handling in the Graphite querier. In this case, try to improve the querier latency.

Check if there is a change in the properties of the queries sent to the Graphite querier, which could lead to the queries being slower to process.

The following query can be used to see the lengths of the time ranges which are being queried, the string <job> must be replaced with the job of the Graphite querier and <percentile> must be replaced with the percentile that you want to see:

histogram_quantile(<percentile>, avg by (le) (rate(graphite_time_range_length_seconds_bucket{job="<job>"}[5m])))

If there is an increase in this metric then users are querying for longer time ranges, then:

a. Check the Memcached eviction rate in the metric name cache and the aggregation cache, to ensure that the cache efficiency is not reduced due to evictions.

If there are lots of evictions then increasing the available memory in Memcached might help.

It might help to increase the Graphite querier concurrency because this will allow it to process the data required to serve queries in more threads concurrently, but keep in mind that this might increase the memory usage as well so watch out for restarts due to out of memory conditions. To do so use the flag -graphite.querier.query-handling-concurrency.

b. The following query can be used to see how many series each Graphite query involves before the function processing, the string <job> must be replaced with the job of the Graphite querier and <percentile> must be replaced with the percentile that you want to see:

histogram_quantile(<percentile>, avg by (le) (rate(graphite_series_per_query_by_phase_bucket{phase="passed_into_graphite_functions", job="<job>"}[5m])))

If there is an increase in this metric then the queries are requiring more series to be processed per query, it may help to adjust the GEM limit max_series_per_query to reject the heavy queries.

Check if more queries are being processed by Graphite instead of the native query engine.

Generally Graphite tends to be much slower than the native query engine, so if queries get processed by Graphite then this can lead to a significant latency increase. To see how many queries get processed by Graphite you can use the following query, the string <job> must be replaced with the Graphite querier’s job:

rate(graphite_proxied_to_graphite{job="<job>"}[1m])

The latency of the Graphite query processing can be seen with the following query, the string <job> must be replaced with the job of the Graphite querier and <percentile> must be replaced with the percentile that you want to see:

histogram_quantile(<percentile>, rate(graphite_proxied_to_graphite_duration_seconds_bucket{job="<job>"}[1m]))

In this situation it might help to scale Graphite to a higher number of processes, there are two things to keep in mind though:

Each query is handled by only one instance of the Graphite process, so once you scaled it to a level where no Graphite instance handles more than one query concurrently then further increasing the number of instances will not help because any additional Graphite instances would just be idle.
Increasing the number of available cores per Graphite process tends to not benefit the latency much because of how Graphite schedules its worker threads. Instead of increasing the cores it is better to increase the number of Graphite instances.

Query results look different from what you would expect

A user reports that they submit a query and they get a result back, but the result looks different from what they’d expect. This could happen due to a wide variety of issues, but one common issue is that the storage-schema or storage-aggregation configurations are incorrect.

A wrong storage-aggregation configuration can lead to results which look incorrect, a very common mistake is that the used aggregation method is not appropriate for the type of data which gets horizontally aggregated in the query handling.

For example a metric of type gauge should never be summed when horizontally aggregated because the resulting values will not be useful.

To read more about the storage-aggregation configuration refer to storage aggregations.

If the target interval which gets determined based on the storage-schema configuration is lower than the interval of the raw data stored in Grafana Mimir, then this can also lead to results which may be wrong in various different ways depending on the Graphite functions used in the query.

For more information about the storage-schema configuration refer to storage schemas.