---
title: "Audit data propagation latency and correctness using Loki Canary | Grafana Enterprise Logs documentation"
description: "Describes how to use Loki Canary to audit the log-capturing performance of a Grafana Loki cluster to ensure Loki is ingesting logs without data loss."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Audit data propagation latency and correctness using Loki Canary

Loki Canary is a standalone app that audits the log-capturing performance of a Grafana Loki cluster.  
This component emits and periodically queries for logs, making sure that Loki is ingesting logs without any data loss. When something is wrong with Loki, the Canary often provides the first indication.

Loki Canary generates artificial log lines. These log lines are sent to the Loki cluster. Loki Canary communicates with the Loki cluster to capture metrics about the artificial log lines, such that Loki Canary forms information about the performance of the Loki cluster. The information is available as Prometheus time series metrics.

```
graph LR
    subgraph nodes["x Node"]
        lc["loki-canary"] --> lf["log file"] --> al["Alloy"]
    end

    al -->|push| loki
    lc -->|websocket| loki
```

Loki Canary writes a log to standard output and stores the timestamp in an internal array. The contents look something like this:

nohighlight ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```nohighlight
1557935669096040040 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
```

The relevant part of the log entry is the timestamp; the `p`s are just filler bytes to make the size of the log configurable.

Loki Canary’s standard output should be captured and written to a file. An agent (like Grafana Alloy) should be configured to read the log file and ship it to Loki.

Meanwhile, Loki Canary will open a WebSocket connection to Loki and will tail the logs it creates. When a log is received on the WebSocket, the timestamp in the log message is compared to the internal array.

If the received log is:

- The next in the array to be received, it is removed from the array and the (current time - log timestamp) is recorded in the `response_latency` histogram. This is the expected behavior for well behaving logs.
- Not the next in the array to be received, it is removed from the array, the response time is recorded in the `response_latency` histogram, and the `out_of_order_entries` counter is incremented.
- Not in the array at all, it is checked against a separate list of received logs to either increment the `duplicate_entries` counter or the `unexpected_entries` counter.

In the background, Loki Canary also runs a timer which iterates through all of the entries in the internal array. If any of the entries are older than the duration specified by the `-wait` flag (defaulting to 60s), they are removed from the array and the `websocket_missing_entries` counter is incremented. An additional query is then made directly to Loki for any missing entries to determine if they are truly missing or only missing from the WebSocket.

This direct query is repeated every `-pruneinterval` (defaulting to 60s) for as long as an entry remains missing. Once an entry has been missing for longer than the duration specified by the `-max-wait` flag (defaulting to 5m), Loki Canary gives up on it, removes it from the list, and increments the `missing_entries` counter.

## Additional Queries

### Spot Check

Starting with version 1.6.0, the canary will spot check certain results over time to make sure they are present in Loki, this is helpful for testing the transition of inmemory logs in the ingester to the store to make sure nothing is lost.

`-spot-check-interval` and `-spot-check-max` are used to tune this feature, `-spot-check-interval` will pull a log entry from the stream at this interval and save it in a separate list up to `-spot-check-max`.

Every `-spot-check-query-rate`, Loki will be queried for each entry in this list and `loki_canary_spot_check_entries_total` will be incremented, if a result is missing `loki_canary_spot_check_missing_entries_total` will be incremented.

The defaults of `15m` for `spot-check-interval` and `4h` for `spot-check-max` means that after 4 hours of running the canary will have a list of 16 entries it will query every minute (default `spot-check-query-rate` interval is 1m), so be aware of the query load this can put on Loki if you have a lot of canaries.

**NOTE:** if you are using `out-of-order-percentage` to test ingestion of out-of-order log lines be sure not to set the two out of order time range flags too far in the past. The defaults are already enough to test this functionality properly, and setting them too far in the past can cause issues with the spot check test.

When using `out-of-order-percentage` you also need to make use of pipeline stages in your Alloy configuration in order to set the timestamps correctly as the logs are pushed to Loki. The [Alloy `loki.process`](/docs/alloy/latest/reference/components/loki/loki.process/) docs have examples of how to do this.

### Metric Test

Loki Canary will run a metric query `count_over_time` to verify that the rate of logs being stored in Loki corresponds to the rate they are being created by Loki Canary.

`-metric-test-interval` and `-metric-test-range` are used to tune this feature, but by default every `1h` the canary will run a `count_over_time` instant-query to Loki for a range of `24h`. The first metric test runs at a random time within the first `-metric-test-interval` after startup, so that many canaries started at the same time don’t all query Loki simultaneously; subsequent tests run on the fixed interval.

If the canary has not run for `-metric-test-range` (`24h`) the query range is adjusted to the amount of time the canary has been running such that the rate can be calculated since the canary was started.

The canary calculates what the expected count of logs would be for the range (also adjusting this based on canary runtime) and compares the expected result with the actual result returned from Loki. The expected count is stored in the gauge `loki_canary_metric_test_expected` and the count Loki returned is stored in the gauge `loki_canary_metric_test_actual`. To see the *difference*, subtract one gauge from the other, which is what the Loki Canary dashboard in the Loki mixin does.

It’s expected that there will be some deviation, the method of creating an expected calculation based on the query rate compared to actual query data is imperfect and will lead to a deviation of a few log entries.

It’s not expected for there to be a deviation of more than 3-4 log entries.

### Cache Test

Loki Canary can also check whether Loki’s query result cache is returning results that are consistent with an uncached query. This is helpful for catching cache invalidation bugs, where a cached response no longer matches what Loki would return if the query were run again from scratch.

`-cache-test-interval`, `-cache-test-range`, and `-cache-test-now` are used to tune this feature. By default, every `15m` the canary runs the same `count_over_time` instant-query twice against a fixed point in the past: once normally, and once with a `Cache-Control: no-cache` header to bypass the cache. `-cache-test-range` (default `24h`) sets the range of the query, and `-cache-test-now` (default `1h`) sets how far back from the current time the query’s execution time (`--now`) is set, so that the test queries a stable window of data rather than a constantly moving one.

Each attempt increments `loki_canary_cache_test_query_results_total`, labeled `status="success"` or `status="failure"`. If the cached and uncached results differ, `loki_canary_cache_test_query_results_diff_total` is incremented.

Because the query window cannot start before the canary process itself started, the cache test is skipped (with a message logged to standard error) until `-cache-test-now` has elapsed since startup.

### Control

Loki Canary responds to two endpoints to allow dynamic suspending/resuming of the canary process. This can be useful if you’d like to quickly disable or reenable the canary. To stop or start the canary issue an HTTP GET request against the `/suspend` or `/resume` endpoints.

### Metrics

Loki Canary exposes all of its metrics on `/metrics` in Prometheus format, on the port set by `-port` (default `3500`). In addition to the metrics named earlier on this page, Loki Canary exposes:

- `loki_canary_entries_total`: total count of log entries written by the canary.
- `loki_canary_duplicate_entries_total`: count of log entries received more than once over the WebSocket.
- `loki_canary_unexpected_entries_total`: count of log entries received that were not expected, for example because they were already reported missing.
- `loki_canary_ws_reconnects_total`: count of WebSocket reconnections.
- `loki_canary_ws_pings_total`: count of ping messages received on the WebSocket connection.
- `loki_canary_metric_test_request_duration_seconds`: histogram of how long the metric test query took to run.
- `loki_canary_spot_check_request_duration_seconds`: histogram of how long the spot check query took to run.

## Installation

### Binary

Loki Canary is provided as a pre-compiled binary as part of the [Loki Releases](https://github.com/grafana/loki/releases) on GitHub. Each release publishes a `loki-canary` archive for common platforms.

1. Download and unpack the archive that matches your platform from the [releases page](https://github.com/grafana/loki/releases).
2. Make the binary executable and move it onto your `PATH`:
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   chmod +x loki-canary
   sudo mv loki-canary /usr/local/bin/loki-canary
   ```

The address of Loki is required. Either pass `-addr` or set the `LOKI_ADDRESS` environment variable; if neither is set, Loki Canary prints `Must specify a Loki address with -addr or set the environment variable LOKI_ADDRESS` and exits.

### Docker

Loki Canary is also published as a Docker container image to Docker Hub as [`grafana/loki-canary`](https://hub.docker.com/r/grafana/loki-canary). Image tags track Loki releases, so use the tag that matches the Loki version you run.

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
# change tag to the most recent release
$ docker pull grafana/loki-canary:3.7.3
```

The image entrypoint is the `loki-canary` binary, so any arguments after the image name are passed straight to the canary:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
docker run --rm \
  -p 3500:3500 \
  grafana/loki-canary:3.7.3 \
  -addr=loki:3100 \
  -labelname=instance \
  -labelvalue=loki-canary-1
```

The metrics port (default `3500`) must be published so that Prometheus or Alloy can scrape it.

### Kubernetes

To run on Kubernetes, you can do something simple like:

shell ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```shell
kubectl run loki-canary --image=grafana/loki-canary:latest --restart=Never \
  --image-pull-policy=IfNotPresent --labels=name=loki-canary -- -addr=loki:3100
```

Or you can do something more complex like deploy it as a DaemonSet, there is a Tanka setup for this in the `production` folder, you can import it using `jsonnet-bundler`:

shell ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```shell
jb install github.com/grafana/loki/production/ksonnet/loki-canary
```

Then in your Tanka environment’s `main.jsonnet` you’ll want something like this:

jsonnet ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```jsonnet
local loki_canary = import 'loki-canary/loki-canary.libsonnet';

loki_canary {
  loki_canary_args+:: {
    addr: "loki:3100",
    port: 80,
    labelname: "instance",
    interval: "100ms",
    size: 1024,
    wait: "3m",
  },
  _config+:: {
    namespace: "default",
  }
}
```

#### Examples

Standalone Pod Implementation of loki-canary

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: loki-canary
    name: loki-canary
  name: loki-canary
spec:
  containers:
  - args:
    - -addr=loki:3100
    image: grafana/loki-canary:latest
    imagePullPolicy: IfNotPresent
    name: loki-canary
    resources: {}
---
apiVersion: v1
kind: Service
metadata:
  name: loki-canary
  labels:
    app: loki-canary
spec:
  type: ClusterIP
  selector:
    app: loki-canary
  ports:
  - name: metrics
    protocol: TCP
    port: 3500
    targetPort: 3500
```

DaemonSet Implementation of loki-canary

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  labels:
    app: loki-canary
    name: loki-canary
  name: loki-canary
spec:
  selector:
    matchLabels:
      app: loki-canary
  template:
    metadata:
      name: loki-canary
      labels:
        app: loki-canary
    spec:
      containers:
      - args:
        - -addr=loki:3100
        image: grafana/loki-canary:latest
        imagePullPolicy: IfNotPresent
        name: loki-canary
        resources: {}
---
apiVersion: v1
kind: Service
metadata:
  name: loki-canary
  labels:
    app: loki-canary
spec:
  type: ClusterIP
  selector:
    app: loki-canary
  ports:
  - name: metrics
    protocol: TCP
    port: 3500
    targetPort: 3500
```

### From Source

If the other options are not sufficient for your use case, you can compile `loki-canary` yourself:

1. Clone the source tree.
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   git clone https://github.com/grafana/loki
   ```
2. Build the binary.
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   make loki-canary
   ```
3. Optional: Build the container image.
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   make loki-canary-image
   ```

### Example invocations

The following examples show common ways to run `loki-canary`. They use only flags that the binary accepts; see [Configuration](#configuration) for the full list.

#### Minimal local run (no authentication, no TLS)

Write artificial logs to standard output, tail them back over a WebSocket, and expose metrics on the default port `3500`:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
loki-canary \
  -addr=localhost:3100 \
  -labelname=instance \
  -labelvalue=loki-canary-1
```

By default the canary only reads logs over the WebSocket; an agent such as Grafana Alloy is still responsible for shipping the canary’s standard output into Loki. See [Monolithic mode setup](#monolithic-mode-setup) for a complete Systemd plus Alloy example.

#### Push mode against an authenticated, multi-tenant Loki

Have the canary push its own logs directly to Loki instead of relying on a separate agent, using basic authentication and an `X-Scope-OrgID` tenant header:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
loki-canary \
  -addr=loki.example.com:3100 \
  -push=true \
  -user=canary \
  -pass="$LOKI_PASSWORD" \
  -tenant-id=team-a \
  -labelname=instance \
  -labelvalue=loki-canary-team-a \
  -interval=500ms \
  -size=512
```

The `-streamvalue` flag always defaults to `stdout`, even when `-push=true` is set. If you rely on the stream label to distinguish push-mode canaries from stdout-mode canaries, set `-streamvalue` explicitly.

#### Run over TLS with a custom client certificate

Connect to a Loki endpoint that terminates TLS. Enabling `-tls` switches the WebSocket connection from `ws://` to `wss://`:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
loki-canary \
  -addr=loki.example.com:443 \
  -tls=true \
  -cert-file=/etc/loki-canary/client.crt \
  -key-file=/etc/loki-canary/client.key \
  -ca-file=/etc/loki-canary/ca.crt \
  -labelname=instance \
  -labelvalue=loki-canary-tls
```

If you supply any of `-cert-file`, `-key-file`, or `-ca-file` without also setting `-tls=true`, the canary exits with `Must set --tls when specifying client certs`.

## Configuration

The address of Loki must be passed in with the `-addr` flag or by setting the environment variable `LOKI_ADDRESS`, and if your Loki server uses TLS, `-tls=true` must also be provided. Note that using TLS will cause the WebSocket connection to use `wss://` instead of `ws://`.

The `-labelname` and `-labelvalue` flags should also be provided, as these are used by Loki Canary to filter the log stream to only process logs for the current instance of the canary. Ensure that the values provided to the flags are unique to each instance of Loki Canary. Grafana Labs’ Tanka config accomplishes this by passing in the Pod name as the label value.

**Be aware** that `-labels` only overwrites the selector used for the direct queries that confirm missing entries and spot-check results. The WebSocket tail connection, and the queries used for the metric test and cache test, always build their selector from `-labelname`/`-labelvalue` and `-streamname`/`-streamvalue`, and ignore `-labels`. If you use `-labels`, make sure `-labelname`/`-labelvalue` (and `-streamname`/`-streamvalue`) still identify the same stream, or these queries will target different data.

If Loki Canary reports a high number of `unexpected_entries`, Loki Canary may not be waiting long enough and the value for the `-wait` flag should be increased to a larger value than 60s.

**Be aware** of the relationship between `pruneinterval` and the `interval`. For example, with an interval of 10ms (100 logs per second) and a prune interval of 60s, you will write 6000 logs per minute. If those logs were not received over the WebSocket, the canary will attempt to query Loki directly to see if they are completely lost. **However** the query return is limited to 1000 results so you will not be able to return all the logs even if they did make it to Loki.

**Likewise**, if you lower the `pruneinterval` you risk causing a denial of service attack as all your canaries attempt to query for missing logs at whatever your `pruneinterval` is defined at.

All options:

![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```none
  -addr string
    	The Loki server URL:Port, e.g. loki:3100. Loki address can also be set using the environment variable LOKI_ADDRESS.
  -buckets int
    	Number of buckets in the response_latency histogram (default 10)
  -cache-test-interval duration
    	The interval the cache test query should be run (default 15m0s)
  -cache-test-now duration
    	Duration how far back from current time the execution time (--now) should be set for running this query in the cache test instant-query. (default 1h0m0s)
  -cache-test-range duration
    	The range value [24h] used in the cache test instant-query. (default 24h0m0s)
  -ca-file string
    	Client certificate authority for optional use with TLS connection to Loki
  -cert-file string
    	Client PEM encoded X.509 certificate for optional use with TLS connection to Loki
  -insecure
    	Allow insecure TLS connections
  -interval duration
    	Duration between log entries (default 1s)
  -key-file string
    	Client PEM encoded X.509 key for optional use with TLS connection to Loki
  -labels string
        Comma-separated string of labels for the query e.g. 'service=loki,app=canary'. The parsing logic for this argument is simple, label values must not contain a comma or special characters and should not be quoted. Overwrites labelname and streamname
  -labelname string
    	The label name for this instance of loki-canary to use in the log selector (default "name")
  -labelvalue string
    	The unique label value for this instance of loki-canary to use in the log selector (default "loki-canary")
  -max-wait duration
    	Duration to keep querying Loki for missing websocket entries before reporting them missing (default 5m0s)
  -metric-test-interval duration
    	The interval the metric test query should be run (default 1h0m0s)
  -metric-test-range duration
    	The range value [24h] used in the metric test instant-query. Note: this value is truncated to the running time of the canary until this value is reached (default 24h0m0s)
  -out-of-order-max duration
    	Maximum amount of time to go back for out of order entries (in seconds). (default 1m0s)
  -out-of-order-min duration
    	Minimum amount of time to go back for out of order entries (in seconds). (default 30s)
  -out-of-order-percentage int
    	Percentage (0-100) of log entries that should be sent out of order.
  -pass string
    	Loki password. This credential should have both read and write permissions to Loki endpoints
  -port int
    	Port which loki-canary should expose metrics (default 3500)
  -pruneinterval duration
    	Frequency to check sent vs received logs, also the frequency which queries for missing logs will be dispatched to loki (default 1m0s)
  -push
    	Push the logs directly to given Loki address
  -query-append string
        LogQL filters to be appended to the Canary query e.g. '| json | line_format `{{.log}}`'  	
  -query-timeout duration
    	How long to wait for a query response from Loki (default 10s)
  -size int
    	Size in bytes of each log line (default 100)
  -spot-check-initial-wait duration
    	How long should the spot check query wait before starting to check for entries (default 10s)
  -spot-check-interval duration
    	Interval that a single result will be kept from sent entries and spot-checked against Loki, e.g. 15min default one entry every 15 min will be saved and then queried again every 15min until spot-check-max is reached (default 15m0s)
  -spot-check-max duration
    	How far back to check a spot check entry before dropping it (default 4h0m0s)
  -spot-check-query-rate duration
    	Interval that the canary will query Loki for the current list of all spot check entries (default 1m0s)
  -streamname string
    	The stream name for this instance of loki-canary to use in the log selector (default "stream")
  -streamvalue string
    	The unique stream value for this instance of loki-canary to use in the log selector (default "stdout")
  -tenant-id string
    	Tenant ID to be set in X-Scope-OrgID header.
  -tls
    	Does the loki connection use TLS?
  -user string
    	Loki username.
  -version
    	Print this builds version information
  -wait duration
    	Duration to wait for log entries on websocket before querying loki for them (default 1m0s)
  -write-max-backoff duration
    	Maximum backoff time between retries  (default 5m0s)
  -write-max-retries int
    	Maximum number of retries when push a log entry  (default 10)
  -write-min-backoff duration
    	Initial backoff time before first retry  (default 500ms)
  -write-timeout duration
    	How long to wait write response from Loki (default 10s)
```

In addition to the options listed above, two flags control how logs are batched when pushing directly to Loki with `-push`:

- `-logs-batch-size` sends logs to Loki in batches of the given size. A value of `0` or `1` disables batching and sends each entry immediately (default `1`).
- `-logs-batch-size-max` is the upper bound on `-logs-batch-size` (default `20`). Only increase it if you have also increased the memory limits for the canary.

## Troubleshooting common misconfigurations

This section lists the most common configuration mistakes and how to resolve them.

### Canary exits immediately with “Must specify a Loki address”

If you see:

nohighlight ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```nohighlight
Must specify a Loki address with -addr or set the environment variable LOKI_ADDRESS
```

You did not provide a Loki address. Set `-addr` (for example `-addr=loki:3100`) or set `LOKI_ADDRESS`. The value is `host:port` only: do not include a scheme such as `http://` or `https://`, and use `-tls=true` to select a secure connection.

### No entries are read, all logs end up missing

If the canary writes logs but the `loki_canary_missing_entries_total` counter keeps climbing and `response_latency` records nothing, the most common cause is a label mismatch between what the canary writes and what it queries:

- The canary filters the log stream using `-labelname`/`-labelvalue` (and `-streamname`/`-streamvalue`). The agent that ships the canary’s standard output into Loki must apply the same labels. If your Alloy or Promtail configuration attaches different labels, the canary’s read query matches nothing.
- Each canary instance must use a unique `-labelvalue`. If two canaries share the same label value, they read each other’s logs and report out-of-order or unexpected entries.
- If you set `-labels`, remember it only overwrites the selector for the direct queries, not for the WebSocket tail. Make sure the selector built from `-labels` and the one built from `-labelname`/`-streamname` both match the labels applied at ingestion time.

To confirm the selector, run the same query the canary builds (for example `{name="loki-canary", stream="stdout"}`) directly in Grafana or `logcli` and check that it returns the canary’s lines.

### Authentication failures (401 or 403)

If pushes or queries fail with HTTP 401 or 403, the basic-auth credentials or tenant header are wrong:

- Provide both `-user` and `-pass`. The password credential must have read and write permissions, because the canary both queries and (with `-push`) writes.
- For multi-tenant Loki, set `-tenant-id` so the canary sends the correct `X-Scope-OrgID` header. A missing or wrong tenant ID typically returns empty query results, or an error such as:
  
  nohighlight ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```nohighlight
  no org id
  ```

### TLS errors

- Enabling client certificates without TLS fails fast with `Must set --tls when specifying client certs`. Always pair `-cert-file`, `-key-file`, and `-ca-file` with `-tls=true`.
- `x509: certificate signed by unknown authority` means the canary does not trust the server certificate. Provide the CA with `-ca-file`, or, for testing only, set `-insecure` to skip verification. Do not use `-insecure` in production.
- After enabling `-tls`, make sure `-addr` points at the TLS port (for example `:443` or your gateway’s HTTPS port), because the WebSocket connection switches to `wss://`.

## Monolithic mode setup

This section describes how to set up Loki Canary for Loki’s [monolithic mode](/docs/loki/latest/get-started/deployment-modes/#monolithic-mode) using Systemd, Alloy, and Prometheus.

### Systemd

Create a systemd service file that writes Loki Canary’s standard output to the file `/var/log/loki-canary.log`.

ini ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```ini
[Unit]
Description=Loki Canary
Documentation=https://grafana.com/docs/loki/latest/operations/loki-canary/

[Service]
User=loki
ExecStart=/usr/bin/loki-canary -addr=localhost:3100 -labelname=job -labelvalue=loki_canary -streamname=job -streamvalue=loki_canary
Restart=on-failure
RestartSec=5
StandardOutput=append:/var/log/loki-canary.log
StandardError=journal

[Install]
WantedBy=multi-user.target
```

`-labelname` and `-labelvalue` flags specify a label pair used to identify Loki Canary’s logs. `-streamname` and `-streamvalue` flags specify an additional label pair; they default to `stream` and `stdout`, but can be set explicitly if you need a different value. The same values can be provided to both label pairs if no additional label exists. Labels can be added when Alloy scrapes the logs.

### Scrape logs

Scrape the `/var/log/loki-canary.log` file with Alloy.

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
loki.source.file "canary" {
  forward_to = [loki.write.local.receiver]
  targets = [{
    __path__ = "/var/log/loki-canary.log",
    job      = "loki_canary",
  }]
}

loki.write "local" {
  endpoint {
    url  = "http://localhost:3100/loki/api/v1/push"
  }
}
```

### Scrape metrics

Scrape Loki Canary’s metrics with Alloy or Prometheus.

#### Scrape metrics with Alloy

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
prometheus.scrape "loki" {
  targets    = [{__address__ = "localhost:3100"}]
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {  
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"
  }  
}
```

#### Scrape metrics with Prometheus

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
scrape_configs:
  - job_name: loki-canary
    static_configs:
      - targets: ['localhost:3500']
```