Capture Kubernetes logs with OpenTelemetry Collector

Mon, 16 Jun 2025 10:29:52 +0000

Capture Kubernetes logs with OpenTelemetry Collector

Grafana Labs recommends sending OpenTelemetry logs with the OTLP protocol. However, some use cases require outputting logs to files or stdout:

The OpenTelemetry SDK for Go, Python, Ruby, JavaScript, and PHP does not provide a stable OTLP implementation for logs.
Organizational constraints, often related to reliability practices, require using files for logs.

You can still collect file-based logs with the OpenTelemetry Collector. This guide shows how to capture logs emitted through Kubernetes stdout, and you can apply the same pattern to logs written to files.

Architecture

To correlate traces and metrics with logs, enrich logs with the same resource attributes and trace and span IDs.

First, add identifying resource attributes to logs, such as service.name, service.namespace, service.instance.id, and deployment.environment, along with trace_id and span_id.

Then, use the same metadata enrichment pipeline in the OpenTelemetry Collector, such as the Kubernetes Attributes Processor or the Resource Detection Processor.

If you already get this enrichment when exporting logs through OTLP, add these attributes to log lines collected from files or stdout.

To carry over resource attributes in log lines, use one of the following export patterns.

Export unstructured logs:

Export unstructured logs and parse them with regular expressions, for example:

2024-09-17T11:29:54  INFO [nio-8080-exec-1] c.e.OrderController  : Order completed - service.name=order-processor, service.instance.id=i-123456, span_id=1d5f8ca3f9366fac...

Export structured logs:

Export structured format logs like JSON logs and parse them with native parsers of the chosen format, for example:

{"timestamp": "2024-09-17T11:29:54", "level": "INFO", "body":"Order completed", "logger": "c.e.OrderController", "service_name": "order-processor", "service_instance_id": "i-123456", "span_id":"1d5f8ca3f9366fac"...}

Both export patterns have advantages and disadvantages:

	JSON logs	Unstructured logs
Correlation	+++	+++
Human Readability	The verbosity of JSON can seriously erode readability	Contextualization attributes can be appended at the end of the log line preserving readability
Reliability of the parsing	It’s simple to define robust JSON parsing rules	Parsing unstructured text with regular expressions is fragile, particularly due to multi-line log messages like stack traces, to the point where it requires monitoring parsing failures

Emit contextualized JSON logs with Java

Mon, 16 Jun 2025 10:29:52 +0000

Emit contextualized JSON logs with Java

Most popular logging frameworks, such as Log4j or SLF4J/Logback in Java, support emitting JSON formatted logs.

To integrate OpenTelemetry with logging libraries, specify the resource attributes you want available in the log line.

To keep your architecture simple, add only the attributes you need to filter and correlate logs, such as service.name, service.namespace, deployment.environment, and service.instance.id.

Example application

The following example uses Spring Boot to enrich logs with OpenTelemetry attributes and emit them formatted in JSON through stdout.

Note
You can find the full code of the example in the Docker LGTM repository.

Create a Spring Boot application 3.3.0+ using the default logging instrumentation with the Logback library and the stdout output.

Instrument the application

Instrument the Spring Boot application with the OpenTelemetry Java Agent by following the setup flow defined by the Grafana Cloud integration for Java:

Open the Grafana Cloud home page in a web browser.
Navigate to Connections > Add new Connection.
Select Java OpenTelemetry and follow the setup instructions.

Enrich Logback logs with OpenTelemetry resource attributes and log attributes using:

export OTEL_INSTRUMENTATION_COMMON_MDC_RESOURCE_ATTRIBUTES=service.namespace,service.name,service.instance.id,service.version,deployment.environment

Change the Logback SpringBoot configuration to emit JSON logs with OpenTelemetry contextualization and add the following logback-spring.xml under src/main/resources:

<!-- tested with Logback 1.5.0 and Spring Boot 3.3.0 -->
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>

    <root level="INFO">
        <appender-ref ref="CONSOLE_JSON"/>
    </root>

    <appender name="CONSOLE_JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.JsonEncoder">
            <withFormattedMessage>true</withFormattedMessage>
            <withMessage>false</withMessage>
            <withArguments>false</withArguments>
            <withSequenceNumber>false</withSequenceNumber>
            <withNanoseconds>false</withNanoseconds>
        </encoder>
    </appender>
</configuration>

Deploy the application

Deploy the Spring Boot application on Kubernetes and verify that the application outputs logs to the container stdout stream with JSON formatting and OpenTelemetry contextualization, for example kubectl logs my_pod_name | jq outputs:

{
  "timestamp": 1727346005788,
  "level": "INFO",
  "threadName": "http-nio-8080-exec-5",
  "loggerName": "com.grafana.example.RollController",
  "context": {
    "name": "default",
    "birthdate": 1727345887787,
    "properties": {
    }
  },
  "mdc": {
    "trace_id": "97a39974ba2dfc9275e4d31dc2730ee4",
    "trace_flags": "01",
    "service.name": "dice",
    "service.instance.id": "0fb18318-06c0-4893-9a8b-353c00b45227",
    "service.version": "1.1",
    "span_id": "08d6d83d645e3ad2",
    "service.namespace": "shop",
    "deployment.environment": "staging"
  },
  "formattedMessage": "Anonymous player is rolling the dice: 6",
  "throwable": null
}

Configure OpenTelemetry Collector

Configure the OpenTelemetry Collector instance with the file log receiver:

Add a file log receiver.
Add the container parser operator.
Add the json parser operator.
- Map body to attributes.formattedMessage
- Map timestamp.parse_from to attributes.timestamp
- Map severity.parse_from to attributes.level
- Map trace.trace_id.parse_from to attributes.mdc.trace_id
- Map trace.span_id.parse_from to attributes.mdc.span_id
- Map trace.trace_flags.parse_from to attributes.mdc.trace_flags
Add the move operator.
- Move top level fields, such as threadName, to their corresponding OTel attributes, such as thread.name
- Move entries from the mdc field to the resource entry of the log entry
Remove all unnecessary fields using the attributes operator.

You can find the full configuration in the reference OTel collector configuration.

Set up OpenTelemetry Collector for Application Observability on Grafana Labs

Capture Kubernetes logs with OpenTelemetry Collector

Capture Kubernetes logs with OpenTelemetry Collector

Architecture

Emit contextualized JSON logs with Java

Emit contextualized JSON logs with Java

Example application

Instrument the application

Deploy the application

Configure OpenTelemetry Collector