K8s monitoring v2: Why are there no logs?

Do you have suddenly no logs in your Grafana Alloy logs after upgrading the “K8s Monitoring helm chart v2”? 😅 This post will help you to solve the issue.

I have introduced Grafanas “K8s Monitoring helm chart v2” recently into my homelab, but I had a bad awakening when trying to debug an issue with my ingress and the logs were not showing up in Grafana. In the end, it has been a layer 8 issue, so behind the keyboard 😅, but I wanted to share the solution with you.

TLDR: You need to set the securityContext for the alloy-logs to allow privileged access to the containers. This is to allow the alloy container to assume the root user and collect logs host.

Symptoms

The following log entries are shown in the alloy logs:

ts=2025-03-01T23:55:08.729084207Z level=info msg="skipping update of position for a file which does not currently exist" component_path=/pod_logs.feature component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/kube-system_kube-router-bjndh_653bf29a-28c8-4dca-8357-bb0b8f65133f/install-cniconf/0.log

Basically, Alloy is not able to write the index files it uses to keep track of the already ingested log lines.

Solution

The solution is to set the securityContext for the alloy-logs to allow privileged access to the containers. This is to allow the alloy container to assume the root user and collect logs from the /var/log/* directory on the host, which is documented in the Grafana Alloy Helm chart values file

alloy-logs:
  enabled: true
  alloy:
    # we need privileged access to collect logs from other containers
    securityContext:
      allowPrivilegeEscalation: true
      privileged: true
      runAsUser: 0

podLogs:
  enabled: true
nodeLogs:
  enabled: true