Kubernetes Monitoring Stack: Prometheus + Grafana + Loki

TL;DR
#

Prometheus for metrics collection and alerting rules
Grafana for dashboards and visualization
Loki for log aggregation (lightweight alternative to Elasticsearch)
Grafana Alloy as the log collector — Promtail reached end of life in March 2026
AlertManager for routing alerts to Slack/PagerDuty
Runs on k3s or any Kubernetes cluster, ~1.3 GB of memory requests once you count everything the charts install
Everything installed via Helm, configured via values files you can version control — with secrets kept out of them

Who this stack is for
#

You run a k3s or small Kubernetes cluster and want real monitoring without paying for Datadog. You’re comfortable with Helm and kubectl. You want logs and metrics in one place, with alerts that actually wake someone up when things break.

The stack
#

Layer	Tool	Why
Metrics	Prometheus	Industry standard, huge ecosystem
Dashboards	Grafana	Flexible, free, large dashboard library
Logs	Loki	Log aggregation without the Elasticsearch overhead
Log collector	Grafana Alloy	Ships logs to Loki, runs as DaemonSet. Replaces Promtail
Alerts	AlertManager	Routes alerts by severity to Slack, PagerDuty, email

Promtail is dead — don’t start a new deployment on it. Grafana moved Promtail to LTS in February 2025 and it reached end of life on 2 March 2026: no features, no bug fixes, no security patches. Grafana Alloy is the replacement, and it can convert an existing Promtail config for you: alloy convert --source-format=promtail --output=config.alloy promtail-config.yaml.

Namespace setup
#

kubectl create namespace monitoring

Secrets first
#

Two things in this stack are secrets, and neither belongs in a values file you commit: the Grafana admin password and the Slack webhook URL. Create them once:

kubectl create secret generic grafana-admin \
  --namespace monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password="$(openssl rand -base64 24)"

kubectl create secret generic alertmanager-slack \
  --namespace monitoring \
  --from-literal=webhook-url='https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

Read the generated Grafana password back when you need it:

kubectl get secret grafana-admin -n monitoring \
  -o jsonpath='{.data.admin-password}' | base64 -d

A Slack incoming webhook URL is a credential — anyone holding it can post into your channel. Treat it like a password, not like configuration.

Prometheus + AlertManager
#

Install via the kube-prometheus-stack Helm chart, which bundles Prometheus, AlertManager, Grafana, and common dashboards.

Save as prometheus-values.yaml:

prometheus:
  prometheusSpec:
    retention: 15d
    resources:
      requests:
        memory: 512Mi
        cpu: 250m
      limits:
        memory: 1Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

    # Scrape all ServiceMonitors across namespaces
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

alertmanager:
  alertmanagerSpec:
    # Mounted at /etc/alertmanager/secrets/<secret-name>/
    secrets:
      - alertmanager-slack
    resources:
      requests:
        memory: 64Mi
        cpu: 50m
      limits:
        memory: 128Mi
  config:
    global:
      resolve_timeout: 5m
      # Read the webhook from the mounted secret instead of inlining it
      slack_api_url_file: /etc/alertmanager/secrets/alertmanager-slack/webhook-url
    route:
      receiver: "slack-notifications"
      group_by: ["alertname", "namespace"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - receiver: "slack-critical"
          matchers:
            - severity = "critical"
          repeat_interval: 1h
        - receiver: "slack-notifications"
          matchers:
            - severity = "warning"
    receivers:
      - name: "slack-notifications"
        slack_configs:
          - channel: "#alerts"
            title: "{{ .GroupLabels.alertname }}"
            text: >-
              {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }}
              *Severity:* {{ .Labels.severity }}
              *Namespace:* {{ .Labels.namespace }}
              {{ end }}
      - name: "slack-critical"
        slack_configs:
          - channel: "#alerts-critical"
            title: "CRITICAL: {{ .GroupLabels.alertname }}"
            text: >-
              {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }}
              *Description:* {{ .Annotations.description }}
              *Namespace:* {{ .Labels.namespace }}
              {{ end }}

grafana:
  admin:
    existingSecret: grafana-admin
    userKey: admin-user
    passwordKey: admin-password
  persistence:
    enabled: true
    size: 5Gi
  resources:
    requests:
      memory: 128Mi
      cpu: 100m
    limits:
      memory: 256Mi
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: "default"
          orgId: 1
          folder: ""
          type: file
          disableDeletion: false
          editable: true
          options:
            path: /var/lib/grafana/dashboards/default
  # This provider only does something once you also set `dashboards:` with
  # actual dashboard definitions — the chart creates and mounts that
  # directory from `.Values.dashboards`, so on its own the block above
  # points at an empty path and Grafana logs a provisioning error at start.
  # The kube-prometheus-stack dashboards arrive separately via the sidecar
  # and do not need this. Either add `dashboards:` or drop both keys.

# Default recording and alerting rules
defaultRules:
  create: true
  rules:
    etcd: false  # Disable if not running etcd (e.g., k3s with SQLite)
    kubeScheduler: false  # Not accessible on most managed clusters

Note the matchers: syntax in the alert routes. The older match: map still works, but it has been superseded since Alertmanager v0.22 and you’ll want the newer form when you start writing anything with regex or negation.

Install it:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-values.yaml

Loki + Alloy
#

The Loki chart has three deployment modes and picks the wrong one for a small cluster by default. Two settings below are the difference between a running Loki and a stuck one — both are called out in comments.

Save as loki-values.yaml:

# Without this the chart defaults to SimpleScalable, refuses to reconcile
# replicas across modes, and never renders the single-binary StatefulSet.
# Valid values are SingleBinary, SimpleScalable and Distributed.
# "Monolithic" appears in the Loki documentation as a deployment topology
# but is not a value this chart accepts — check values.yaml for your version.
deploymentMode: SingleBinary

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h
  limits_config:
    retention_period: 168h  # 7 days
    max_query_series: 500
  compactor:
    working_directory: /var/loki/compactor
    # Loki 3.x refuses to start with retention enabled and no delete store:
    # "invalid compactor config: compactor.delete-request-store should be
    #  configured when retention is enabled"
    delete_request_store: filesystem
    retention_enabled: true

singleBinary:
  replicas: 1
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 512Mi
  persistence:
    enabled: true
    size: 10Gi

# The chart enables two memcached tiers by default and sizes them for a
# real cluster: chunksCache asks for ~9.6Gi and resultsCache ~1.2Gi
# (requests are computed as 1.2x allocatedMemory). Leave them on and every
# pod sits Pending on a small node.
chunksCache:
  enabled: false
resultsCache:
  enabled: false

gateway:
  enabled: false

# Minimal deployment — no read/write separation
read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0

Save as alloy-values.yaml:

alloy:
  configMap:
    content: |-
      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pods" {
        targets = discovery.kubernetes.pods.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pods.output
        forward_to = [loki.process.default.receiver]
      }

      loki.process "default" {
        stage.cri {}
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push"
        }
      }

  resources:
    requests:
      memory: 128Mi
      cpu: 50m
    limits:
      memory: 256Mi

controller:
  type: daemonset

Install both. Pin the chart versions — the Loki chart has changed its defaults more than once, and an unpinned helm install six months from now will not produce the deployment you tested:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install loki grafana/loki \
  --namespace monitoring \
  --version 6.24.0 \
  --values loki-values.yaml

helm install alloy grafana/alloy \
  --namespace monitoring \
  --version 0.10.1 \
  --values alloy-values.yaml

Check the current chart versions with helm search repo grafana/loki --versions and pin whatever you actually tested against.

Connect Loki to Grafana
#

Add Loki as a data source in Grafana. You can do this via the UI or by adding to your prometheus-values.yaml under the Grafana section:

grafana:
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki.monitoring.svc.cluster.local:3100
      access: proxy
      isDefault: false

Then upgrade the Helm release:

helm upgrade kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-values.yaml

Custom alert rules
#

Add application-specific alerts. Save as custom-alerts.yaml and apply with kubectl apply -f:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
    - name: app.rules
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
            > 0.05
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate on {{ $labels.service }}"
            description: "More than 5% of requests are failing on {{ $labels.service }} for the last 5 minutes."

        - alert: HighLatency
          expr: |
            histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
            > 1.0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High p95 latency on {{ $labels.service }}"
            description: "p95 latency is above 1s on {{ $labels.service }}."

        - alert: PodRestartLoop
          expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} is restart-looping"
            description: "Pod {{ $labels.pod }} in {{ $labels.namespace }} has restarted more than 5 times in the last hour."

        - alert: PersistentVolumeSpaceLow
          expr: |
            kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "PV {{ $labels.persistentvolumeclaim }} is running low on space"
            description: "Less than 15% space remaining."

Access Grafana
#

Start with a port-forward. It needs no certificates, no DNS and no exposure:

kubectl port-forward svc/kube-prometheus-grafana 3000:80 -n monitoring

If you do want it on a hostname, k3s ships Traefik but without any ACME certificate resolver configured — referencing certResolver: letsencrypt on a stock k3s gets you a Traefik error in the logs and its default self-signed certificate. Configure the resolver first, via k3s’s HelmChartConfig:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--certificatesresolvers.letsencrypt.acme.email=you@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
    persistence:
      enabled: true
      path: /data

Then the IngressRoute works as written:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: monitoring
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.your-domain.com`)
      kind: Rule
      services:
        - name: kube-prometheus-grafana
          port: 80
  tls:
    certResolver: letsencrypt

Before you point DNS at this, confirm the admin password came from the secret above and not from a chart default, and put something in front of Grafana that authenticates — an identity-aware proxy or at minimum an IP allowlist. A Grafana on a public hostname is a browsable map of your infrastructure, and its login page will be found by scanners within hours.

Resource totals
#

Charts install more than the five components people remember. The kube-prometheus-stack also brings node-exporter (one per node), kube-state-metrics and the operator itself, and Alloy replaces Promtail at a slightly higher footprint:

Component	Memory request	Memory limit	CPU request
Prometheus	512Mi	1Gi	250m
AlertManager	64Mi	128Mi	50m
Grafana	128Mi	256Mi	100m
Loki (single binary, caches off)	256Mi	512Mi	100m
Alloy (per node)	128Mi	256Mi	50m
node-exporter (per node)	~32Mi	64Mi	10m
kube-state-metrics	~64Mi	128Mi	10m
prometheus-operator	~64Mi	128Mi	50m
Total (single node)	~1.3 Gi	~2.5 Gi	~620m

This fits a 4 GB node if that node is mostly doing monitoring. If it also runs your workloads, plan for 8 GB — Prometheus in particular grows with active series, and the 1 Gi limit above is a starting point, not a ceiling you should expect to hold at scale.

And to repeat the comment in the values file: if you leave the Loki chart’s memcached tiers enabled, requests jump by roughly 11 Gi and nothing schedules at all. That single default is the most common reason this stack appears to “not work” on a small cluster.

When to outgrow this stack
#

Logs exceed 50 GB/day: Consider Loki with S3 backend or switch to a managed solution
Multi-cluster: Add Thanos or Mimir on top of Prometheus for cross-cluster metrics
Compliance needs: Switch to a managed service that handles retention policies and audit trails
Team grows past 15 engineers: The dashboard sprawl becomes real, consider Datadog or Grafana Cloud

TL;DR #

Who this stack is for #

The stack #

Namespace setup #

Secrets first #

Prometheus + AlertManager #

Loki + Alloy #

Connect Loki to Grafana #

Custom alert rules #

Access Grafana #

Resource totals #

When to outgrow this stack #

Related reads #