Loki, Traefik, and K3s

Create Fluent Bit Config for Traefik

Create a fluent-bit.conf

 1[SERVICE]
 2    flush        1
 3    daemon       Off
 4
 5    log_level    info
 6
 7    parsers_file parsers.conf
 8
 9    http_server  Off
10    http_listen  0.0.0.0
11    http_port    2020
12
13    storage.metrics on
14
15[INPUT]
16    name tail
17    parser traefik
18    path /var/log/traefik/access.log
19
20[OUTPUT]
21    Name              loki
22    Match             *
23    Host              loki.monitoring
24    Labels            job=traefik,hostname=$HOSTNAME
25
26[OUTPUT]
27    name  stdout
28    match *

and a parsers.conf

1[PARSER]
2    Name        traefik
3    Format      json
4    Time_Key    time
5    Time_Format %Y-%m-%dT%H:%M:%S %z

create the configmap within the same namespace as traefik

1kubectl create configmap --from-file=fluent-bit.conf=fluent-bit.conf --from-file=parsers.conf=parsers.conf fluentbit`

Update Traefik k3s Manifest

Update /var/lib/rancher/k3s/server/manifests/traefik.yaml to add in a sidecar, a shared volume mount for storing logs, and project down the fluent bit config into the sidecar.

 1logs:
 2	access:
 3		enabled: true
 4		format: json
 5		filePath: /var/log/traefik/access.log
 6additionalVolumeMounts:
 7- name: log
 8	mountPath: /var/log/traefik
 9deployment:
10	additionalContainers:
11	- name: fluentbit
12		image: fluent/fluent-bit:4.0
13		volumeMounts:
14		- name: logs
15			mountPath: /var/log/traefik
16		- name: fluentbit
17			mountPath: /fluent-bit/etc/
18	additionalVolumes:
19	- name: logs
20	- name: fluentbit
21		configMap:
22			name: fluentbit

Deploy Loki

Deploy Loki in single binary mode with multi tenancy disabled. I also exposed the Loki gateway with tailscale

 1gateway:
 2  ingress:
 3    enabled: true
 4    ingressClassName: "tailscale"
 5    tls:
 6    - hosts:
 7      - loki
 8    hosts:
 9      - host: 'loki'
10        paths:
11          - path: /
12            pathType: Prefix
13
14
15loki:
16  auth_enabled: false # disable multi tenancy
17  commonConfig:
18    replication_factor: 1
19  schemaConfig:
20    configs:
21      - from: "2024-04-01"
22        store: tsdb
23        object_store: s3
24        schema: v13
25        index:
26          prefix: loki_index_
27          period: 24h
28  pattern_ingester:
29      enabled: true
30  limits_config:
31    allow_structured_metadata: true
32    volume_enabled: true
33  ruler:
34    enable_api: true
35
36minio:
37  enabled: true
38      
39deploymentMode: SingleBinary
40
41singleBinary:
42  replicas: 1
43
44# Zero out replica counts of other deployment modes
45backend:
46  replicas: 0
47read:
48  replicas: 0
49write:
50  replicas: 0
51
52ingester:
53  replicas: 0
54querier:
55  replicas: 0
56queryFrontend:
57  replicas: 0
58queryScheduler:
59  replicas: 0
60distributor:
61  replicas: 0
62compactor:
63  replicas: 0
64indexGateway:
65  replicas: 0
66bloomCompactor:
67  replicas: 0
68bloomGateway:
69  replicas: 0

Connect Grafana to Loki

Grafana > Connections > Add new connection > Loki > Add new data source > Set the URL to http://loki-gateway.monitoring.svc.cluster.local > Save & test