Log Management with Loki & Fluentd

Why Centralized Logging?

In distributed systems, logs are scattered across dozens of containers and servers. Centralized logging brings them together.

Logging Stack Options

Stack	Components	Best For
PLG	Promtail + Loki + Grafana	Lightweight, K8s native
ELK	Elasticsearch + Logstash + Kibana	Full-text search, enterprise
EFK	Elasticsearch + Fluentd + Kibana	Kubernetes standard

Grafana Loki

Loki is a log aggregation system designed to be cost-effective and easy to operate. Unlike Elasticsearch, it only indexes labels (not full text).

Architecture

┌──────────┐    ┌──────────┐    ┌─────────┐
│ Promtail │───▶│   Loki   │───▶│ Grafana │
│(Collector)│   │ (Storage)│    │ (Query) │
└──────────┘    └──────────┘    └─────────┘

Docker Compose Deployment

version: "3.8"
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
 
  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/config.yml
 
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
 
volumes:
  loki_data:

Promtail Configuration

# promtail-config.yml
server:
  http_listen_port: 9080
 
positions:
  filename: /tmp/positions.yaml
 
clients:
  - url: http://loki:3100/loki/api/v1/push
 
scrape_configs:
  - job_name: system
    static_configs:
      - targets: ['localhost']
        labels:
          job: system
          __path__: /var/log/*.log
 
  - job_name: docker
    static_configs:
      - targets: ['localhost']
        labels:
          job: docker
          __path__: /var/lib/docker/containers/**/*.log
    pipeline_stages:
      - json:
          expressions:
            log: log
            stream: stream
      - output:
          source: log

LogQL — Querying Logs

# Basic label filter
{job="nginx"}
 
# Filter by content
{job="nginx"} |= "error"
{job="nginx"} != "healthcheck"
{job="nginx"} |~ "status=(4|5)\\d\\d"
 
# Parse and filter structured logs
{job="app"} | json | level="error"
{job="app"} | json | response_time > 1000
 
# Aggregations
count_over_time({job="nginx"} |= "error" [5m])
rate({job="nginx"} |= "POST" [1m])
sum by (status) (count_over_time({job="nginx"} | json [5m]))

Fluentd

Fluentd is a unified logging layer that collects, transforms, and routes logs.

Kubernetes DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-forward
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch.logging"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: containers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: containers
        hostPath:
          path: /var/lib/docker/containers

Structured Logging Best Practices

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "error",
  "service": "nextgen-api",
  "message": "Database connection failed",
  "error": "connection refused",
  "host": "pod-abc123",
  "trace_id": "abc-def-123",
  "duration_ms": 5000
}

Practice	Reason
Use JSON format	Easy to parse and query
Include timestamps	Correlation across services
Add trace IDs	Connect logs to traces
Use severity levels	Filter noise
Include context	Faster debugging

Summary

You've learned:

Centralized logging architecture and stack options
Deploying Loki with Promtail for log aggregation
Querying logs with LogQL
Fluentd for Kubernetes log collection
Structured logging best practices

Next Steps

Next, we'll explore distributed tracing for understanding request flows across microservices.

Learning Objectives