Jenkins Distributed Builds & Performance

Distributed Build Architecture

┌─────────────────────────────────────────────┐
│              Jenkins Master                   │
│  (Orchestration, UI, scheduling only)        │
└──────────┬──────────┬──────────┬────────────┘
           │          │          │
    ┌──────▼───┐ ┌───▼─────┐ ┌─▼──────────┐
    │ Agent 1  │ │ Agent 2 │ │ Agent 3    │
    │ (Linux)  │ │(Docker) │ │(Kubernetes)│
    │ 4 exec.  │ │ 8 exec. │ │ Dynamic    │
    └──────────┘ └─────────┘ └────────────┘

Agent Types

Type	Provisioning	Best For
Permanent	Always running	Steady workloads
SSH	On-demand via SSH	Linux/macOS agents
JNLP	Agent connects to master	Behind firewalls
Docker	Container per build	Isolated builds
Kubernetes	Pod per build	Auto-scaling

Docker-Based Agents

pipeline {
    agent {
        docker {
            image 'node:20-alpine'
            args '-v /tmp/.npm:/root/.npm'  // Cache npm
            label 'docker-host'
        }
    }
    stages {
        stage('Build') {
            steps {
                sh 'npm ci'
                sh 'npm run build'
            }
        }
    }
}

Multiple Docker Agents

pipeline {
    agent none
    stages {
        stage('Build Frontend') {
            agent { docker { image 'node:20' } }
            steps { sh 'cd frontend && npm ci && npm run build' }
        }
        stage('Build Backend') {
            agent { docker { image 'golang:1.21' } }
            steps { sh 'cd backend && go build ./...' }
        }
        stage('Integration Tests') {
            agent {
                docker {
                    image 'docker/compose:latest'
                    args '-v /var/run/docker.sock:/var/run/docker.sock'
                }
            }
            steps { sh 'docker-compose -f docker-compose.test.yml up --abort-on-container-exit' }
        }
    }
}

Kubernetes Agents

pipeline {
    agent {
        kubernetes {
            yaml '''
            apiVersion: v1
            kind: Pod
            spec:
              containers:
              - name: node
                image: node:20
                command: ['sleep', 'infinity']
                resources:
                  requests:
                    memory: "512Mi"
                    cpu: "500m"
              - name: docker
                image: docker:dind
                securityContext:
                  privileged: true
                volumeMounts:
                - name: docker-storage
                  mountPath: /var/lib/docker
              volumes:
              - name: docker-storage
                emptyDir: {}
            '''
        }
    }
    stages {
        stage('Build') {
            steps {
                container('node') {
                    sh 'npm ci && npm run build'
                }
            }
        }
        stage('Docker Build') {
            steps {
                container('docker') {
                    sh 'docker build -t myapp:latest .'
                }
            }
        }
    }
}

Performance Optimization

Master Configuration

// jenkins.yaml (Configuration as Code)
jenkins:
  numExecutors: 0          // No builds on master!
  mode: EXCLUSIVE
  quietPeriod: 5
  scmCheckoutRetryCount: 3
 
  nodes:
    - permanent:
        name: "build-agent-1"
        numExecutors: 4
        remoteFS: "/var/jenkins"
        labelString: "linux docker"

Build Caching

pipeline {
    agent any
    options {
        // Discard old builds
        buildDiscarder(logRotator(numToKeepStr: '20'))
        // Timeout
        timeout(time: 30, unit: 'MINUTES')
        // Don't run concurrent builds
        disableConcurrentBuilds()
    }
    stages {
        stage('Build') {
            steps {
                // Cache node_modules
                cache(maxCacheSize: 500, caches: [
                    arbitraryFileCache(path: 'node_modules', cacheValidityDecidingFile: 'package-lock.json')
                ]) {
                    sh 'npm ci'
                }
                sh 'npm run build'
            }
        }
    }
}

Resource Management

Setting	Recommendation
Master executors	0 (never build on master)
Agent executors	2× CPU cores
Heap size	4-8 GB for master
Build history	Keep last 20-30 builds
Workspace cleanup	After every build
Plugin count	Minimize — each adds overhead

High Availability

Active/Passive Setup

┌──────────────┐     ┌──────────────┐
│ Jenkins      │     │ Jenkins      │
│ Primary      │────▶│ Standby      │
│ (Active)     │     │ (Passive)    │
└──────┬───────┘     └──────────────┘
       │
┌──────▼───────┐
│ Shared NFS   │
│ (JENKINS_HOME)│
└──────────────┘

Backup Strategy

# Backup JENKINS_HOME
tar czf jenkins-backup-$(date +%Y%m%d).tar.gz \
  --exclude='workspace' \
  --exclude='builds/*/archive' \
  /var/lib/jenkins/
 
# Critical files to backup:
# - config.xml (global config)
# - jobs/*/config.xml (job configs)
# - users/ (user data)
# - secrets/ (encrypted credentials)
# - plugins/ (installed plugins)

Summary

You've learned:

Distributed build architecture with multiple agent types
Docker and Kubernetes dynamic agents
Performance optimization (caching, resource management)
High availability and backup strategies
Scaling Jenkins for enterprise workloads

Next Steps

Next, we'll cover Jenkins pipeline testing and advanced Groovy patterns.

Learning Objectives

Distributed Build Architecture

Agent Types

Docker-Based Agents

Multiple Docker Agents

Kubernetes Agents

Performance Optimization

Master Configuration

Build Caching

Resource Management

High Availability

Active/Passive Setup

Backup Strategy

Summary

Next Steps