Server Operating System Analysis

• 1: Ubuntu Analysis
• 2: Fedora Analysis
• 3: Talos Linux Analysis
• 4: Harvester Analysis

1 - Ubuntu Analysis

Overview

Ubuntu Server is a popular general-purpose Linux distribution developed by Canonical. It provides Long Term Support (LTS) releases with 5 years of standard support and optional Extended Security Maintenance (ESM).

Key Facts:

• Latest LTS: Ubuntu 24.04 LTS (Noble Numbat)
• Support Period: 5 years standard, 10 years with Ubuntu Pro (free for personal use)
• Kernel: Linux 6.8+ (LTS), regular HWE updates
• Package Manager: APT/DPKG, Snap
• Init System: systemd

Kubernetes Installation Methods

Ubuntu supports multiple Kubernetes installation approaches:

1. kubeadm (Official Kubernetes Tool)

Installation:

# Install container runtime (containerd)
sudo apt-get update
sudo apt-get install -y containerd

# Configure containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd

# Install kubeadm, kubelet, kubectl
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Cluster Initialization:

# Initialize control plane
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# Configure kubectl for admin
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Install CNI (e.g., Calico, Flannel)
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

# Join worker nodes
kubeadm token create --print-join-command

Pros:

• Official Kubernetes tooling, well-documented
• Full control over cluster configuration
• Supports latest Kubernetes versions
• Large community and extensive resources

Cons:

• More manual steps than turnkey solutions
• Requires understanding of Kubernetes architecture
• Manual upgrade process for each component
• More complex troubleshooting

2. k3s (Lightweight Kubernetes)

Installation:

# Single-command install on control plane
curl -sfL https://get.k3s.io | sh -

# Get node token for workers
sudo cat /var/lib/rancher/k3s/server/node-token

# Install on worker nodes
curl -sfL https://get.k3s.io | K3S_URL=https://control-plane:6443 K3S_TOKEN=<token> sh -

Pros:

• Extremely simple installation (single command)
• Lightweight (< 512MB RAM)
• Built-in container runtime (containerd)
• Automatic updates via Rancher System Upgrade Controller
• Great for edge and homelab use cases

Cons:

• Less customizable than kubeadm
• Some features removed (e.g., in-tree storage, cloud providers)
• Slightly different from upstream Kubernetes

3. MicroK8s (Canonical’s Distribution)

Installation:

# Install via snap
sudo snap install microk8s --classic

# Join cluster
sudo microk8s add-node
# Run output command on worker nodes

# Enable addons
microk8s enable dns storage ingress

Pros:

• Zero-ops, single package install
• Snap-based automatic updates
• Addons for common services (DNS, storage, ingress)
• Canonical support available

Cons:

• Requires snap (not universally liked)
• Less ecosystem compatibility than vanilla Kubernetes
• Ubuntu-specific (less portable)

Cluster Initialization Sequence

kubeadm Approach

sequenceDiagram
    participant Admin
    participant Server as Ubuntu Server
    participant K8s as Kubernetes Components
    
    Admin->>Server: Install Ubuntu 24.04 LTS
    Server->>Server: Configure network (static IP)
    Admin->>Server: Update system (apt update && upgrade)
    Admin->>Server: Install containerd
    Server->>Server: Configure containerd (CRI)
    Admin->>Server: Install kubeadm/kubelet/kubectl
    Server->>Server: Disable swap, configure kernel modules
    Admin->>K8s: kubeadm init --pod-network-cidr=10.244.0.0/16
    K8s->>Server: Generate certificates
    K8s->>Server: Start etcd
    K8s->>Server: Start API server
    K8s->>Server: Start controller-manager
    K8s->>Server: Start scheduler
    K8s-->>Admin: Control plane ready
    Admin->>K8s: kubectl apply -f calico.yaml
    K8s->>Server: Deploy CNI pods
    Admin->>K8s: kubeadm join (on workers)
    K8s->>Server: Add worker nodes
    K8s-->>Admin: Cluster ready

k3s Approach

sequenceDiagram
    participant Admin
    participant Server as Ubuntu Server
    participant K3s as k3s Components
    
    Admin->>Server: Install Ubuntu 24.04 LTS
    Server->>Server: Configure network (static IP)
    Admin->>Server: Update system
    Admin->>Server: curl -sfL https://get.k3s.io | sh -
    Server->>K3s: Download k3s binary
    K3s->>Server: Configure containerd
    K3s->>Server: Start k3s service
    K3s->>Server: Initialize etcd (embedded)
    K3s->>Server: Start API server
    K3s->>Server: Start controller-manager
    K3s->>Server: Start scheduler
    K3s->>Server: Deploy built-in CNI (Flannel)
    K3s-->>Admin: Control plane ready
    Admin->>Server: Retrieve node token
    Admin->>Server: Install k3s agent on workers
    K3s->>Server: Join workers to cluster
    K3s-->>Admin: Cluster ready (5-10 minutes total)

Maintenance Requirements

OS Updates

Security Patches:

# Automatic security updates (recommended)
sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# Manual updates
sudo apt-get update
sudo apt-get upgrade

Frequency:

• Security patches: Weekly to monthly
• Kernel updates: Monthly (may require reboot)
• Major version upgrades: Every 2 years (LTS to LTS)

Kubernetes Upgrades

kubeadm Upgrade:

# Upgrade control plane
sudo apt-get update
sudo apt-get install -y kubeadm=1.32.0-*
sudo kubeadm upgrade apply v1.32.0
sudo apt-get install -y kubelet=1.32.0-* kubectl=1.32.0-*
sudo systemctl restart kubelet

# Upgrade workers
kubectl drain <node> --ignore-daemonsets
sudo apt-get install -y kubeadm=1.32.0-* kubelet=1.32.0-* kubectl=1.32.0-*
sudo kubeadm upgrade node
sudo systemctl restart kubelet
kubectl uncordon <node>

k3s Upgrade:

# Manual upgrade
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.0+k3s1 sh -

# Automatic upgrade via system-upgrade-controller
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml

Upgrade Frequency: Every 3-6 months (Kubernetes minor versions)

Resource Overhead

Minimal Installation (Ubuntu Server + k3s):

• RAM: ~512MB (OS) + 512MB (k3s) = 1GB total
• CPU: 1 core minimum, 2 cores recommended
• Disk: 10GB (OS) + 10GB (container images) = 20GB
• Network: 1 Gbps recommended

Full Installation (Ubuntu Server + kubeadm):

• RAM: ~512MB (OS) + 1-2GB (Kubernetes components) = 2GB+ total
• CPU: 2 cores minimum
• Disk: 15GB (OS) + 20GB (container images/etcd) = 35GB
• Network: 1 Gbps recommended

Security Posture

Strengths:

• Regular security updates via Ubuntu Security Team
• AppArmor enabled by default
• SELinux support available
• Kernel hardening features (ASLR, stack protection)
• Ubuntu Pro ESM for extended CVE coverage (free for personal use)

Attack Surface:

• Full general-purpose OS (larger attack surface than minimal OS)
• Many installed packages by default (can be minimized)
• Requires manual hardening for production use

Hardening Steps:

# Disable unnecessary services
sudo systemctl disable snapd.service
sudo systemctl disable bluetooth.service

# Configure firewall
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 6443/tcp  # Kubernetes API
sudo ufw allow 10250/tcp # Kubelet
sudo ufw enable

# CIS Kubernetes Benchmark compliance
# Use tools like kube-bench for validation

Learning Curve

Ease of Adoption: ⭐⭐⭐⭐⭐ (Excellent)

• Most familiar Linux distribution for many users
• Extensive documentation and tutorials
• Large community support (forums, Stack Overflow)
• Straightforward package management
• Similar to Debian-based systems

Required Knowledge:

• Basic Linux system administration (apt, systemd, networking)
• Kubernetes concepts (pods, services, deployments)
• Container runtime basics (containerd, Docker)
• Text editor (vim, nano) for configuration

Community Support

Ecosystem Maturity: ⭐⭐⭐⭐⭐ (Excellent)

• Documentation: Comprehensive official docs, community guides
• Community: Massive user base, active forums
• Commercial Support: Available from Canonical (Ubuntu Pro)
• Third-Party Tools: Excellent compatibility with all Kubernetes tools
• Tutorials: Abundant resources for Kubernetes on Ubuntu

Resources:

• Ubuntu Server Documentation
• Kubernetes on Ubuntu Guide
• k3s Documentation
• MicroK8s Documentation

Pros and Cons Summary

Pros

• Good, because most familiar and well-documented Linux distribution
• Good, because 5-year LTS support (10 years with Ubuntu Pro)
• Good, because multiple Kubernetes installation options (kubeadm, k3s, MicroK8s)
• Good, because k3s provides extremely simple setup (single command)
• Good, because extensive package ecosystem (60,000+ packages)
• Good, because strong community support and resources
• Good, because automatic security updates available
• Good, because low learning curve for most administrators
• Good, because compatible with all Kubernetes tooling and addons
• Good, because Ubuntu Pro free for personal use (extended security)

Cons

• Bad, because general-purpose OS has larger attack surface than minimal OS
• Bad, because more resource overhead than purpose-built Kubernetes OS (1-2GB RAM)
• Bad, because requires manual OS updates and reboots
• Bad, because kubeadm setup is complex with many manual steps
• Bad, because snap packages controversial (for MicroK8s)
• Bad, because Kubernetes upgrades require manual intervention (unless using k3s auto-upgrade)
• Bad, because managing OS + Kubernetes lifecycle separately increases complexity
• Neutral, because many preinstalled packages (can be removed, but require effort)

Recommendations

Best for:

• Users familiar with Ubuntu/Debian ecosystem
• Homelabs requiring general-purpose server functionality (not just Kubernetes)
• Teams wanting multiple Kubernetes installation options
• Users prioritizing community support and documentation

Best Installation Method:

• Homelab/Learning: k3s (simplest, auto-updates, lightweight)
• Production-like: kubeadm (full control, upstream Kubernetes)
• Ubuntu-specific: MicroK8s (Canonical support, snap-based)

Avoid if:

• Seeking minimal attack surface (consider Talos Linux)
• Want infrastructure-as-code for OS layer (consider Talos Linux)
• Prefer hyperconverged platform (consider Harvester)

2 - Fedora Analysis

Overview

Fedora Server is a cutting-edge Linux distribution sponsored by Red Hat, serving as the upstream for Red Hat Enterprise Linux (RHEL). It emphasizes innovation with the latest software packages and kernel versions.

Key Facts:

• Latest Version: Fedora 41 (October 2024)
• Support Period: ~13 months per release (shorter than Ubuntu LTS)
• Kernel: Linux 6.11+ (latest stable)
• Package Manager: DNF/RPM, Flatpak
• Init System: systemd

Kubernetes Installation Methods

Fedora supports standard Kubernetes installation approaches:

1. kubeadm (Official Kubernetes Tool)

Installation:

# Install container runtime (CRI-O preferred on Fedora)
sudo dnf install -y cri-o
sudo systemctl enable --now crio

# Add Kubernetes repository
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
EOF

# Install kubeadm, kubelet, kubectl
sudo dnf install -y kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

Cluster Initialization:

# Initialize control plane
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/crio/crio.sock

# Configure kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Install CNI
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

# Join workers
kubeadm token create --print-join-command

Pros:

• CRI-O is native to Fedora ecosystem (same as RHEL/OpenShift)
• Latest Kubernetes versions available quickly
• Familiar to RHEL/CentOS users
• Fully upstream Kubernetes

Cons:

• Manual setup process (same as Ubuntu/kubeadm)
• Requires Kubernetes knowledge
• More complex than turnkey solutions

2. k3s (Lightweight Kubernetes)

Installation:

# Same single-command install
curl -sfL https://get.k3s.io | sh -

# Retrieve token
sudo cat /var/lib/rancher/k3s/server/node-token

# Install on workers
curl -sfL https://get.k3s.io | K3S_URL=https://control-plane:6443 K3S_TOKEN=<token> sh -

Pros:

• Simple installation (identical to Ubuntu)
• Lightweight and fast
• Well-tested on Fedora/RHEL family

Cons:

• Less customizable
• Not using native CRI-O by default (uses embedded containerd)

3. OKD (OpenShift Kubernetes Distribution)

Installation (Single-Node):

# Download and install OKD
wget https://github.com/okd-project/okd/releases/download/4.15.0-0.okd-2024-01-27-070424/openshift-install-linux-4.15.0-0.okd-2024-01-27-070424.tar.gz
tar -xvf openshift-install-linux-*.tar.gz
sudo mv openshift-install /usr/local/bin/

# Create install config
./openshift-install create install-config --dir=cluster

# Install cluster
./openshift-install create cluster --dir=cluster

Pros:

• Enterprise features (operators, web console, image registry)
• Built-in CI/CD and developer tools
• Based on Fedora CoreOS (immutable, auto-updating)

Cons:

• Very heavy resource requirements (16GB+ RAM)
• Complex installation and management
• Overkill for simple homelab use

Cluster Initialization Sequence

kubeadm with CRI-O

sequenceDiagram
    participant Admin
    participant Server as Fedora Server
    participant K8s as Kubernetes Components
    
    Admin->>Server: Install Fedora 41
    Server->>Server: Configure network (static IP)
    Admin->>Server: Update system (dnf update)
    Admin->>Server: Install CRI-O
    Server->>Server: Configure CRI-O runtime
    Server->>Server: Enable crio.service
    Admin->>Server: Install kubeadm/kubelet/kubectl
    Server->>Server: Disable swap, load kernel modules
    Server->>Server: Configure SELinux (permissive for Kubernetes)
    Admin->>K8s: kubeadm init --cri-socket=unix:///var/run/crio/crio.sock
    K8s->>Server: Generate certificates
    K8s->>Server: Start etcd
    K8s->>Server: Start API server
    K8s->>Server: Start controller-manager
    K8s->>Server: Start scheduler
    K8s-->>Admin: Control plane ready
    Admin->>K8s: kubectl apply CNI
    K8s->>Server: Deploy CNI pods
    Admin->>K8s: kubeadm join (workers)
    K8s->>Server: Add worker nodes
    K8s-->>Admin: Cluster ready

k3s Approach

sequenceDiagram
    participant Admin
    participant Server as Fedora Server
    participant K3s as k3s Components
    
    Admin->>Server: Install Fedora 41
    Server->>Server: Configure network
    Admin->>Server: Update system (dnf update)
    Admin->>Server: Disable firewalld (or configure)
    Admin->>Server: curl -sfL https://get.k3s.io | sh -
    Server->>K3s: Download k3s binary
    K3s->>Server: Configure containerd
    K3s->>Server: Start k3s service
    K3s->>Server: Initialize embedded etcd
    K3s->>Server: Start API server
    K3s->>Server: Deploy built-in CNI
    K3s-->>Admin: Control plane ready
    Admin->>Server: Retrieve node token
    Admin->>Server: Install k3s agent on workers
    K3s->>Server: Join workers
    K3s-->>Admin: Cluster ready (5-10 minutes)

Maintenance Requirements

OS Updates

Security and System Updates:

# Automatic updates (dnf-automatic)
sudo dnf install -y dnf-automatic
sudo systemctl enable --now dnf-automatic.timer

# Manual updates
sudo dnf update -y
sudo reboot  # if kernel updated

Frequency:

• Security patches: Weekly to monthly
• Kernel updates: Monthly (frequent updates)
• Major version upgrades: Every ~13 months (Fedora releases)

Version Upgrade:

# Upgrade to next Fedora release
sudo dnf upgrade --refresh
sudo dnf install dnf-plugin-system-upgrade
sudo dnf system-upgrade download --releasever=42
sudo dnf system-upgrade reboot

Kubernetes Upgrades

kubeadm Upgrade:

# Upgrade control plane
sudo dnf update -y kubeadm
sudo kubeadm upgrade apply v1.32.0
sudo dnf update -y kubelet kubectl
sudo systemctl restart kubelet

# Upgrade workers
kubectl drain <node> --ignore-daemonsets
sudo dnf update -y kubeadm kubelet kubectl
sudo kubeadm upgrade node
sudo systemctl restart kubelet
kubectl uncordon <node>

k3s Upgrade: Same as Ubuntu (curl script or system-upgrade-controller)

Upgrade Frequency: Kubernetes every 3-6 months, Fedora OS every ~13 months

Resource Overhead

Minimal Installation (Fedora Server + k3s):

• RAM: ~600MB (OS) + 512MB (k3s) = 1.2GB total
• CPU: 1 core minimum, 2 cores recommended
• Disk: 12GB (OS) + 10GB (containers) = 22GB
• Network: 1 Gbps recommended

Full Installation (Fedora Server + kubeadm + CRI-O):

• RAM: ~700MB (OS) + 1.5GB (Kubernetes) = 2.2GB total
• CPU: 2 cores minimum
• Disk: 15GB (OS) + 20GB (containers) = 35GB
• Network: 1 Gbps recommended

Note: Slightly higher overhead than Ubuntu due to SELinux and newer components.

Security Posture

Strengths:

• SELinux enabled by default (stronger than AppArmor)
• Latest security patches and kernel (bleeding edge)
• CRI-O container runtime (security-focused, used by OpenShift)
• Shorter support window = less legacy CVEs
• Active security team and rapid response

Attack Surface:

• General-purpose OS (larger surface than minimal OS)
• More installed packages than minimal server
• SELinux can be complex to configure for Kubernetes

Hardening Steps:

# Configure firewall (firewalld default on Fedora)
sudo firewall-cmd --permanent --add-port=6443/tcp  # API server
sudo firewall-cmd --permanent --add-port=10250/tcp # Kubelet
sudo firewall-cmd --reload

# SELinux configuration for Kubernetes
sudo setenforce 0  # Permissive (Kubernetes not fully SELinux-ready)
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

# Disable unnecessary services
sudo systemctl disable bluetooth.service

Learning Curve

Ease of Adoption: ⭐⭐⭐⭐ (Good)

• Familiar for RHEL/CentOS/Alma/Rocky users
• DNF package manager (similar to APT)
• Excellent documentation
• SELinux learning curve can be steep

Required Knowledge:

• RPM-based system administration (dnf, systemd)
• SELinux basics (or willingness to use permissive mode)
• Kubernetes concepts
• Firewalld configuration

Differences from Ubuntu:

• DNF vs APT package manager
• SELinux vs AppArmor
• Firewalld vs UFW
• Faster release cycle (more frequent upgrades)

Community Support

Ecosystem Maturity: ⭐⭐⭐⭐ (Good)

• Documentation: Excellent official docs, Red Hat resources
• Community: Large user base, active forums
• Commercial Support: RHEL support available (paid)
• Third-Party Tools: Good compatibility with Kubernetes tools
• Tutorials: Abundant resources, especially for RHEL ecosystem

Resources:

• Fedora Server Documentation
• Fedora Kubernetes SIG
• CRI-O Documentation
• k3s on Fedora

Pros and Cons Summary

Pros

• Good, because latest kernel and software packages (bleeding edge)
• Good, because SELinux enabled by default (stronger MAC than AppArmor)
• Good, because native CRI-O support (same as RHEL/OpenShift)
• Good, because upstream for RHEL (enterprise compatibility)
• Good, because multiple Kubernetes installation options
• Good, because k3s simplifies setup dramatically
• Good, because strong security focus and rapid CVE response
• Good, because familiar to RHEL/CentOS ecosystem
• Good, because automatic updates available (dnf-automatic)
• Neutral, because shorter support cycle (13 months) ensures latest features

Cons

• Bad, because short support cycle requires frequent OS upgrades (every ~13 months)
• Bad, because bleeding-edge packages can introduce instability
• Bad, because SELinux configuration for Kubernetes is complex (often set to permissive)
• Bad, because smaller community than Ubuntu (though still large)
• Bad, because general-purpose OS has larger attack surface than minimal OS
• Bad, because more resource overhead than purpose-built Kubernetes OS
• Bad, because OS upgrade every 13 months adds maintenance burden
• Bad, because less beginner-friendly than Ubuntu
• Bad, because managing OS + Kubernetes lifecycle separately
• Neutral, because rapid release cycle can be pro or con depending on preference

Recommendations

Best for:

• Users familiar with RHEL/CentOS/Rocky/Alma ecosystem
• Teams wanting latest kernel and software features
• Environments requiring SELinux (compliance, enterprise standards)
• Learning OpenShift/OKD ecosystem (Fedora CoreOS foundation)
• Users comfortable with frequent OS upgrades

Best Installation Method:

• Homelab/Learning: k3s (simplest, lightweight)
• Enterprise-like: kubeadm + CRI-O (OpenShift compatibility)
• Advanced: OKD (if resources available, 16GB+ RAM)

Avoid if:

• Prefer long-term stability (choose Ubuntu LTS)
• Want minimal maintenance (frequent Fedora upgrades required)
• Seeking minimal attack surface (consider Talos Linux)
• Uncomfortable with SELinux complexity
• Want infrastructure-as-code for OS (consider Talos Linux)

Comparison with Ubuntu

Verdict: Fedora is excellent for those wanting latest features and comfortable with frequent upgrades. Ubuntu LTS is better for long-term stability and minimal maintenance.

3 - Talos Linux Analysis

Overview

Talos Linux is a modern operating system designed specifically for running Kubernetes. It is API-driven, immutable, and minimal, with no SSH access, shell, or package manager. All configuration is done via a declarative API.

Key Facts:

• Latest Version: Talos 1.9 (supports Kubernetes 1.31)
• Support: Community-driven, commercial support available from Sidero Labs
• Kernel: Linux 6.6+ LTS
• Architecture: Immutable, API-driven, no shell access
• Management: talosctl CLI + Kubernetes API

Kubernetes Installation Methods

Talos Linux has built-in Kubernetes - there is only one installation method.

Built-in Kubernetes (Only Option)

Installation Process:

1. Boot Talos ISO/PXE (maintenance mode)
2. Apply machine configuration via talosctl
3. Bootstrap Kubernetes via talosctl bootstrap

Machine Configuration (YAML):

# controlplane.yaml
version: v1alpha1
machine:
  type: controlplane
  install:
    disk: /dev/sda
  network:
    hostname: control-plane-1
    interfaces:
      - interface: eth0
        dhcp: false
        addresses:
          - 192.168.1.10/24
        routes:
          - network: 0.0.0.0/0
            gateway: 192.168.1.1
cluster:
  clusterName: homelab
  controlPlane:
    endpoint: https://192.168.1.10:6443
  network:
    cni:
      name: custom
      urls:
        - https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

Cluster Initialization:

# Generate machine configs
talosctl gen config homelab https://192.168.1.10:6443

# Apply config to control plane node (booted from ISO)
talosctl apply-config --insecure --nodes 192.168.1.10 --file controlplane.yaml

# Wait for install to complete, then bootstrap
talosctl bootstrap --nodes 192.168.1.10 --endpoints 192.168.1.10

# Retrieve kubeconfig
talosctl kubeconfig --nodes 192.168.1.10 --endpoints 192.168.1.10

# Apply config to worker nodes
talosctl apply-config --insecure --nodes 192.168.1.11 --file worker.yaml

Pros:

• Kubernetes built-in, no separate installation
• Declarative configuration (GitOps-friendly)
• Extremely minimal attack surface (no shell, no SSH)
• Immutable infrastructure (config changes require reboot)
• Automatic updates via Talos controller
• Designed from ground up for Kubernetes

Cons:

• Steep learning curve (completely different paradigm)
• No SSH/shell access (all via API)
• Troubleshooting requires different mindset
• Limited to Kubernetes workloads only (not general-purpose)
• Smaller community than traditional distros

Cluster Initialization Sequence

sequenceDiagram
    participant Admin
    participant Server as Bare Metal Server
    participant Talos as Talos Linux
    participant K8s as Kubernetes Components
    
    Admin->>Server: Boot Talos ISO (PXE or USB)
    Server->>Talos: Start in maintenance mode
    Talos-->>Admin: API endpoint ready (no shell)
    Admin->>Admin: Generate configs (talosctl gen config)
    Admin->>Talos: talosctl apply-config (controlplane.yaml)
    Talos->>Server: Partition disk
    Talos->>Server: Install Talos to /dev/sda
    Talos->>Server: Write machine config
    Server->>Server: Reboot from disk
    Talos->>Talos: Load machine config
    Talos->>K8s: Start kubelet
    Talos->>K8s: Start etcd
    Talos->>K8s: Start API server
    Admin->>Talos: talosctl bootstrap
    Talos->>K8s: Initialize cluster
    K8s->>Talos: Start controller-manager
    K8s->>Talos: Start scheduler
    K8s-->>Admin: Control plane ready
    Admin->>K8s: Apply CNI (via talosctl or kubectl)
    K8s->>Talos: Deploy CNI pods
    Admin->>Talos: Apply worker configs
    Talos->>K8s: Join workers to cluster
    K8s-->>Admin: Cluster ready (10-15 minutes)

Maintenance Requirements

OS Updates

Declarative Upgrades:

# Upgrade Talos version (rolling upgrade)
talosctl upgrade --nodes 192.168.1.10 --image ghcr.io/siderolabs/installer:v1.9.0

# Kubernetes version upgrade (also declarative)
talosctl upgrade-k8s --nodes 192.168.1.10 --to 1.32.0

Automatic Updates (via Talos System Extensions):

# machine config with auto-update extension
machine:
  install:
    extensions:
      - image: ghcr.io/siderolabs/system-upgrade-controller

Frequency:

• Talos releases: Every 2-3 months
• Kubernetes upgrades: Follow upstream cadence (quarterly)
• Security patches: Built into Talos releases
• No traditional OS patching (immutable system)

Configuration Changes

All changes via machine config:

# Edit machine config YAML
vim controlplane.yaml

# Apply updated config (triggers reboot if needed)
talosctl apply-config --nodes 192.168.1.10 --file controlplane.yaml

No manual package installs - everything declarative.

Resource Overhead

Minimal Footprint (Talos Linux + Kubernetes):

• RAM: ~256MB (OS) + 512MB (Kubernetes) = 768MB total
• CPU: 1 core minimum, 2 cores recommended
• Disk: ~500MB (OS) + 10GB (container images/etcd) = 10-15GB total
• Network: 1 Gbps recommended

Comparison:

• Ubuntu + k3s: ~1GB RAM
• Talos: ~768MB RAM (lighter)
• Ubuntu + kubeadm: ~2GB RAM
• Talos: ~768MB RAM (much lighter)

Minimal install size: ~500MB (vs 10GB+ for Ubuntu/Fedora)

Security Posture

Strengths: ⭐⭐⭐⭐⭐ (Excellent)

• No SSH access - attack surface eliminated
• No shell - cannot install malware
• No package manager - no additional software installation
• Immutable filesystem - rootfs read-only
• Minimal components: Only Kubernetes and essential services
• API-only access - mTLS-authenticated talosctl
• KSPP compliance: Kernel Self-Protection Project standards
• Signed images: Cryptographically signed Talos images
• Secure Boot support: UEFI Secure Boot compatible

Attack Surface:

• Smallest possible: Only Kubernetes API, kubelet, and Talos API
• ~30 running processes (vs 100+ on Ubuntu/Fedora)
• ~200MB filesystem (vs 5-10GB on Ubuntu/Fedora)

No hardening needed - secure by default.

Security Features:

# Built-in security (example config)
machine:
  sysctls:
    kernel.kptr_restrict: "2"
    kernel.yama.ptrace_scope: "1"
  kernel:
    modules:
      - name: br_netfilter
  features:
    kubernetesTalosAPIAccess:
      enabled: true
      allowedRoles:
        - os:reader

Learning Curve

Ease of Adoption: ⭐⭐ (Challenging)

• Paradigm shift: No shell/SSH, API-only management
• Requires understanding of declarative infrastructure
• Talosctl CLI has learning curve
• Excellent documentation helps
• Different troubleshooting approach (logs via API)

Required Knowledge:

• Kubernetes fundamentals (critical)
• YAML configuration syntax
• Networking basics (especially CNI)
• GitOps concepts helpful
• Comfort with “infrastructure as code”

Debugging without shell:

# View logs via API
talosctl logs --nodes 192.168.1.10 kubelet

# Get system metrics
talosctl dashboard --nodes 192.168.1.10

# Interactive mode (limited shell in emergency)
talosctl dashboard --nodes 192.168.1.10

# Service status
talosctl service --nodes 192.168.1.10

Community Support

Ecosystem Maturity: ⭐⭐⭐ (Growing)

• Documentation: Excellent official docs
• Community: Smaller but very active (Slack, GitHub Discussions)
• Commercial Support: Available from Sidero Labs
• Third-Party Tools: Growing ecosystem (Cluster API, GitOps tools)
• Tutorials: Increasing number of community guides

Resources:

• Talos Linux Documentation
• Talos Community Slack
• Sidero Labs GitHub
• Awesome Talos

Community Size: Smaller than Ubuntu/Fedora, but dedicated and helpful.

Pros and Cons Summary

Pros

• Good, because Kubernetes is built-in (no separate installation)
• Good, because minimal attack surface (no SSH, shell, or package manager)
• Good, because immutable infrastructure (config drift impossible)
• Good, because API-driven management (GitOps-friendly)
• Good, because extremely low resource overhead (~768MB RAM)
• Good, because automatic security patches via Talos upgrades
• Good, because declarative configuration (version-controlled)
• Good, because secure by default (no hardening required)
• Good, because smallest disk footprint (~500MB OS)
• Good, because designed specifically for Kubernetes (opinionated and optimized)
• Good, because UEFI Secure Boot support
• Good, because upgrades are simple and declarative (talosctl upgrade)

Cons

• Bad, because steep learning curve (no shell/SSH paradigm shift)
• Bad, because limited to Kubernetes workloads only (not general-purpose)
• Bad, because troubleshooting without shell requires different approach
• Bad, because smaller community than Ubuntu/Fedora
• Bad, because relatively new (less mature than traditional distros)
• Bad, because no escape hatch for manual intervention
• Bad, because requires comfort with declarative infrastructure
• Bad, because debugging is harder for beginners
• Neutral, because opinionated design (pro for K8s-only, con for general use)

Recommendations

Best for:

• Kubernetes-dedicated infrastructure (no general-purpose workloads)
• Security-focused environments (minimal attack surface)
• GitOps workflows (declarative configuration)
• Immutable infrastructure advocates
• Teams comfortable with API-driven management
• Production Kubernetes clusters (once team is trained)

Best Installation Method:

• Only option: Built-in Kubernetes via talosctl

Avoid if:

• Need general-purpose server functionality (SSH, cron jobs, etc.)
• Team unfamiliar with Kubernetes (too steep a learning curve)
• Require shell access for troubleshooting comfort
• Want traditional package management (apt, dnf)
• Prefer familiar Linux administration tools

Comparison with Ubuntu and Fedora

Verdict: Talos is the most secure and efficient option for Kubernetes-only infrastructure, but requires team buy-in to API-driven, immutable paradigm. Ubuntu/Fedora better for general-purpose servers or teams wanting shell access.

Advanced Features

Talos System Extensions

Extend Talos functionality with extensions:

machine:
  install:
    extensions:
      - image: ghcr.io/siderolabs/intel-ucode:20240312
      - image: ghcr.io/siderolabs/iscsi-tools:v0.1.4

Cluster API Integration

Talos works natively with Cluster API:

# Install Cluster API + Talos provider
clusterctl init --infrastructure talos

# Create cluster from template
clusterctl generate cluster homelab --infrastructure talos > cluster.yaml
kubectl apply -f cluster.yaml

Image Factory

Custom Talos images with extensions:

# Build custom image
curl -X POST https://factory.talos.dev/image \
  -d '{"talos_version":"v1.9.0","extensions":["siderolabs/intel-ucode"]}'

Disaster Recovery

Talos supports etcd backup/restore:

# Backup etcd
talosctl etcd snapshot --nodes 192.168.1.10

# Restore from snapshot
talosctl bootstrap --recover-from ./etcd-snapshot.db

Production Readiness

Production Use: ✅ Yes (many companies run Talos in production)

High Availability:

• 3+ control plane nodes recommended
• External etcd supported
• Load balancer for API server

Monitoring:

• Prometheus metrics built-in
• Talos dashboard for health
• Standard Kubernetes observability tools

Example Production Clusters:

• Sidero Metal (bare metal provisioning)
• Various cloud providers (AWS, GCP, Azure)
• Edge deployments (minimal footprint)

4 - Harvester Analysis

Overview

Harvester is a Hyperconverged Infrastructure (HCI) platform built on Kubernetes, designed to provide VM and container management on a unified platform. It combines compute, storage, and networking with built-in K3s for orchestration.

Key Facts:

• Latest Version: Harvester 1.4 (based on K3s 1.30+)
• Foundation: Built on RancherOS 2.0, K3s, and KubeVirt
• Support: Supported by SUSE (acquired Rancher)
• Architecture: HCI platform with VM + container workloads
• Management: Web UI + kubectl + Rancher integration

Kubernetes Installation Methods

Harvester includes K3s as its foundation - Kubernetes is built-in.

Built-in K3s (Only Option)

Installation Process:

1. Boot Harvester ISO (interactive installer or PXE)
2. Complete installation wizard (web UI or console)
3. Create cluster (automatic K3s deployment)
4. Access via web UI or kubectl

Interactive Installation:

# Boot from Harvester ISO
1. Choose "Create a new Harvester cluster"
2. Configure:
   - Cluster token
   - Node role (management/worker/witness)
   - Network interface (management network)
   - VIP (Virtual IP for cluster access)
   - Storage disk (Longhorn persistent storage)
3. Install completes (15-20 minutes)
4. Access web UI at https://<VIP>

Configuration (cloud-init for automated install):

# config.yaml
token: my-cluster-token
os:
  hostname: harvester-node-1
  modules:
    - kvm
  kernel_parameters:
    - intel_iommu=on
install:
  mode: create
  device: /dev/sda
  iso_url: https://releases.rancher.com/harvester/v1.4.0/harvester-v1.4.0-amd64.iso
  vip: 192.168.1.100
  vip_mode: static
  networks:
    harvester-mgmt:
      interfaces:
        - name: eth0
      default_route: true
      ip: 192.168.1.10
      subnet_mask: 255.255.255.0
      gateway: 192.168.1.1

Pros:

• Complete HCI solution (VMs + containers)
• Web UI for management (no CLI required)
• Built-in storage (Longhorn CSI)
• Built-in networking (multus, SR-IOV)
• VM live migration
• Rancher integration for multi-cluster management
• K3s built-in (no separate Kubernetes install)

Cons:

• Heavy resource requirements (8GB+ RAM per node)
• Complex architecture (steep learning curve)
• Larger attack surface than minimal OS
• Overkill for container-only workloads
• Requires 3+ nodes for production HA

Cluster Initialization Sequence

sequenceDiagram
    participant Admin
    participant Server as Bare Metal Server
    participant Harvester as Harvester HCI
    participant K3s as K3s / KubeVirt
    participant Storage as Longhorn Storage
    
    Admin->>Server: Boot Harvester ISO
    Server->>Harvester: Start installation wizard
    Harvester-->>Admin: Interactive console/web UI
    Admin->>Harvester: Configure cluster (token, VIP, storage)
    Harvester->>Server: Partition disks (OS + Longhorn storage)
    Harvester->>Server: Install RancherOS 2.0 base
    Harvester->>Server: Install K3s components
    Server->>Server: Reboot
    Harvester->>K3s: Start K3s server
    K3s->>Server: Initialize control plane
    K3s->>Server: Deploy Harvester operators
    K3s->>Storage: Deploy Longhorn for persistent storage
    K3s->>Server: Deploy KubeVirt for VM management
    K3s->>Server: Deploy multus CNI (multi-network)
    Harvester-->>Admin: Web UI ready at https://<VIP>
    Admin->>Harvester: Add additional nodes (join cluster)
    Harvester->>K3s: Join nodes to cluster
    K3s->>Storage: Replicate storage across nodes
    Harvester-->>Admin: Cluster ready (20-30 minutes)
    Admin->>Harvester: Create VMs or deploy containers

Maintenance Requirements

OS Updates

Harvester Upgrades (includes OS + K3s):

# Via Web UI:
# Settings → Upgrade → Select version → Start upgrade

# Via kubectl (after downloading upgrade image):
kubectl apply -f https://releases.rancher.com/harvester/v1.4.0/version.yaml

# Monitor upgrade progress
kubectl get upgrades -n harvester-system

Frequency:

• Harvester releases: Every 2-3 months (minor versions)
• Security patches: Included in Harvester releases
• K3s upgrades: Bundled with Harvester upgrades
• No separate OS patching (managed by Harvester)

Kubernetes Upgrades

K3s is upgraded with Harvester - no separate upgrade process.

Version Compatibility:

• Harvester 1.4.x → K3s 1.30+
• Harvester 1.3.x → K3s 1.28+
• Harvester 1.2.x → K3s 1.26+

Upgrade Process:

1. Web UI or kubectl to trigger upgrade
2. Rolling upgrade of nodes (one at a time)
3. VM live migration during node upgrades
4. Automatic rollback on failure

Resource Overhead

Single Node (Harvester HCI):

• RAM: 8GB minimum (16GB recommended for VMs)
• CPU: 4 cores minimum (8 cores recommended)
• Disk: 250GB minimum (SSD recommended)
  • 100GB for OS/Harvester components
  • 150GB+ for Longhorn storage (VM disks)
• Network: 1 Gbps minimum (10 Gbps for production)

Three-Node Cluster (Production HA):

• RAM: 32GB per node (64GB for VM-heavy workloads)
• CPU: 8 cores per node minimum
• Disk: 500GB+ per node (NVMe SSD recommended)
• Network: 10 Gbps recommended (separate storage network ideal)

Comparison:

• Ubuntu + k3s: 1GB RAM
• Talos: 768MB RAM
• Harvester: 8GB+ RAM (much heavier)

Note: Harvester is designed for multi-node HCI, not single-node homelabs.

Security Posture

Strengths:

• SELinux-based (RancherOS 2.0 foundation)
• Immutable OS layer (similar to Talos)
• RBAC built-in (Kubernetes + Rancher)
• Network segmentation (multus CNI)
• VM isolation (KubeVirt)
• Signed images and secure boot support

Attack Surface:

• Larger than Talos/k3s: Includes web UI, VM management, storage layer
• KubeVirt adds additional components
• Web UI is additional attack vector
• More processes than minimal OS (~50+ services)

Security Features:

# VM network isolation example
apiVersion: network.harvesterhci.io/v1beta1
kind: VlanConfig
metadata:
  name: production-vlan
spec:
  vlanID: 100
  uplink:
    linkAttributes: 1500

Hardening:

• Firewall rules (web UI or kubectl)
• RBAC policies (restrict VM/namespace access)
• Network policies (isolate workloads)
• Rancher authentication integration (LDAP, SAML)

Learning Curve

Ease of Adoption: ⭐⭐⭐ (Moderate)

• Web UI simplifies management (no CLI required for basic tasks)
• Requires understanding of VMs + containers
• Kubernetes knowledge helpful but not required initially
• Longhorn storage concepts (replicas, snapshots)
• KubeVirt for VM management (learning curve)

Required Knowledge:

• Basic Kubernetes concepts (pods, services)
• VM management (KubeVirt/libvirt)
• Storage concepts (Longhorn, CSI)
• Networking (VLANs, SR-IOV optional)
• Web UI navigation

Debugging:

# Access via kubectl (kubeconfig from web UI)
kubectl get nodes -n harvester-system

# View Harvester logs
kubectl logs -n harvester-system <pod-name>

# VM console access (via web UI or virtctl)
virtctl console <vm-name>

# Storage debugging
kubectl get volumes -A

Community Support

Ecosystem Maturity: ⭐⭐⭐⭐ (Good)

• Documentation: Excellent official docs
• Community: Active Slack, GitHub Discussions, forums
• Commercial Support: Available from SUSE/Rancher
• Third-Party Tools: Rancher ecosystem integration
• Tutorials: Growing number of guides and videos

Resources:

• Harvester Documentation
• Harvester GitHub
• Rancher Community Slack
• SUSE Support

Pros and Cons Summary

Pros

• Good, because unified platform for VMs + containers (no separate hypervisor)
• Good, because built-in K3s (Kubernetes included)
• Good, because web UI simplifies management (no CLI required)
• Good, because built-in persistent storage (Longhorn CSI)
• Good, because VM live migration (no downtime during maintenance)
• Good, because multi-network support (multus CNI, SR-IOV)
• Good, because Rancher integration (multi-cluster management)
• Good, because automatic upgrades (OS + K3s + components)
• Good, because commercial support available (SUSE)
• Good, because designed for bare-metal HCI (no cloud dependencies)
• Neutral, because immutable OS layer (similar to Talos benefits)

Cons

• Bad, because very heavy resource requirements (8GB+ RAM minimum)
• Bad, because complex architecture (KubeVirt, Longhorn, multus, etc.)
• Bad, because overkill for container-only workloads (use k3s/Talos instead)
• Bad, because larger attack surface than minimal OS (web UI, VM layer)
• Bad, because requires 3+ nodes for production HA (not single-node friendly)
• Bad, because steep learning curve for full feature set (VMs + storage + networking)
• Bad, because relatively new platform (less mature than Ubuntu/Fedora)
• Bad, because limited to Rancher ecosystem (vendor lock-in)
• Bad, because slower to adopt latest Kubernetes versions (depends on K3s bundle)
• Neutral, because opinionated HCI design (pro for VM use cases, con for simplicity)

Recommendations

Best for:

• Hybrid workloads (VMs + containers on same platform)
• Homelab users wanting to consolidate VM hypervisor + Kubernetes
• Teams familiar with Rancher ecosystem
• Multi-node clusters (3+ nodes)
• Environments requiring VM live migration
• Users wanting web UI for infrastructure management
• Replacing VMware/Proxmox + Kubernetes with unified platform

Best Installation Method:

• Only option: Interactive ISO install or PXE with cloud-init

Avoid if:

• Running container-only workloads (use k3s or Talos instead)
• Limited resources (< 8GB RAM per node)
• Single-node homelab (Harvester designed for multi-node)
• Want minimal attack surface (use Talos)
• Prefer traditional Linux shell access (use Ubuntu/Fedora)
• Need latest Kubernetes versions immediately (Harvester lags upstream)

Comparison with Other Options

Verdict: Harvester is excellent for VM + container hybrid workloads with 3+ nodes, but overkill for container-only infrastructure. Use Talos or k3s for Kubernetes-only clusters, Ubuntu/Fedora for general-purpose servers.

Advanced Features

VM Management (KubeVirt)

Create VMs via YAML:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: ubuntu-vm
spec:
  running: true
  template:
    spec:
      domain:
        devices:
          disks:
            - name: root
              disk:
                bus: virtio
        resources:
          requests:
            memory: 4Gi
            cpu: 2
      volumes:
        - name: root
          containerDisk:
            image: docker.io/harvester/ubuntu:22.04

Live Migration

Move VMs between nodes:

# Via web UI: VM → Actions → Migrate

# Via kubectl
kubectl patch vm ubuntu-vm --type merge -p '{"spec":{"running":false}}'
kubectl patch vm ubuntu-vm --type merge -p '{"spec":{"running":true}}'

Backup and Restore

Harvester supports VM backups:

# Configure S3 backup target (web UI)
# Create VM snapshot
# Restore from snapshot or backup

Rancher Integration

Manage multiple clusters:

# Import Harvester cluster into Rancher
# Deploy workloads across clusters
# Central authentication and RBAC

Use Case Examples

Use Case 1: Replace VMware + Kubernetes

Scenario: Currently running VMware ESXi for VMs + separate Kubernetes cluster

Harvester Solution:

• Consolidate to 3-node Harvester cluster
• Migrate VMs to KubeVirt
• Deploy containers on same cluster
• Save VMware licensing costs

Benefits:

• Single platform for VMs + containers
• Unified management (web UI + kubectl)
• Built-in HA and live migration

Use Case 2: Homelab with Mixed Workloads

Scenario: Need Windows VMs + Linux containers + storage server

Harvester Solution:

• Windows VMs via KubeVirt (GPU passthrough supported)
• Linux containers via K3s workloads
• Longhorn for persistent storage (NFS export supported)

Benefits:

• No need for separate Proxmox/ESXi
• Kubernetes-native management
• Learn enterprise HCI platform

Use Case 3: Edge Computing

Scenario: Deploy compute at remote sites (3-5 nodes each)

Harvester Solution:

• Harvester cluster at each edge location
• Rancher for central management
• VM + container workloads

Benefits:

• Autonomous operation (no cloud dependency)
• Rancher multi-cluster management
• Built-in storage and networking

Production Readiness

Production Use: ✅ Yes (used in enterprise environments)

High Availability:

• 3+ nodes required for HA
• Witness node for even-node clusters
• VM live migration during maintenance
• Longhorn 3-replica storage

Monitoring:

• Built-in Prometheus + Grafana
• Rancher monitoring integration
• Alerting and notifications

Disaster Recovery:

• VM backups to S3
• Cluster backups (etcd + config)
• Restore to new cluster

Enterprise Features:

• Rancher authentication (LDAP, SAML, OAuth)
• Multi-tenancy (namespaces, RBAC)
• Audit logging
• Network policies