web-log/container/site/content/posts/infrastructure-overhaul/index.md

+++ categories = ["software"] tags = ["hypervisor","virtualization"] date = 2025-06-27T19:00:00-05:00 description = "Separating storage from application services across dedicated machines and redesigning the home network's virtualized infrastructure." draft = false slug = "infrastructure-overhaul" title = "🔨 Infrastructure Overhaul" author = "nicholas" +++

This overhaul of my infrastructure is mostly about separating concerns. The two main concerns being separated are: storage and services. Mainly, I am moving all of the containers off my storage server and onto their own machine. They call this storage-service decoupling. There are also some improvements I will make along the way.

⚠️ Problems

There are a couple main problems with current configuration that I will need to address:

• Storage-device coupling
• Miscellaneous infrastructure smell / anti-patterns

🔗 Storage-Service Decoupling

I was running a lot of containers on my so-called "storage" server, and this was eating into resources originally meant for ZFS, mainly RAM. Every service I added was jeopardizing the performance and integrity of my most precious piece of infrastructure. There is no inherent problem with this configuration, and there are even good reasons to have services local to the storage. After all, ZFS managed to carry on just fine for years in spite of being somewhat RAM-starved. However, I wanted to move services off my storage server so it could be free to simply store things in the way ZFS was designed. It seems the proper way.

💩 Miscellaneous Infrastructure Anti-Patterns

The overhaul of my infrastructure is also meant to remove a general sense that things were wrong. I was running Docker Desktop in Windows and Docker in Linux, across two different machines. Worse still, I was running a Kubernetes swarm inside a Docker container on a NAS server. Resources were spread across multiple machines and containers. Things like this, they smell. It is not necessarily wrong, but it smells wrong and I do not like it.

✅ Solutions

I go through the general solution to each of the problems presented above.

1️⃣ MACHINE 1 (NAS Server)

Addressing storage-service coupling, anti-patterns

(Debian → TrueNAS SCALE)

Since the storage server is no longer general purpose compute and storage, which requires the flexibility of a good general purpose OS (Debian 12), but instead a dedicated NAS server, I replace the OS with a dedicated NAS OS (TrueNAS SCALE) to make storage server management easier. TrueNAS SCALE comes with a all of the management tools/features I need.

I move the docker services off the Debian host and onto a VM on another machine.

NAS Configuration Diagram

graph LR
subgraph host2["NAS (TrueNAS SCALE)"]
        subgraph zpool[ZFS Pool]
            subgraph nfs[NFS Dataset]
                nfsshare[NFS Shares]
            end
            subgraph smb[SMB Dataset]
               smbshare[SMB Shares]
            end
        end
    end

---

🔂 Process

The process of migrating from Debian to TrueNAS SCALE is straightforward, mostly thanks to highly portable nature of ZFS pools and my decision to store almost everything important there. I do not need to move any Docker-related files since they are stored on the ZFS pool. I can painlessly wipe the OS disk without worrying about destroying anything important. This is the way it should be.

The rest of the process involves configuration in TrueNAS SCALE, including things like configuring datasets, shares, users, email alerts, etc.

• Make bootable USB with Etcher tool
• Make backups of important OS disk configuration files, etc.
  • cron, etc.
• Export ZFS pool
  • sudo zpool export data-pool
• Boot into TrueNAS SCALE installer environment and install OS
• Import ZFS pool in TrueNas SCALE web GUI

✅ Done. Basic storage functionality is now restored to the NAS server.

{{< image
src="images/truenas-scale-memory-usage.PNG" caption="TrueNAS SCALE using every bit of RAM it can" >}}

{{< image src="images/truenas-scale-storage.PNG" caption="TrueNAS SCALE storage information" >}}

---

💾 ZFS, Datasets Configurations

Next I need to complete some configuration to restore complete functionality to my service and network storage.

• Creating ZFS datasets for each service
  • ✅ Separation of concerns -- isolates data per service. Makes resources more manageable, secure, fault tolerant
  • ✅ Granular control per dataset (compression, quotas, ACLs)
  • ✅ Snapshot/replication: Enables service-specific backups
• Create, configure POSIX service users, groups
• Configure POSIX permissions, ACLs for users
  • ✅ Principle of least privilege
  • ✅ ACLs allow flexible permit
• Share datasets
  • NFS shares (Linux clients):
    • Kubernetes PVs
  • SMB shares (Windows clients):
    • Nicholas
    • User 2
    • ...
  • ✅ Dedicated storage per workload: benefit from ZFS performance and integrity

📧 Email Alerts Configuration

I configured my old Debian OS to [send email via Amazon SES]({{< relref "posts/email-alerts">}}), so I will want to replicate this in TrueNAS SCALE. Luckily, I do not have to re-implement my email alerts since this feature is built-in to the OS. I just need to configure a few things in TrueNAS GUI:

{{< image src="images/truenas-scale-email-alert-configuration.png" caption="📧 TrueNAS SCALE alert email configuration" width="400" >}}

{{< image src="images/truenas-scale-alert-email-test.png" caption="✅ TrueNAS SCALE alert email test success" >}}

That is pretty much it.

2️⃣ MACHINE 2 (SERVER)

Windows + Hyper-V

This machine runs on relatively powerful hardware, which I would not like to waste by handing it all over to container services. The host OS will remain Windows, which I will use as a normal workstation. The Docker and Kubernetes services will run in virtual machines under Hyper-V hypervisor.

graph LR
    subgraph host1["SERVER (Windows)"]
        subgraph hypervisor["Hypervisor (Hyper-V)"]
            subgraph vm1[Ubuntu Server VM 1]
                docker[Docker]
            end
            subgraph vm2[Ubuntu Server VM 2]
                kubernetes[Kubernetes host]
            end
        end 
    end

🔂 Process

All I need to do here is install Windows OS on the machine and provision 2 VMs. The basic setup process is as follows:

• Make bootable USB with Etcher tool
• Make backups of important files:
  • Basically nothing. Once again, everything I care about is stored on the NAS server pool.
• Boot into Windows installer environment and install OS
• Configure 2 VMs:
  • Ubuntu Server (Docker host)
  • Ubuntu Server (Kubernetes host)

✅ Done. Basic desktop service and container infrastructure is restored to Server.

{{< image src="images/windows-hyper-v-containers.png" caption="Windows Hyper-V containers" >}}

🖥️ VM Configuration

• Install Ubuntu Server on each VM
• On Kubernetes VM:
  • Install Snap Microk8s package
• On Docker VM:
  • Install Docker Engine
• Create Netplan configuration files:
  • Static IPs
  • DNS search address

🌐 DNS Configuration

I need to configure my DNS to resolve the correct addresses inside my network. Each service is hosted at its own URL, e.g. https://personal-website.domain.com. To reach this, I need:

• A records for each host
• CNAME records for each service

Record Type	Name	Value
A	docker.domain.com	192.168.1.2
A	kubernetes.domain.com	192.168.1.3
CNAME	personal-website.domain.com	kubernetes.domain.com
CNAME	pastebin.domain.com	docker.domain.com

This set of DNS records will resolve client requests to the right hosts. On each host, traffic will need to be routed to the right container/service. To do this on the Docker host, I use NGINX reverse proxy to route incoming requests to the correct backend container based on the domain name.

e.g.

Service	FQDN	External Port	Container Port
PasteBin	pastebin.domain.com	443	8000

On the Kubernetes host, I use Ingress controller (NGINX-based) that serves as the reverse proxy in the same way.

---

👀 Infrastructure Diagrams

I diagram the relationships between the NAS server and the Linux and Windows clients below. I produce 2 different diagrams only for clarity. Also for clarity, I only show example services and storage, not the actual configuration present on my machine.

🗺️ Infrastructure topology (Windows Client)

graph TD
    subgraph windows[Windows]
        n[nicholas]
        user2[User 2]
    end
    subgraph nas[NAS]
        subgraph smb[SMB Shares]
            smbuser2[User 2 Data]
            smbn[Nicholas Data]
        end
    end

    n <-- SMB share --> smbn
    user2 <-- SMB share --> smbuser2

---

🗺️ Infrastructure topology (Linux Clients)

graph TD
    subgraph windows[Windows VMs]
        subgraph kubernetes[Kubernetes]
            weblog[Web Log]
        end
        subgraph docker[Docker]
            pastebin[PasteBin]
        end
    end
    subgraph nas[NAS]
        subgraph nfs[NFS Shares]
            weblogfiles[Web Log Files]
            pastebindata[Pastebin Files]
        end
    end

    weblogfiles <-- Kubernetes Persistent Volume --> weblog
    pastebindata <-- Docker Volume --> pastebin

---

📝 Summary

• Move containers/services off NAS server
• Switch NAS server OS to TrueNAS SCALE for purpose-built storage management features
• Configure two VMs that become new Kubernetes and Docker hosts
• Host VMs in Hyper-V hypervisor on Windows OS
• Centralize persistent container storage in Kubernetes and Docker using NAS server NFS/SMB shares

✅ Done.