When ChatGPT announced their LLM to the world, I immediately started using it to write code. I’m huge into automation and have a list of projects and ideas I would like to get executed. Problem is I’m one measly weak human that requires sleep, food and such… I would open the console, ask a question, get a response or a code snippet, test, iterate, test, iterate, test, rinse and repeat. I would get useful code sometimes, but other times I would ask for something resembling a simple sandwich and was served with a 12-course meal and all the fixings but not a sandwich in sight. Exceptionally exciting if you want to see what it can do, but not very useful when you have a detailed task list and a deadline.

So began my journey to figure out what actually works.

Phase 1 - Prompt Engineering

Initially we believed that the key to making this work was to improve the prompt. It’s a computer after all and it’s supposed to follow instructions, right? If it’s making a mistake it’s probably because I wasn’t clear enough or left a detail out, something…

So we ended up writing long detailed prompts with examples, gotchas, lists of do/don’t actions, etc. everything we could think of to make sure the system knew what we wanted.

This only got us so far. It worked a bit but definitely didn’t give the results we needed and certainly wouldn’t ship production-ready code. We also noticed that the agents would have a tendency to overcomplicate simple solutions—the wheelbarrow becomes a tricycle big rig. It works but definitely not what you need, way too complicated and a city-sized mountain of technical debt right out the gate.

Not the solution.

Phase 2 - Gates/Guardrails

As a systems admin I want to trust the process and remove the human element as much as possible. I also believe that people, if given enough opportunity, will find ways around constraints. So we took a different approach: we don’t let the agent do the thing—we build a process that ensures it does the thing correctly, so the process has permission but the agent doesn’t.

That didn’t work out. The agent would work and realize there is a guardrail and spend the time and tokens trying to circumvent the git-hooks rather than fixing the linting problems. It would spend more time trying to get around the guardrail than actually doing the work.

Not the solution.

Phase 3 - Structure

A little digression first:

Jocko Willink has a great podcast and one of the things he talks about is “Discipline Equals Freedom.” The principles are outlined in his book Discipline Equals Freedom: Field Manual Mk1-MOD1. The more structure you have, the more freedom you have to operate. This is counter-intuitive to most people as they think structure is confining and limiting. It’s not, it’s liberating.

We started to figure out that if we would give the agent smaller focused tasks within a clear process and definitions of success, it would stay on track and deliver a bit better results.

Not perfect but better! Definitely making progress.

Phase 4 - Process

The key to the “Structure” component was the process. We needed to define for the agent a clear step-by-step process to follow. This doesn’t just include actions but also intent. If the agent is aware of the intent and final result, it’s better able to make decisions about the required steps to keep moving towards the goal and not take a hard left into hallucination land.

This is the part that made all the difference. We decided to develop a simple framework built around three fundamental questions:

• What’s important to you and your team?
• What does success look like?
• What is the definition of done?

We found that these three simple questions help the agent understand the context of the task and what is required to complete it. Everything else is supporting information, but these questions are essential for making the right decisions.

This was about the time Claude Code was released and we were actually able to leverage the agents directly and not have to use ChatGPT as a middleman with a lot of copy/paste. (I was behind a bit.) This also allowed for the use of slash-commands. Leveraging this functionality, we were able to develop a practical workflow.

The Evolution: From Failure to Success

Here’s a visual representation of the journey through these four phases:

%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
graph TB
    Start([Start: Agentic Programming Journey]) --> Phase1

    %% Phase 1: Prompt Engineering
    Phase1[🔴 Phase 1: Prompt Engineering<br/>Long detailed prompts with examples,<br/>gotchas, do/don't lists]
    Phase1 --> Result1{Result}
    Result1 -->|❌ FAILED| Problem1[Problem: Overcomplicated solutions<br/>Wheelbarrow → Tricycle Big Rig<br/>Too much technical debt]
    Problem1 --> Phase2

    %% Phase 2: Gates/Guardrails
    Phase2[🔴 Phase 2: Gates/Guardrails<br/>Build process with guardrails<br/>to ensure correct execution]
    Phase2 --> Result2{Result}
    Result2 -->|❌ FAILED| Problem2[Problem: Agent circumvented guardrails<br/>Spent time fighting git-hooks<br/>instead of doing work]
    Problem2 --> Phase3

    %% Phase 3: Structure
    Phase3[🟡 Phase 3: Structure<br/>Smaller focused tasks within<br/>clear process & success definitions]
    Phase3 --> Result3{Result}
    Result3 -->|⚠️ BETTER| Problem3[Better but not perfect<br/>Making progress!]
    Problem3 --> Phase4

    %% Phase 4: Process - The Solution
    Phase4[🟢 Phase 4: Process<br/>Three Key Questions]
    Phase4 --> Questions{What's important?<br/>What does success look like?<br/>What is definition of done?}

    Questions --> Workflow[Three-Step Workflow]

    %% Step 1: Draft PRP
    Workflow --> Step1[Step 1: /draft-prp command]
    Step1 --> Draft[Create detailed draft document<br/>20-150k tokens]
    Draft --> Sections[Sections:<br/>• Constraints<br/>• Requirements<br/>• Definitions<br/>• Success Criteria<br/>• Definition of Done<br/>• Stories/Tasks<br/>• Acceptance Criteria<br/>• Test Cases]
    Sections --> Review{Review &<br/>Validate Draft}
    Review -->|Needs Changes| Step1
    Review -->|✅ Approved| Step2

    %% Step 2: Generate PR
    Step2[Step 2: /generate-prp command]
    Step2 --> Breakdown[Break large document into<br/>smaller discrete PRs]
    Breakdown --> Step3[Step 3: /execute-prp command]
    Step3 --> Execute[Execute individual PRs<br/>with quality gates]
    Execute --> Success[✅ SUCCESS: Production-Ready Code]

    %% Styling
    classDef failedPhase fill:#ffcccc,stroke:#cc0000,stroke-width:3px
    classDef betterPhase fill:#ffffcc,stroke:#ccaa00,stroke-width:3px
    classDef successPhase fill:#ccffcc,stroke:#00cc00,stroke-width:3px
    classDef processBox fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    classDef doneBox fill:#d4edda,stroke:#28a745,stroke-width:3px

    class Phase1,Phase2,Result1,Result2,Problem1,Problem2 failedPhase
    class Phase3,Result3,Problem3 betterPhase
    class Phase4,Questions,Workflow successPhase
    class Step1,Draft,Sections,Review,Step2,Breakdown,Step3,Execute processBox
    class Success doneBox

The Three-Step Workflow That Actually Works

Once we understood that process was the key, we built a workflow that embodies those three fundamental questions. Here’s how it works in practice:

Step 1: Create the Draft PRP

/draft-prp Create a webpage with a blue background and white text that says "Hello World"

Our /draft-prp command is a custom prompt that includes the requirements for the task. It lays out the process, definitions, and context needed. This creates a draft Product Requirements Proposal (PRP) that is placed in the ./prp/drafts directory for review.

This document will contain all the sections and requirements you’ve defined such as:

• Constraints
• Requirements
• Definitions
• Success Criteria
• Definition of Done
• Stories/Tasks
• Acceptance Criteria
• Test Cases

Yes, all of that! The more structure the better. This will create a large document but it’s worth it (20-150k tokens). Review the document and make sure it meets your needs. If not, update the doc or work with the agents to adjust. This is your starting point so make sure it’s complete and correct.

Step 2: Generate Individual PRs

/generate-prp <draft-filename>

This command takes that detailed draft and breaks it down into smaller, more manageable pieces. If you noticed, the document that was created is very detailed, includes a lot of information and is a bit overwhelming to implement as one monolithic task. This step creates individual PRs for each discrete unit of work.

Out of that 100k token file we will break it into individual PRs that are easier to manage, review and implement. Remember, the point is to accomplish the work and keep the agent focused, so smaller discrete tasks are better than large complex ones where the agent will get lost.

I’ve configured my system to organize them using this pattern:

###-[a-z]-###-task-title.md
[PRP #]-[Major Task ID]-[Sequence Number]-[Task Title].md

So now we have a directory of PRs that are ready to be worked on:

./prp/active/333-a-001-create-index-html.md
./prp/active/333-a-002-create-styles-css.md
./prp/active/333-a-003-create-tests.md
etc...

You need to figure out your system and process that works for you, but this is a good starting point.

Step 3: Execute the PR

/execute-prp <pr-filename>

This is where the magic happens. The agent will read the PR, understand the requirements and make the changes to your codebase. It will also run any tests, linters or actions you have configured to make sure the code is correct.

If the PR is completed successfully it will move the PR to the ./prp/completed directory and commit and push the change. If there are any issues it will move the PR to the ./prp/failed directory and leave the PR open for review.

Now let’s dive into what’s actually happening behind the scenes during execution.

Behind the Scenes: What Actually Happens During /execute-prp

When you run /execute-prp, you’re not just executing a script—you’re launching a sophisticated multi-agent orchestration system that coordinates 12+ specialized AI agents to deliver production-ready code. Here’s what happens behind the scenes.

The 12-Phase Pipeline

The execution follows a carefully choreographed sequence where each agent specializes in one domain and hands off to the next:

Phase 1-4: Planning & Design

• Project Manager coordinates the entire pipeline and tracks progress
• Architect Reviewer designs the system architecture and validates design patterns
• Security Reviewer performs threat modeling before any code is written
• Compliance Officer ensures regulatory and standards compliance (NIST, WCAG, etc.)

Phase 5-9: Implementation & Validation

• Developer Agents (language-specific) implement features using strict TDD
• Test Automation validates test quality and coverage
• Performance Profiler benchmarks and optimizes critical paths
• API Designer validates API contracts and endpoints
• Database Administrator handles all data layer concerns

Phase 10-12: Documentation & Deployment

• Documentation Writer creates README, API docs, ADRs, and deployment guides
• Deployment Manager configures infrastructure and deployment pipelines
• Code Management creates the pull request with comprehensive description

Each agent is a specialist. The Python developer doesn’t touch security. The security reviewer doesn’t write database schemas. This separation ensures expertise at every step.

The TDD Engine: RED-GREEN-REFACTOR

Here’s where things get interesting. Every developer agent is enforced to follow Test-Driven Development:

1. RED: Write a failing test first (proves the feature doesn’t exist yet)
2. GREEN: Write minimal code to make the test pass
3. REFACTOR: Clean up the code while keeping tests green

The system blocks any commit that doesn’t follow this pattern. You can’t skip straight to implementation. This ensures:

• Every feature has tests from day one
• Tests actually validate behavior (not just passing by accident)
• Code remains maintainable through refactoring

This is real TDD, not “tests eventually” or “we’ll add tests later” development.

Quality Gates That Actually Enforce Quality

Between each phase, automated gates validate the work. These aren’t suggestions—they’re enforced requirements:

Test Coverage Gate

• Requires 100% coverage (lines, branches, functions, statements)
• No “we’ll add tests later”—the code won’t merge without them
• Enforced by automated tooling, not trust

Mutation Testing Gate

• Validates that tests actually catch bugs (not just execute code)
• Requires ≥95% mutation score
• Prevents weak tests that just achieve coverage numbers without actually validating behavior

For those unfamiliar with mutation testing: the system deliberately introduces bugs into your code (changing + to -, inverting boolean conditions, etc.) and ensures your tests catch them. If your tests still pass when the code is broken, they’re not really testing anything useful. Mutation testing catches this.

Production-Ready Gate

• Zero stubs allowed: No pass, no NotImplementedError, no “TODO: implement this”
• Complete error handling: Every external call wrapped in try/catch with meaningful errors
• Comprehensive logging: Entry/exit logging, error logging, state changes tracked
• No unresolved TODOs: Either implement it or link to a future issue

Security Gate

• No hardcoded secrets or credentials
• Encryption requirements validated
• Auth/authorization patterns verified
• Vulnerability scan passes

These gates aren’t optional. They’re enforced by the system and you can’t merge without passing them all.

What You Get at the End

After all 12 phases complete, you receive:

1. Production-Ready Code

• 100% test coverage (not aspirational—enforced)
• Zero stub implementations
• Complete error handling and logging
• ≥95% mutation score
• Security-validated
• Performance-benchmarked

2. Comprehensive Documentation

• README with installation, usage, and examples
• API documentation (if applicable)
• Architecture Decision Records (ADRs)
• Deployment guides with infrastructure setup
• Security documentation

3. Quality Reports

• Coverage report (coverage/index.html)
• Mutation testing results (mutation/results.html)
• Performance benchmarks
• Security scan results
• Accessibility audit (if UI-related)

4. Pull Request

• Comprehensive PR description with context
• All changes explained with rationale
• Test results embedded
• Ready for human review

5. Complete Audit Trail

• Every agent action logged
• Every decision documented
• Every test result recorded
• Full traceability from requirement to implementation

The Human Still Owns the Decision

Here’s the crucial part: /execute-prp creates a pull request, not a direct commit. The system delivers production-ready code, but you still review and merge. This keeps humans in control while automating the grunt work.

The PR description includes:

• What was built and why
• Design decisions made
• Test coverage summary
• Performance characteristics
• Security considerations
• Breaking changes (if any)

You review it like any other PR, except it’s already passed 6+ quality gates and has 100% test coverage.

Why This Matters

Traditional development: Developer writes code → manually writes tests → hopes they caught everything → ships and prays.

Agentic development: PRP defines requirements → 12 agents orchestrate delivery → every quality gate passes → human reviews and merges → ship with confidence.

The difference? You can’t skip quality. The gates enforce it. You can’t ship incomplete code. The production-ready validator blocks it. You can’t merge without tests. The coverage gate prevents it.

It’s not about AI replacing developers—it’s about AI handling the tedious parts (tests, docs, quality checks) so developers can focus on architecture, design, and strategic decisions.

Next time you run /execute-prp, you’ll know there are 12 agents working in parallel, following strict TDD, enforcing quality gates, and delivering code that’s actually production-ready—not “hope it works” ready.

Lessons Learned: From Four Phases to One Framework

Looking back at this journey, the pattern becomes clear:

Phase 1 (Prompt Engineering) taught us that more words don’t equal better results. The agent doesn’t need a novel—it needs structure.

Phase 2 (Gates/Guardrails) showed us that fighting the agent is counterproductive. Guardrails without context create adversarial relationships where the agent spends energy circumventing controls instead of doing work.

Phase 3 (Structure) was the breakthrough that smaller, focused tasks within clear processes work better. But we were still missing something.

Phase 4 (Process) brought it all together. The three questions—What’s important? What does success look like? What is the definition of done?—combined with the three-step workflow (draft, generate, execute) created a framework that actually delivers.

The key insight? Agents need the same things humans need to do good work: clear objectives, defined success criteria, and a process to follow. The difference is that with agents, we can enforce quality gates that humans might be tempted to skip under pressure.

This process has worked well for us and we are able to get a lot of work done effectively. The agents stay focused and deliver quality code that meets our requirements because the process keeps them on track.

We’re still learning and iterating, but this framework has proven solid. If you’re struggling with agentic programming, consider whether you’re giving your agents structure and process, or just hoping better prompts will fix everything.

The technology I dreamed about in the 90s is finally here. Now we know how to use it effectively.

This post lays out the “why”: the technical leverage, the design space, and the practical upside of shipping a modern MUD today.

TL;DR

• Low cost to build, host, and scale—perfect for iterative development.
• Forces great systems design: protocols, state, persistence, and concurrency.
• Ideal sandbox for applied AI: agents, content tooling, and testing.
• Accessible UX: works everywhere, inclusive for screen readers, low bandwidth.
• Recaptures the fun of tight feedback loops that many modern games have lost.

What I’m Building

I’m working on a networked MUD in Go with a modular architecture: dedicated services for auth, world simulation, NPCs/agents, economy, and observability. It’s engineered for clarity and testability, not just nostalgia.

• Repo (private for now): ~/Repositories/mud
• Language: Go
• Shape: service-oriented core with focused modules (auth, energy, NPCs, network, advertising)
• Tests: high coverage with per-module coverage reports and profiling artifacts
• Deployment: Kubernetes; local dev + CI; S3 + CloudFront for web docs

This series will cover the design and evolution of the system—starting with the “why,” then the network and world simulation, then tools and content.

Why a MUD Still Makes Sense

1) Systems Thinking with Instant Feedback

MUDs require you to design clear interfaces between networking, world state, persistence, and AI. There’s nowhere to hide: strong contracts and repeatable tests win. Small mistakes are visible instantly—and fixable just as fast.

What you learn transfers cleanly to backend engineering, multiplayer infrastructure, and agent systems.

2) Agentic AI Is Better in Text-First Worlds

Text-first worlds are a natural habitat for AI agents:

• Observations are structured text; actions are tokens; no rendering pipeline required.
• You can mix GOAP/BT planners with LLM tooling for dynamic behaviors.
• Content generation is safer and cheaper: descriptions, quests, items, chatter.
• Simulation speed is limited by CPU and your update loop, not frame rate.

The result: richer NPCs, faster iteration, and better telemetry for evaluation.

3) Accessibility and Reach

MUDs run anywhere—terminal, web, mobile SSH client (I’m not enabling telnet for security)—and are friendly to screen readers. Latency tolerance is high, bandwidth use is tiny, and players can drop in for 5 minutes without a 10 GB patch. That’s a superpower.

4) Operates Well on a Small Budget

The stack is lean and predictable. A single modest server can host thousands of concurrent connections when the code is efficient. It’s the right canvas for one person (or a small team) to ship something real.
I’m known for overengineering things, so this is running on a Kubernetes cluster with network policies, lots of observability, and workflows—but it could easily be pared down to a single VM or even a serverless function.

5) Teaches the Hard Parts—Safely

You’ll touch concurrency, protocols, persistence, migrations, observability, and liveops. You’ll also build tooling: map editors, item pipelines, NPC behavior fixtures. All of this pays dividends on any backend system.

Architecture Notes

I’m leveraging agents and LLMs for development, but I believe the future is in learning to build with these systems using a systematic methodology. This allows for stretch goals and learning that would have been otherwise unachievable. This project is also about what makes architecture and design so important, interesting, reliable, and predictable.

Key principles I’m following:

• Contract-first: define message formats and service boundaries up front.
• Deterministic core: world simulation tick with clear ordering and isolation.
• Observability: logs, metrics, and traces around network, AI, and economy.
• Test harnesses: soak tests for performance; fuzz tests for protocol edges.
• Zero-trust mindset: auth, rate limiting, and resource isolation per subsystem.

Planned posts will include:

1. Network protocol and session lifecycle
2. World tick, ECS-like entities, and state persistence
3. NPCs and agent behaviors (rule-based + LLM-assisted)
4. Economy design and anti-exploit checks
5. Tooling, profiling, and deploys

What “Success” Looks Like

• A stable, well-documented server with clean contracts.
• A small, active player base enjoying tight loop gameplay.
• A flexible AI layer that’s fun to extend and evaluate.
• Reusable components applicable to other multiplayer/back-end work.
• A series of posts sharing what I learned along the way.
• API access for bots and extensions. (ReadOnly)

Call for collaborators - Maybe?!?

Interested in systems, networking, or AI behaviors? I’m sure I would love discussing ideas and approaches. I don’t plan on opening the repo as it is a personal project, but if you want to help, look around or kick the tires when it’s ready, ping me or just stay tuned!

– Cheers

Hardware

Really same as before - I haven’t had the need to upgrade until recently when I’ve started actual development on projects.

• Ubiquiti Router
• Cisco Switches
• pfSense (BGP)
• Lenovo Tiny Desktops
  • Dedicated to Kubernetes Master Nodes, freeing me from relying on my Pi Clusters.
  • One system runs the automation systems via HomeAssistant.
• Raspberry Pi CM4 Cluster (6 nodes, 8GB RAM each, NVMe storage)
• DeskPi Super6C Raspberry Pi CM4 Cluster Board
• Raspberry Pi 4 - Solely handling DNS duties.
• Synology NAS

Differences

• Networking
  • BGP handles the Kubernetes routing
  • pfSense is providing the BGP services
  • Default gateway routes the designated networks to the pfSense gateway address. These load balancer IPs are dynamically provisioned by the pfSense.
• Storage
  • The Raspberry Pi are all sporting NVMe for stateful-set storage
  • Half the Raspberry Pi have internal eMMC storage
  • Half are using SD Cards (Don’t do this)

Software

• OS: Talos Linux
• Networking: Cilium
• Storage: local-path-provider
• Network Monitoring: Hubble-UI
• Monitoring: Prometheus
• Reporting: Grafana
• Secrets: Vault

A few YouTube videos, bit of reading about Talos and I knew I wanted to learn more. It literally had everything I wanted:

• API access
• Required config management (if you’re doing it right, please don’t just apply random patches, use the pipeline)
• Security by default

If I was going to learn how to leverage kubernetes I may as well learn it right!

So I wiped the functional Debian kub cluster and started from scratch… It was more difficult than I thought and I brought a lot of the pain upon myself. A few issues getting the Pi configured properly, a desire to PXE boot to realize my setup wouldn’t work as configured. Hardware failures, changing jobs, moving homes, get everything setup again, buy new SD card (128GB), they are all bad… Flash what I have and start building the cluster.

Mistakes

• SD Cards too small
  • I needed to use what I had and the Talos OS is pretty small. I’m using NVMe for the persistent storage so I’m good! WRONG! I forgot about ephemeral storage! The pods that don’t have stateful-sets still need storage to operate and this goes in the ephemeral storage. Since I needed to dedicate the entire drive to the local-path-provisioner it only had what was left on the SD card for ephemeral storage. This meant my pods are beating the hell out of the SD cards on half the worker nodes. This is less than ideal and led to an idea for expansion…
• Didn’t plan out the networking
  • I was trying to get everything working and assigned an entire /24 subnet to the loadbalancer and it worked! CHEERS! I went to bed. Later I came back and realized that wasn’t what I wanted. I had to go back and plan out the environments like I wanted them and touch it a second time. Now they are configured and everything is working as planned. More of a personal ‘damn it’ than a mistake but I’m learning and should have put some thought into it beforehand.
  • Why do I want to separate my networking environments? It’s a lab that I’m trying to simulate an enterprise environment with while having minimal hardware. Network separation is simple and effective if you have proper network controls. I would like to eventually migrate this to leverage VLANs but I’ve got mixed results when testing with the Ubiquiti and MY Cisco gear. It’s on the list but not a priority until the expansion plans.
• Hardware
  • The Pi CM4 are handling my current load without an issue - I’m actually quite impressed! My issue is the networking. It’s a hardware limitation that each of the CM4 connect to a 1GB switch which has one 1GB external port. My control nodes are x64 systems and connected to 1GB ports each. While everything can communicate the network is… less than reliable right now.
  • Control Nodes are older systems and it certainly shows. They need to be replaced and I would like to expand the cluster with additional Pi nodes to do that with. They are efficient and do the job.
• Configuration - Jeez, here are the big ones…
  • I’m learning as I go so first thing I did is change the cluster name. While this can be overcome and may be a normal practice, it’s not something you should do when learning.
  • I tried to get MetalLB working on the talos nodes. It worked on the Debian cluster and I really didn’t get the security setup of Talos. MetalLB requires more access than I was comfortable trying to configure, but it started me down the BGP path. So, it all worked out.
  • Try and initialize, unlock, join-nodes and configure auto-unlock in a single step. Why?!? Well like I said I wanted this to be a production grade, security first cluster so I don’t want Vault to be locked if the cluster or pod goes down. I certainly want it in Production mode, using Raft and clustered. Like I said, secure and production grade. I currently have everything working from bootstrap to unlocked and ready for use, but I haven’t revisited the auto-unlock yet.

Things to Finish

They are actually running just not configured. They are also going to be collecting the stats from the HomeAssistant installation.

• Prometheus
• Grafana
• HomeAssistant Integration - I want to trigger events or start pods/apps based on external inputs. These plans are still being developed and have not yet been researched.
• BDR - Not if, When! I have this setup and now it needs to protect it, and the workloads. I want to do this two ways:
  • Local: Backup to NFS, both the Kubernetes components (etcd, configs, secrets, etc.) and the applications. While snapshots are great in general they aren’t in this environment, so I’m going to require other solutions.
  • Remote: I’m going to leverage StorJ for the remote option for backup. Why? Well it’s a lot less expensive than S3, client side encrypted, interesting project and I don’t need compliance in my lab.
• DHCP
  • I’m currently using DHCP off my Ubiquiti router and it’s not very… feature rich currently when it comes to these features. I’m already using BIND for DNS and would like to migrate to Kea for DHCP. This will allow me to start trying to leverage PXE which will bring a new new game to the lab!
• PXE
  • Currently the systems are booting from an SD card located underneath the cluster board. If it’s mounted they can’t be accessed. If the SD card accidentally gets wiped from say forgetting to specify the partition. You need to bring the entire cluster down, disassemble, remove and flash the SD, reverse. Less than ideal for a lab, when this is configured it removes that dependency and now the SD card can just be ephemeral storage.

Things I would do differently

This is a loaded idea because there’s not much I would really keep the same.

• I would do a lot more planning now that I know what to look for in a general sense. That would reduce the number of mistakes, changes, confusing configurations and general cruft of the system. I am going to cut myself a little slack since this is the first cluster leveraging Talos and learning the general workings. There’s a lot more but I learn by doing so I’m definitely going to break something.
• Different Hardware: I would NOT use the cluster board for what I’m trying to do. While it’s completely functional for learning and small projects, for my use case it’s not appropriate. I would replace the cluster board with individual PoE Pi boards or blades. Boot from PXE (preferred) or SD card, eMMC for ephemeral storage and NVMe for stateful-sets.

Current Uses

I’m writing cloud first apps leveraging Kubernetes and just like the cluster they are a continuous work in progress but a great learning platform. How to leverage different components, proper secret management, pipeline development and checks, etc.

The Stack

Database: MongoDB stateful-set

Caching: Redis

Frontend: Streamlit and FastAPI/Kong

Monitoring: Prometheus

Reporting: Grafana

Security: Vault

Pipeline: GitHub Actions

Runners: Local

Current Development

Health & Safety App

Agent Analysis App

Agent Development Company (Ambitious but it’s fun and am learning lots)

Immediate Plans

I’m going to need to expand this soon and this is one of the things I think is great about Talos. I’m leveraging KubeSpan and am going to configure autoscaling into GCP (price - it’s always price). This creates a WireGuard VPN across the nodes of the cluster encrypting all traffic in-flight. This allows me to extend the cluster into multiple cloud providers as needed leveraging the least expensive resource possible, dynamically. This also allows me to seamlessly protect against a major single cloud provider failure. Leveraging Terraform I’ll be able to maintain standards and change control across multiple providers. This is an awesome win!

Less Immediate Plans

Significant upgrades to the entire “datacenter”, as financing permits:

• 10G redundant networking (Storage and Master Nodes)
• ProxMox Nodes for Ceph storage, Talos Master Node VMs & x64 workloads/pods
• Raspberry Pi Blades or other compact units with POE and storage options (Primary worker nodes)
• PXE Boot all nodes (ARM & X64)
• HomeAssistant Integrations

Anyway I think that’s enough rambling, but that’s the status of the lab and why it is in its current state. I’m looking forward to working on these projects and hope to transition into working this type of stack full-time. I’m excited about the possibilities, where it can lead and I’m certainly not short on ideas of how to leverage this in so many settings.

Thanks for sticking around Cheers

• Ubiquiti RouterAmazon Link
• Cisco Switches - Cisco (Because you can never go wrong with reliability.)
• Lenovo Tiny Desktops
  • Dedicated to Kubernetes Master Nodes, freeing me from relying on my Pi Clusters.
  • One system runs the automation systems via HomeAssistant.
• Raspberry Pi CM4 Cluster (6 nodes, 8GB RAM each, NVMe storage)
• DeskPi Super6C Raspberry Pi CM4 Cluster Board
• Raspberry Pi 4 - Solely handling DNS duties.
• Synology NAS - It’s currently hanging around but soon to be retired. All the systems are statically assigned IPv4 addresses, and DHCP/DNS records are maintained using Ansible (Stay tuned—I’ll be releasing the role soon!). Basic configurations—like NTP settings, package installations, updates, and user setups—are also automated with Ansible.

Here’s how I typically bring a new system online:

1. Flash the system image (using PI Imager or a custom Debian image with a predefined hostname and user account).
2. Boot up and assign a static IP via the router.
3. Add the system to the /etc/hosts file on my workstation.
4. Update the Ansible inventory.yml to reflect the new system’s role—DNS names only, no IP addresses.
5. Use ssh-copy-id to transfer SSH keys to the new system (FQDN all the way).
6. Verify SSH is set up and functioning properly.
7. Run the necessary Ansible playbooks:
   • Master Playbook
   • DNS Playbook

By this point, the new system should be ready, configured to the baseline, and potentially even further if its role calls for it (e.g., becoming a Kubernetes worker node). Most of my systems are headless, and I rarely need to interact with them directly. They’re here for testing and workload purposes, so unless I need to tweak a configuration or investigate something, they’re largely hands-off. These servers run themselves, allowing me to focus on the more interesting stuff. Cheers!

image: path: /assets/img/blog/Ansible_Wireguard_blog.jpg srcset: 1920w: /assets/img/blog/Ansible_Wireguard_blog@0,5x.jpg 960w: /assets/img/blog/Ansible_Wireguard_blog@0,25x.jpg 480w: /assets/img/blog/Ansible_Wireguard_blog@0,125x.jpg

Create WireGuard VPN with Ansible

Currently we are using Cisco AnyConnect for our VPN solution. It works well, but it is a bit of a pain to install and configure. We are retiring the environment that the ASA is located in and it’s time to retire this piece of equipment since it doesn’t fit into our current infrastructure or future plans.

We are a fully remote company and as a result all of our resources and assets are also in the cloud. That said we would like to control access to our resources and have some ability to quickly revoke access if needed. Since we may have multiple endpoints or environments we would need to connect to we would like it to be simple and flexible. Additionally we would like to reduce expenses as much as possible so open-source is certainly preferred.

That said the simple and honest choices were between OpenVPN and WireGuard. Discussions with the technical resources and managers quickly settled on WireGuard. WireGuard is simple, lightweight, easy to configure and troubleshoot and has wide acceptance in the market. We can have multiple configurations on a single system without conflict. It’s simple enough to deploy and manage that if we needed to have multiple environments it’s pretty straight forward.

But we can always do better!

Scenario

Here is the basic scenario:

• We are a small firm and we have a small presence in a co-located datacenter and we need to give our employees access to the datacenter resources. Management does not want us to use the resources provided by the datacenter to terminate the VPN due to cost, however we are able to spin up as many virtual machines as we want without incurring any additional cost.

Seems simple enough and this is what we’re going to work through. We have a bunch of remote users that need to connect to the datacenter and we don’t have any real central method of authentication here so using files is acceptable to this company’s risk profile.

WHY VPN?

So it’s the end of 2023 and I’m about to explain why a VPN is important. People don’t understand the costs or risks of being spied on, either in a corporate sense or personal sense either. This is worth its own post which I’ll link here when I get to it :)

• Corporate: We want to protect our customer’s data and prevent internet snoops¹ from being able to read or prioritize our private data.
• Personal: We don’t want to be a larger target for advertising, data theft, man-in-the-middle attacks, etc. Using a VPN reduces the risks of a lot of these, even at home. Using a VPN will keep your ISP from being able to tell where you like to shop, what you like to watch, what you’re searching. Additionally, in some people’s lives it’s a matter of personal security.
• Government: I live in the United States where currently (Dec, 2023) we are not exceptionally concerned with government persecuting us for our political views however, if you’re in a country where this is a problem a VPN may be a life-line.

Planning

Generally when developing a playbook you’re going to complete a series of steps that would normally be completed on the commandline, (Linux assumptions here) so I like to run through those steps to build out my playbook(s) or role(s).

1. Configure Server

   • Install Software (WireGuard package)

   • Create server config file

   • Create WireGuard service

   • Configure service to start on boot

2. Create Client Configurations

   • Create list of users

   • Create client configuration

3. TEST

   • TEST

   • TEST

   • TEST

4. Start playbook development

As you can tell we are just running through the process right now to make sure everything works as planned. We aren’t building the playbook or writing any code just doing the job. Take copious amounts of notes while doing this and make sure you’ve gotten every step. That is going to be the template you write the playbook from.²

Doing

Create WireGuard Server

1. Install WireGuard and firewall

   sudo apt install wireguard ufw -y
   

2. Create your WireGuard keys

   Create the private key with wg genkey which will output the private key. This private key is the input for the next step which derives the public key. echo <key> | wg pubkey

   wg genkey
   GO/b82NFwTXApR1CzO2MHtwYg0qWSpGgGgO//GgL5Xo=  # Create and use your own keys!
   
   echo GO/b82NFwTXApR1CzO2MHtwYg0qWSpGgGgO//GgL5Xo= | wg pubkey
   6Gj/JTnJjfFnqnAIA8l6pr718rfEGYK94G9RttzTUwE=
   

   These are the private key and public key you will need for the server. The clients get the public key while the server retains the private key in the configuration file. These values will eventually be stored in the ansible vault so we can start creating that yml file now.

3. (Side Quest) Create vault.yml file

   server:
     private_key: GO/b82NFwTXApR1CzO2MHtwYg0qWSpGgGgO//GgL5Xo=
     public_key: 6Gj/JTnJjfFnqnAIA8l6pr718rfEGYK94G9RttzTUwE=
   

4. Create the server config file /etc/wireguard/wg0.conf

   [Interface]
   Address = 172.20.100 .1/23 # Private Address IPv4
   Address = fd0d:27ad:deda::1/64 # Private IPv6
   SaveConfig = false
   PostUp = ufw route allow in on wg0 out on eth0
   PostUp = iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
   PostUp = ip6tables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
   PreDown = ufw route delete allow in on wg0 out on eth0
   PreDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
   PreDown = ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
   ListenPort = 51820
   PrivateKey = GO/b82NFwTXApR1CzO2MHtwYg0qWSpGgGgO//GgL5Xo=
   

5. Update server networking so it can route and forward packets

   sudo echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf
   sudo echo net.ipv6.conf.all.forwarding=1 >> /etc/sysctl.conf
   sudo sysctl -p
   

6. Update firewall so it can accept VPN connections and SSH. This block allows the correct traffic through, resets the firewall service then prints out the current status.

   sudo ufw allow 51820/udp
   sudo ufw allow OpenSSH
   sudo ufw disable
   sudo ufw enable
   sudo ufw status
   

7. Start and enable the WireGuard service

   sudo systemctl enable wg-quick@wg0.service
   sudo systemctl start wg-quick@wg0.service
   sleep 10
   sudo systemctl status wg-quick@wg0.service
   

At this point you should get a lot of output but it should be green and show a status of ‘Running’

YAY! We have a server running!

But it can’t accept any connections… And we have no clients…

Creating Clients

Part of what makes WireGuard so easy to use is one o the thngs that makes it difficlyt to adopt. Each configuratin is different and there is no method to distribute or update cient configurations after deployment. We will distribute these files using features of HashiCorp’s Vault.

Since we are using ansible to configure the server and it already has the keys for the server let’s leverage that to create the clients files.

1. Review the client config anatomomy

   [Interface]
   PrivateKey = <CLIENT PRIVATE KEY>
   Address = <CLIENT STATIC VPN IP ADDRESS>/32
   DNS = 9.9.9.11, 149.112.112.11, 2620:fe::11, 2620:fe::fe:11 # DNS Servers to use when connected
   
   [Peer]
   PublicKEy = <SERVER PUBLIC KEY>
   AllowedIPs = 0.0.0.0/0, ::/0  # IPs to route through VPN
   Endpoint = vpn.domain.com:51820 # VPN server DNS address
   

2. Server Updates for client - We need to add the [Peer] Section for each

   # Entry for each peer on server
   PublicKey = <SERVER PUBLIC KEY>
   AllowedIPs = <CLIENT STATIC VPN IP ADDRESS>/32
   

Looking at this we need four pieces of information to make our configs:

• Client Private Key
• Client Public Key
• Server Public Key
• IP Address for client when connected to VPN

Since all this needs to be retained and can’t change we are going to store this information in ansible vault. The datastructure is simple:

   users:
      ex_user_001:
         key: 6AGIjiJIHoas8Ew0vENLJvslbsZGo6d+rerRPK++lUI=
         pubkey: R08mgG3FAfNOMn/qjSUMWO7L83XV7y5IFpjHRg6CKiY=
         number: 148

      ex_user_002:
         key: MPl1exnEnE2vAfhUoUKy9BZnWAucUpYtk5HAaTJnDUc=
         pubkey: 4q9IpJxBCZ6HRtWbHgogPlDY9G2LvUkmwQNupLmXrlQ=
         number: 149

This gives us all the information we need to create all of our config entries for the clients. It’s tedious to create this for a bunch of users and keep it straight so I created a shell script to create the users. Execute the script to create the users we will copy the relevant entries into your vault file later.

cat /dev/null > new_clients.yml
echo "users:"
for i in {1..150}; do
    KEY=$(wg genkey)
    PUBKEY=$(echo $KEY | wg pubkey)
    echo "  user-$i": >> new_clients.yml
    echo "    key: $KEY" >> new_clients.yml
    echo "    pubkey: $PUBKEY" >> new_clients.yml
    echo "    number: $i" >> new_clients.yml
    echo "" >> new_clients.yml
done

Create Ansible Playbook

Now we have all the steps we need to accomplish to make this work and all the data required.

1. Create MOTD that tells people this is an Ansible Managed system
2. Deploy WireGuard to the server
3. Create Server Configurations
4. Create Client configs that need to be distributed to clients

• Secondary Objective: Configure so that it can configure new server from scratch

  Create the project directory structure

Create the project directory and the required files. We are also creating and populating the pwfile for ansible vault bucause I know you would never store this in a public location or repo if it wasn’t encrypted right.

export PROJECT_DIR=~/wg_project
mkdir -P $PROJECT_DIR/ansible
mkdir -p $PROJECT_DIR/ansible/handlers
mkdir -p $PROJECT_DIR/ansible/templates
mkdir -p $PROJECT_DIR/ansible/vars
touch $PROJECT_DIR/ansible/inventory.yml
touch $PROJECT_DIR/ansible/handlers/main.yml
touch $PROJECT_DIR/ansible/vars/vars.yml
echo "ansible/pwfile.txt" >> .gitignore
openssl rand -base64 32 > $PROJECT_DIR/ansible/pwfile.txt  # Ansible Vault PW File

Build Data & Ansible Files

1. Populate Vault file Populate the values with what you have previously or generate new values. These values will be used by the templates to create the config files. This keeps us from needing to have the key values in an unsecured code repo.

   wireguard_private_key: OPq2MSt6jd48rwz1Rl+xerrNGoD9x5nQwQm3aC15UlM=
   wireguard_public_key: Z1lsQcYzaGSrQhm8I9kDOezr+wWjPqJFS1JvuLgjM1A=
   
   users:
      cool-user:
         key: GALbYcYdUZAdj1Vimr/Dgd9+ig+O675jEas8/58GAUA=
         pubkey: PUNU2maOfqX+Z1gCj4BG7pdVjxw7maDK+U/lHQ1nrFM=
         number: 15
   
      bob:
         key: CO2K6Z5hqCOIth5e6DXDzD9mPXhkNadi99ytPNxHR0g=
         pubkey: sTyfg//FYtNeTyUk78ZT0+16E8DhwoAeiPMWUzHa4WI=
         number: 46
   
      jake:
         key: UF2zByFRLA63qj9aZ/ZGAdxrigWPasnfLjLojnPubnI=
         pubkey: dQACpWMf93+JO3oPJqoIchT6UOW8BLBTNe9g5koNUls=
         number: 47
   
      ishmail:
         key: ECu4XIhX9HZdwkshU/lvGQcaCNt4/OBZO9xXfmUaMF4=
         pubkey: uo8iMNSFrQQ/Ynqp3FtBJuRsXgZwqx3luzU+VxMfwUM=
         number: 48
   
      # Repeat for each user
   

2. Define our inventory file
   • We define the group of one so it can be incorporated into larger workflows later

   all:
      hosts:
         wireguard-server:
            ansible_host: 172.20.200.15
   

3. Define the top matter for our playbook in main.yml.

   ---
   - name: Update and Install WireGuard on Localhost
     hosts: wireguard-server
     become: true
     gather_facts: true
   
     vars_files:
       - vars/vars.yml
       - vars/vault.yml
   

   • This will only be executed on the WireGuard server(s) not clients as limited to the group wireguard-server in the inventory file
   • We are logging as not root server and will become root via sudo
   • We are going to collect inventory of the host(s)

4. Create the MOTD Template - templates/motd.j2

      
      
   ================================================================================
   
   This system is managed and configured via Ansible.  The Ansible playbooks are
   located in the `ansible` directory of the repo.  
   
   Repo: "{{ repo }}"
   Last Execution: "{{ ansible_date_time.year }}-{{ ansible_date_time.month }}-{{ ansible_date_time.day }} {{ ansible_date_time.hour }}:{{ ansible_date_time.minute }} UTC"
   
   
   ================================================================================
      
      
   

   • Update the message as appropriate for your organization
   • Ensure the repo variable is defined - vars/vars.yaml
   • Last Execution fills in the values from Ansible

5. Execute the Ansible template to create the MOTD

      tasks:
         - name: "Update MOTD"
            ansible.builtin.template:
            src: "motd.j2"
            dest: "/etc/motd"
            owner: root
            group: root
            mode: "0644"
   

   • name: Defines the name of the task and allows it to be identified if needed
   • ansible.builtin.template: this is the full reference to the module we are using
   • src: (required) the filename in the templates/ directory to use
   • owner: (required)
   • group: (required)
   • mode: (required) input as string and allow ansible to interpret - hence quotes

6. Install WireGuard and Firewall

      - name: Install WireGuard and Firewall
        ansible.builtin.apt:
          state: present
          pkg:
           - wireguard
           - ufw
          update_cache: true
   

7. Create the template for the server config - templates/wg.conf.j2

         
      [Interface]
      Address = 172.20.100.1/23
      Address = fd0d:27ad:abcd::1/64
      SaveConfig = false
      PostUp = ufw route allow in on wg0 out on eth0
      PostUp = iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
      PostUp = ip6tables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
      PreDown = ufw route delete allow in on wg0 out on eth0
      PreDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
      PreDown = ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
      ListenPort = 51820
      PrivateKey = {{ wireguard_private_key }}
   
   
      {% for user_name, user_data in users.items() %}
   
      [Peer]
      # {{ user_name }}
      PublicKey = {{ user_data.pubkey }}
      Address = 172.20.100.{{ user_data.number }}, fd0d:27ad:abcd::{{ user_data.number }}/128
   
      {% endfor %}
         
   

   • This is a copy of the configuration we had prior with the sensitive and repetitive variables swapped out
   • wireguard_private_key is in the vault from a prior step
   • The data for the users was placed in the vault from a prior step and is looped through creating a [Peer] stanza for each
   • user_name: is the id in the data structure in vault
   • user_data.pubkey: pubkey in the vault
   • user_data.number: defines the IP address of the client since it must be static
8. Create Ansible step to create config from template

      
    - name: Create WireGuard Server Config
      ansible.builtin.template:
        src: "wg0.conf.j2"
        dest: "/etc/wireguard/wg0.conf"
        owner: root
        group: root
        mode: "0600"
      # notify: Restart WireGuard Server
   

   • Using the fine templates/wg0.conf.j2
   • Create the file based off the template in /etc/wireguard/wg0.conf

9. Create the Client Config Directory This needs to be treated reasonably securely - I’m putting it in the directory here as an example but it should really be retained in Vault or someplace similar.

      - name: Create Client config directory
        ansible.builtin.file:
          dest: "/etc/wireguard/clients"
          owner: root
          group: root
          state: directory
          mode: "0600"
   
   

10. Create the template for the client configs - templates/client.conf.j2

          
       # Configuration for {{ item.key }} ONLY
       [Interface]
       PrivateKey = {{ item.value.key }}
       Address = 172.20.100.{{ item.value.number }}/32, fd0d:27ad:abcd::{{ item.value.number }}/128
    
       [Peer]
       PublicKey = {{ wireguard_public_key}}
       AllowedIPs = 0.0.0.0/0, ::/0
       Endpoint = vpn.domain.com:51820
    
    
       
    

    This loops through the times and creates the file for each of them.

11. Create Ansible step to create configs from template

       
       - name: Create Client Configurations
         ansible.builtin.template:
           src: "client.conf.j2"
           dest: "/etc/wireguard/clients/{{ item.key }}.conf"
           owner: root
           group: root
           mode: "0600"
         loop: "{{ lookup('dict', users) }}"
         loop_control:
           label: "{{ item.key }}"
       
    

    • This loops through every item of users
    • Creates an individual file for each item with the contents of the template

12. Setting services to start on boot and starting them

    
       - name: Enable wg-quick@wg0.service
          ansible.builtin.systemd:
          name: wg-quick@wg0.service
          enabled: yes
    
       - name: Start wg-quick@wg0.service
          ansible.builtin.systemd:
          name: wg-quick@wg0.service
          state: started
    

13. Check the services

       
       - name: Check status of wg-quick@wg0.service
          ansible.builtin.command:
          cmd: systemctl status wg-quick@wg0.service
          register: service_status
    
       - name: Show service status
          ansible.builtin.debug:
          msg: "{{ service_status.stdout_lines }}"
       
    

    Honestly - this is a personal preference that I’m getting into the habit of doing as a final check. This will eventually be put into OpenSearch but that’s a topic for another day. ;)

14. Define Handler(s) handlers/main.yml

    The purpose of the handler is to take some action when called. Multiple tasks can call the handler but it will only execute once when all the actions that call that handler are complete. This can be anything that needs to happen - here we are using it to restart the service.

       ---
       - name: Restart_wireguard
          ansible.builtin.systemd:
             name: wg-quick@wg0.service  # Replace wg0 with your WireGuard interface name if different
             state: restarted
             daemon_reload: true
    

15. Register Handler in playbook ${PROJECT_DIR}/main.yml

    In the top matter of the file include the include lines. This tells Ansible where to look for the handlers.

    
       handlers:
          - include: handlers/main.yml
       
    

Execute Ansible Playbook

1. Encrypt the Vault

   cd $PROJECT_DIR/ansible
   ansible-vault encrypt vars/vault.yml --vault-password-file=pwfilt.txt
   

2. Execute Playbook

   ansible-playbook -i inventory.yml playbook.yml --vault-password-file=pwfile.txt
   

3. Output³

   ansible-playbook -i inventory.yml main.yml --vault-password-file=pwfile.txt --limit wireguard-test
      [DEPRECATION WARNING]: "include" is deprecated, use include_tasks/import_tasks instead. See https://docs.ansible.com/ansible-core/2.14/user_guide/playbooks_reuse_includes.html for details. This feature will be 
      removed in version 2.16. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
   
      PLAY [Update and Install WireGuard on Localhost] ******************************************************************************************************************************************************************
      skipping: no hosts matched
   
      PLAY [Configure WireGuard Server] *********************************************************************************************************************************************************************************
   
      TASK [Gathering Facts] ********************************************************************************************************************************************************************************************
      ok: [wireguard-test]
   
      TASK [Update MOTD] ************************************************************************************************************************************************************************************************
      changed: [wireguard-test]
   
      TASK [Install WireGuard and Firewall] *****************************************************************************************************************************************************************************
      ok: [wireguard-test]
   
      TASK [Create WireGuard Server Config] *****************************************************************************************************************************************************************************
      ok: [wireguard-test]
   
      TASK [Create Client config directory] *****************************************************************************************************************************************************************************
      ok: [wireguard-test]
   
      TASK [Create Client Configurations] *******************************************************************************************************************************************************************************
      ok: [wireguard-test] => (item=temp_user_10)
      ok: [wireguard-test] => (item=temp_user_20)
      ok: [wireguard-test] => (item=temp_user_30)
      ok: [wireguard-test] => (item=temp_user_40)
      ok: [wireguard-test] => (item=temp_user_50)
      ok: [wireguard-test] => (item=temp_user_60)
   
      TASK [Enable wg-quick@wg0.service] ********************************************************************************************************************************************************************************
      changed: [wireguard-test]
   
      TASK [Start wg-quick@wg0.service] *********************************************************************************************************************************************************************************
      changed: [wireguard-test]
   
      TASK [Check status of wg-quick@wg0.service] ***********************************************************************************************************************************************************************
      changed: [wireguard-test]
   
      TASK [Show service status] ****************************************************************************************************************************************************************************************
      ok: [wireguard-test] => {
         "msg": [
            "● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0",
            "     Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; preset: enabled)",
            "     Active: active (exited) since Tue 2023-12-26 15:12:18 UTC; 1s ago",
            "       Docs: man:wg-quick(8)",
            "             man:wg(8)",
            "             https://www.wireguard.com/",
            "             https://www.wireguard.com/quickstart/",
            "             https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8",
            "             https://git.zx2c4.com/wireguard-tools/about/src/man/wg.8",
            "    Process: 4897 ExecStart=/usr/bin/wg-quick up wg0 (code=exited, status=0/SUCCESS)",
            "   Main PID: 4897 (code=exited, status=0/SUCCESS)",
            "        CPU: 152ms"
         ]
      }
   
      PLAY RECAP ********************************************************************************************************************************************************************************************************
      wireguard-test             : ok=10   changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
   
   
   

Conclusion

This process should successfully execute and configure your WireGuard server from start to finish. Initially, we manually configured the server to develop the Ansible playbook. While this approach was sufficient to get the server up and running, it lacked the scalability and replicability essential for efficient management. Without Ansible, each new user configuration would require manual setup, a time-consuming and error-prone process. Now, by simply making an entry into the vault and executing the playbook, everything is deployed in a standardized and consistent manner. This automation not only saves time but also ensures uniformity across different environments or machines. Imagine the complexity of explaining the addition of a new user over the phone using the manual method versus the streamlined process we have now established with Ansible. This approach significantly simplifies the management of the VPN server, making it more accessible and manageable.

– Cheers

Ansible for DevOps - If you’re just starting with Ansible this is the book you want. Get it from Leanpub and if there are any updates to the book you will also receive them. (Digital Only)

• Bad Reasons
  • Active Directory Replication - Depending on your convergence time, number of domain controllers and DNS zones. This might be a bad idea! I’m doing this in my home lab with two domain controllers so I expect it’s not going to be a big deal
  • You can turn Pihole off if it’s not working properly. This would require you to delete the zone from DNS and allow it to replicate - See first reason! Additionally you would need to stop the scheduled process if you have it automatically updating on a schedule.
  • It’s exceptionally easy to have Pihole running in a docker container and forward your DNS to the container then to the internet.
• Maybe Reasons
  • Malware domain lists - It’s all about tradeoffs so it may be worth saving a bit of a potential headache especally if you have click happy users.

First we need to setup our lab which is simply going to consist of two AD/DNS servers. DNS will be replicated as AD integrated zones and forwarding to local DNS.

• Install ActiveDirectory Domain Services on both systems

  Install-WindowsFeature -name AD-Domain-Services -IncludeManagementTools
  

• This is a new forest and domain so let’s get it all going. Starting on DC01:

  Install-ADDSForest -DomainName lab.local -DomainMode Win2012 -ForestMode Win2012 -InstallDNS:$true
  

  • It will prompt you for a SafeMode Password so enter your super secret password.
  • Confirm reboot - I hit ‘A’ for Yes to All
• Installation starts and wait for it to complete and reboot. On the second system (DC02) we need to wait for the first to finish then we can add the system to the domain.

  Install-ADDSDomainController -DomainName lab.local -Credential (Get-Credential lab\Administrator) -InstallDNS:$true
  

• I always use the Get-Credential when possible. The scripts and command-line are horrible places for passwords.

Everything is complete so let’s check to make sure DNS is installed and running:

• Simple check of DNS to make sure everything is good and we notice we don’t have any reverse lookup zone for our network 10.10.10.0/24 so let’s create it real quick

• Check and make sure it’s replicated to the second DC

OK! We have our domain setup and configured without any major issues so let’s get our DNS data and see what we can do.

Going for the simple list we are going to be using Stephen Black’s block list. Stephen has an awesome setup that let’s you update the /etc/hosts file on your system to block ads, malware, etc. We are just extending this to AD DNS. His host files are formatted as 0.0.0.0 <domain.name> so the query fails. It’s very effective and I use it on my personal laptop for when I’m not behind my home filters.

We are going to pull the domain names from the second column and use that to create a zone. Within that zone we will have a record doing the exact same as Stephen and point to 0.0.0.0.

• First let’s create some sample data - I’ll just copy the first few lines out of his files on (GitHub)[https://github.com]{:”target”=”_blank”}

May do this again with BIND just because…

At the university, we had students whose status changed frequently throughout the day - they would enroll, drop classes, and so on. We needed to create, enable, or disable accounts and update majors. These changes were made in Oracle software and were not directly connected to AD. The Oracle administrators created a script that would export all the changes since the last run into a CSV file, which I would then use to make the necessary updates.

The next version of the script was supposed to query the database directly, eliminating the need for file swapping, but I left that job before it could be developed.

User Maintenance with PowerShell

Here’s the basic senario, you need to create a static website and get it deployed but don’t know what resources you need, don’t want to deal with servers or a large hosting provider. You want to have control of your assets, be able to add services as needed all with the ability to deploy your site with some level of change managment.

Simple!

Use Amazon Cloud Development Kit!

The AWS CDK allows for Infrastructure as Code for your applications or systems. Simply put it allows you to define your resources the application or in this case website, is going to need so everything is deployed and managed as one cohesive unit. If you need to add a database you define the database resources, security and connections. Need an ec2 server that’s fine too, define the size, type, availability and VPC and you’re off to the races. The same goes for almost any Amazon service, DynamoDB, Lex, S3, EKS, etc… I think you get the point. Everything is defined in code so the same development tools used for management and version control apply here, including CI/CD pipelines.

1. CDK Basics and Installation
2. Defining your project
3. Creating Resources
4. Creating your static site

CDK Basics and Installation

I would alway recommend checking out the docs and information, Amazon does a great job of documenting their services which can be found here –> AWS CDK. I use multiple operating systems so I’ve installed it via the installer (Windows), npm (Linux) and brew (Mac) with some mix depending on issues or requirements. The default language for CDK is TypeScript so if you have Node installed it’s the most forward way to get everything working. Node is used on the backend of CDK so it’s a requirement and can be found here Node Download.

Here’s the currently supported languages from the AWS documentation. I’ll be using TypeScript and Linux in this tutorial.

Now that you have Node and the CDK installed you need to configure the environment the easiest way is with the cli by executing the configure command.

aws configure

CDK leverages AWS CloudFormation and S3 to deploy and manage the enviroment for those to be created and configured execure the bootstrap command.

cdk bootstrap aws://<ACCOUNT-NUMBER>/<DESIRED-REGION>

This will create a bucket similar to this:

And a CloudFormation template like this:

We are now ready to create our project!!!

Creating our Project

Lets create or working directory and initialize our project

mkdir ~/static-website
cd ~/static-website
cdk init app --language typescript

This will create the project and initialize a git repository for you so all changes from here will be tracked. The directory stucture in your project should look like this:

Now that your project is created we need to define some environment variables AWS and deployment. Open the file ./bin/static-website.ts and uncomment and update the line

env: { account: '123456789012', region: 'us-east-1' },

so that it matches your AWS account and region you want to deploy to.

The file will look like this:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { StaticWebsiteStack } from '../lib/static-website-stack';

const app = new cdk.App();
new StaticWebsiteStack(app, 'StaticWebsiteStack', {
  /* If you don't specify 'env', this stack will be environment-agnostic.
   * Account/Region-dependent features and context lookups will not work,
   * but a single synthesized template can be deployed anywhere. */

  /* Uncomment the next line to specialize this stack for the AWS Account
   * and Region that are implied by the current CLI configuration. */
  // env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },

  /* Uncomment the next line if you know exactly what Account and Region you
   * want to deploy the stack to. */
 env: { account: '123456789012', region: 'us-east-1' }, //  <------------Change this line

  /* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */
});

Now we get to add the resources you will need to deploy your site.

You need to have a few things we are going to configure. All resource configuration happens in the ./lib/static-website-stack.ts file.

1. DNS Domain hosted in S3
2. S3 Bucket to host your content
3. Content to host

First let’s create the directory for your web content - depending on what you’re deploying this will be the content we are putting in S3.

mkdir ./web-content

Since we need to have those resources we need to import the associated libraries for them. Add these lines to the top of the file under the Construct:

You can comment out or delete the aws-sqs line since we aren’t using that service.

import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3Deployment from 'aws-cdk-lib/aws-s3-deployment'
import * as route53 from 'aws-cdk-lib/aws-route53'

This imports the constructs, methods and other various parts we need to for these services.

• We need s3 because it’s going to be hosting our content
• s3Deployment gives us the ability to populate or S3 Bucket from local disk or other buckets
• route53 is AWS’ DNS service and we will need to create and update the records there

Let’s first define the subdomain ‘www’, domain and web asset directory:

export class StaticWebsiteStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const recordName = 'www';
    const siteDomainName = 'mydomain.com';
    const webAssetDirectory = 'web-content';

We are going to use those variables later and it’s a good practice to not include static definitions in your code, if possible.

Define our s3 bucket properties:

      const webBucket = new s3.Bucket(this, 'webbucket', {
        bucketName: [recordName, siteDomainName].join('.'),     // Bucket name needs to match the website DNS 
        publicReadAccess: true,                                 // Enables public access for a website
        removalPolicy: cdk.RemovalPolicy.DESTROY,               // When we update it's actually a delete and replace
        autoDeleteObjects: true,                                // Don't ask us for permission just do it 
        websiteIndexDocument: 'index.html'                      // Entry file for the site
      });

Copy our data from the ./web-content directory to S3

      // Copy the web assets to the S3 bucket
      new s3Deployment.BucketDeployment(this, "deployStaticWebsite", {
        sources: [s3Deployment.Source.asset(webAssetDirectory)],
        destinationBucket: webBucket,
      });

Now we have our bucket created at www.mydomain.com and we have assets from the ./web-content directory placed in the bucket. Now we need to redirect traffic from the bucket of mydomain.com so everyone ends up in the same place.

      // redirect the root domain bucket to the www bucket
      const redirectBucket = new s3.Bucket(this, "redirectBucket", {
        bucketName: siteDomainName,                                             // defines the name of the bucket 
        websiteRedirect: { hostName: [recordName, siteDomainName].join('.') }   // Redirects 'mydomain.com'
      });

Update the DNS records as needed

    // Get the DNS base zone to update the record in the next step
    const zone = route53.HostedZone.fromLookup(this, 'baseZone', {
      domainName: siteDomainName
    });

    new route53.ARecord(this, 'AliasRecord', {
      zone,                             // mydomain.com
      recordName,                       // www
      // creates the target record with the bucket information
      target: route53.RecordTarget.fromAlias(new cdk.aws_route53_targets.BucketWebsiteTarget(webBucket)),
    });

Our CDK configuration is complete!

Head back to the command line and execute cdk synth if everything is configured correctly you should get a CloudFormation template with all the changes it will deploy to your environment.

Now we create the web content - to keep it simple we are going to create a simple Hello World page and you can go wild from there!

cd ./web-content
touch index.html

Open the file index.html in your favorite editor and add the following content.

<!DOCTYPE HTML>
<html>
    <head>
        <title>Hello World!</title>
    </head>
    <body>
        <h1>Hello World!</h1>
    </body>
</html>

Back at the command line execute cdk deploy in the root directory of your project. It’s going to take a minute to deploy the resources and will ask for any changes to permissions. Once complete browse to your address at http://www.mydomain.com and you should get your new static Hello World page!

Cheers!

While these units got the job done, they were far from perfect. Not only were they unable to communicate with each other, they also lacked any kind of smart features. But I wasn’t about to let some basic appliances dictate the temperature of my home. With the power of HomeAssistant, I was determined to take back control.

My first step was to determine if the window units would maintain their settings when unplugged. After a few hours, I was relieved to find that they did. Next, I put my trusty smart-plugs to the test, using them to turn the AC units on and off remotely through HomeAssistant.

But simply turning the units on and off wasn’t enough. I wanted to take things to the next level and create a smart thermostat that would allow me to control the temperature across the house, rather than just in the individual rooms. Using a combination of Sonoff and ESP-based temperature sensors, I was able to create a generic thermostat that would keep the house at just the right temperature.

Create the direcories for your configs

mkdir -p config/climate 

edit the configuration.yaml file to contain this line and read the configs from the directory that was just created.

climate: !include_dir_merge_list config/climate/

Next create a file in the directory that was just created - mine is climate.yaml and paste the following code.

- platform: generic_thermostat
  name: Master Bedroom Thermostat
  unique_id: '123456789456123456789'
  heater: switch.master_bedroom_ac
  ac_mode: true
  target_sensor: sensor.master_bedroom_temp
  min_temp: 64
  max_temp: 85
  cold_tolerance: 0.3
  hot_tolerance: 0.3
  precision: 1.0
  min_cycle_duration:
    minutes: 3

Reload your integrations and you should have a thermostat on your dashboard

With the help of a bit of code and some creative problem-solving, I was able to turn my basic window units into fully-functioning smart appliances.

Cheers