Troubleshooting CephFS CSI

󰃭 2024-04-07 | #ceph #k8s

A quick writeup for a longstanding issue I’ve had that’s rendered CephFS-backed PersistentVolumes useless on my home K8S cluster. libceph on worker nodes complains about mons speaking the wrong protocol and that wasn’t a red herring, it was the key to the problem all along.

Continue reading 


ArgoCD Image Updater

󰃭 2023-07-15 | #k8s

ArgoCD Image Updater is a tool that works in concert with ArgoCD to update images automatically in your Kubernetes cluster based on a set of rules. Its primary limitation is it requires you use either Helm or Kustomize to deploy your application, as it uses properties of those tools to update the image.

But since Kustomize is so lightweight, it’s actually straightforward to (ab)use it to make ArgoCD Image Updater work for you.

Continue reading 


In-Cluster Prometheus

󰃭 2022-07-27 | #k8s #prometheus

Up until recently, I was running two separate Prometheus instances – one on a Raspberry Pi, and the other in my k3s cluster using kube-prometheus-stack. I wanted to unify them, ideally to simplify management and version control. The challenge here is in how to manage the scrape targets for out-of-cluster resources.

Thanks to my friend Justin, I was able to use a much more elegant solution.

Options

The basic way

When deploying kube-prometheus-stack, define additionalScrapeConfigs. Since I deployed via the Helm chart, that would mean doing a helm upgrade each time I needed to change things. Gross.

Continue reading 


K3S, split-horizon DNS, DNSSEC, and PowerDNS

󰃭 2022-05-30 | #dns #k8s

On my home K3S cluster, I was running into a string of weird DNS issues. Here’s how I ended up fixing it.

Architecture Overview

  • 6-node K3S cluster (3x worker, 3x control plane)
  • In-cluster DNS using standard off-the-shelf CoreDNS
  • Intranet DNS provided by PowerDNS
    • 2 DNS servers
    • Each runs PDNS Authoritative (for internal zone example.com and reverse)
    • Each runs PDNS Recursor (for all other queries; forwards to auth for example.com)
  • Internet DNS for example.com provided by Cloudflare and DNSSEC-signed.
  • Note the split-horizon DNS using the same domain.

Symptoms

I was mostly having issues resolving hosts in example.com. For example, if a pod were making a query to look for foo.example.com, I would see in the PDNS logs queries for:

Continue reading 


ALL the Zeldas (pt. 1)

󰃭 2022-05-18 | #games #zelda

Last week, I completed a longtime personal goal. I can now say I’ve beaten all the mainline Zelda games!

How do I define “mainline Zelda”? More or less, anything in the Hyrule Historia, plus BotW which wasn’t released when the book came out. This doesn’t include the CD-i games, nor any of the spinoff games such as Hyrule Warriors or Link’s Crossbow Training. It also doesn’t include the games that require you to have friends to play…

Continue reading 