Troubleshooting CephFS CSI

2024-04-07 | #ceph #k8s

A quick writeup for a longstanding issue I’ve had that’s rendered CephFS-backed PersistentVolumes useless on my home K8S cluster. libceph on worker nodes complains about mons speaking the wrong protocol and that wasn’t a red herring, it was the key to the problem all along.

Continue reading 


ArgoCD Image Updater

2023-07-15 | #k8s

ArgoCD Image Updater is a tool that works in concert with ArgoCD to update images automatically in your Kubernetes cluster based on a set of rules. Its primary limitation is it requires you use either Helm or Kustomize to deploy your application, as it uses properties of those tools to update the image. But since Kustomize is so lightweight, it’s actually straightforward to (ab)use it to make ArgoCD Image Updater work for you.

Continue reading 


In-Cluster Prometheus

2022-07-27 | #k8s #prometheus

Up until recently, I was running two separate Prometheus instances – one on a Raspberry Pi, and the other in my k3s cluster using kube-prometheus-stack. I wanted to unify them, ideally to simplify management and version control. The challenge here is in how to manage the scrape targets for out-of-cluster resources. Thanks to my friend Justin, I was able to use a much more elegant solution. Options The basic way When deploying kube-prometheus-stack, define additionalScrapeConfigs.

Continue reading 


K3S, split-horizon DNS, DNSSEC, and PowerDNS

2022-05-30 | #dns #k8s

On my home K3S cluster, I was running into a string of weird DNS issues. Here’s how I ended up fixing it. Architecture Overview 6-node K3S cluster (3x worker, 3x control plane) In-cluster DNS using standard off-the-shelf CoreDNS Intranet DNS provided by PowerDNS 2 DNS servers Each runs PDNS Authoritative (for internal zone example.com and reverse) Each runs PDNS Recursor (for all other queries; forwards to auth for example.com) Internet DNS for example.

Continue reading 


ALL the Zeldas (pt. 1)

2022-05-18 | #games #zelda

Last week, I completed a longtime personal goal. I can now say I’ve beaten all the mainline Zelda games! How do I define “mainline Zelda”? More or less, anything in the Hyrule Historia, plus BotW which wasn’t released when the book came out. This doesn’t include the CD-i games, nor any of the spinoff games such as Hyrule Warriors or Link’s Crossbow Training. It also doesn’t include the games that require you to have friends to play…

Continue reading 