basically the idea is, only worker nodes and the big astridowski (via VPN) can access the control plane and nobody else can

@astrid allow me to introduce you to building management networks, where people do exactly that on the grounds of “security”, and then run both subnets on the same physical network without VLANs.

@littlefox ah yeah I wanna make the control plane nodes properly control plane, no actual workloads running

@littlefox well I'd basically allow cp <-> worker, worker <->internet, cp -> internet, but no internet -> cp is my plan

i wonder how hard I'm gonna lean into the kubernetes stuff. I plan on having edge nodes to do caching and I wonder if I should make them kubernetes too

i do really like a lot of the high availability features k8s provides. the rolling updates are funny

to start out with, I have a single hypervisor that will run 1 control plane and 2 workers. oh no spof yea yea whatever. the second machine will become a hypervisor and then I'll have 3 control planes and 3? 4? workers. and when I'm really advanced I'll have 3 hypervisors each with its own control plane! talk about high availability amirite (they are connected to one power strip

@astrid you could totally do edge nodes as a separate node group, control plane in different subnet should be fine so long as inbound access to api server is allowed. Depending on CNI you may need to allow kube-proxy or whatever through also, some need to access the kube api server via the internal service when bootstrapping

@arichtman i think i might do that (assuming the vps can support k8s and an nginx)

but barring acts of god (oops i ran heater and toaster together) i will have uptime for dayz

@astrid Lol. Why not do vClusters??

@arichtman i like the entropy of just. throwing nodes wherever lol

@astrid fair warning that stretch clusters are highly advised against. Etcd is strongly consistent and I hear even more than a whiff of latency causes big issues.

@arichtman yeah but also, the nodes won't be more than 3-10ms apart

@arichtman they're all being switched together it's yuri

@arichtman oh yeah also the edge nodes would just be in a different zone anyways. i mean worst case they run a single-node k3s/talos cluster

@astrid @arichtman
two nodes, chilling in a cluster, 5ms away because they're not gay

@arichtman actually the biggest availability issue would be the router appliance that all my k8s nodes would be behind. i could carp it tho

@astrid very true. My virtualized router is easily one of the most critical things. Second only to the hypervisor it's on I guess

@arichtman I have several virtualized routers happening

1. inferno/Lucifer, the one hooked to the modem, it's a firewall
2. asmodeus, who will handle dn42 peering
3. charon, who will handle all packets entering my production services
4. nyaanet, the wifi router

Lucifer is opnsense and I don't like opnsense very much after almost a year of using it and I might just move the role into nyaanet.

@astrid whew that's complicated! What ticked you off about opnsense? Seems more libre than pfsense and not trash GUI like openwrt?
Also - if you really wanna have fun, you can use bgo peering for the worker nodes to advertise their ipv6 routes to your main router and skip virtual load balancing.

@arichtman doesn't allow next hop to use link local routes. maybe i should just submit a patch tho. also honestly I wanna just have a really basic front-of-house so I don't kill the Internet with my shenanigans