I have been running some workloads on Raspberry Pi’s / Leap for some time. I manage them using salt-ssh along with a Pine64 running OpenBSD. You can read more about using Salt this way in my Using Salt like Ansible post.
The workloads ran on containers, which were managed with systemd and podman. Salt managed the systemd service files on /etc/systemd, which start, monitor and stops the containers. For example, the homeassistant.sls state, managed the service file for mosquitto:
homeassistant.mosquito.deploy:
file.managed:
- name: /root/config/eclipse-mosquitto/mosquitto.conf
- source: salt://homeassistant/files/mosquitto/mosquitto.conf
homeassistant.eclipse-mosquitto.container.service:
file.managed:
- name: /etc/systemd/system/eclipse-mosquitto.service
- contents: |
[Unit]
Description=%N Podman Container
After=network.target
[Service]
Type=simple
TimeoutStartSec=5m
ExecStartPre=-/usr/bin/podman rm -f "%N"
ExecStart=/usr/bin/podman run -ti --rm --name="%N" -p 1883:1883 -p 9001:9001 -v /root/config/eclipse-mosquitto:/mosquitto/config -v /etc/localtime:/etc/localtime:ro --net=host docker.io/library/eclipse-mosquitto
ExecReload=-/usr/bin/podman stop "%N"
ExecReload=-/usr/bin/podman rm "%N"
ExecStop=-/usr/bin/podman stop "%N"
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
service.running:
- name: eclipse-mosquitto
- enable: True
- require:
- pkg: homeassistant.podman.pkgs
- file: /etc/systemd/system/eclipse-mosquitto.service
- file: /root/config/eclipse-mosquitto/mosquitto.conf
- watch:
- file: /root/config/eclipse-mosquitto/mosquitto.conf
The Salt state also made sure the right packages and other details where ready before the service was started.
This was very simple and worked well so far. One disadvantage is that the workloads are tied to a particular Pi. I was not going to make the setup more complex by building my own orchestrator.
Another disadvantage is that I was pulling the containers into the SD card. I was not hoping for a long life of these. After it died, I took it as a good opportunity to re-do this setup.
My long term goal would be to netboot the Pi’s, and have the storage mounted. I am not very familiar with all the procedure, so I will go step by step.
I decided for the the initial iteration:
- k3s (Lightweight Kubernetes) on the Pi’s
- The k3s server to use a USB disks/SSDs with btrfs as storage
- The worker nodes to /var/lib/rancher/k3s from USB storage
- Applying the states over almost stock Leap 15.2 images should result in a working cluster
All the above managed with salt-ssh tree on a git repository just like I was used to
k3s installation
We start by creating k3s/init.sls. For the k3s state I defined a minimal pillar defining the server and the shared token:
k3s: token: xxxxxxxxx server: rpi03
The first part of the k3s state ensures cgroups are configured correctly and disables swap:
k3s.boot.cmdline:
file.managed:
- name: /boot/cmdline.txt
- contents: |
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
k3s.disable.swap:
cmd.run:
- name: swapoff -a
- onlyif: swapon --noheadings --show=name,type | grep .
As the goal was to avoid using the SD card, the next state makes sure /var/lib/rancher/k3s is a mount. I have to admit I wasted quite some time getting right the state for the storage mount. Using mount.mounted did not work because it is buggy and took different btrfs subvolume mounts from the same device as the same mount.
k3s.volume.mount:
mount.mounted:
- name: /var/lib/rancher/k3s
- device: /dev/sda1
- mkmnt: True
- fstype: btrfs
- persist: False
- opts: "subvol=/@k3s"
I resorted then to write my own state. I discovered the awesome findmnt command, and my workaround looked like:
k3s.volume.mount:
cmd.run:
- name: mount -t btrfs -o subvol=/@{{ grains['id'] }}-data /dev/sda1 /data
- unless: findmnt --mountpoint /data --noheadings | grep '/dev/sda1[/@k3s]'
- require:
- file: k3s.volume.mntpoint
This turned later to be a pain, as the k3s installer started k3s without caring much if this volume was mounted or not. Then I remembered: systemd does exactly that. It manages mount and dependencies. This simplified the mount state to:
k3s.volume.mount:
file.managed:
- name: /etc/systemd/system/var-lib-rancher-k3s.mount
- contents : |
[Unit]
[Install]
RequiredBy=k3s
RequiredBy=k3s-agent
[Mount]
What=/dev/sda1
Where=/var/lib/rancher/k3s
Options=subvol=/@k3s
Type=btrfs
cmd.run:
- name: systemctl daemon-reload
- onchanges:
- file: k3s.volume.mount
service.running:
- name: var-lib-rancher-k3s.mount
The k3s state works as follows: it runs the installation script in server or agent mode depending if the pillar k3s:server entry matches with the node where the state is applied.
{%- set k3s_server = salt['pillar.get']('k3s:server') -%}
{%- if grains['id'] == k3s_server %}
{%- set k3s_role = 'server' -%}
{%- set k3s_suffix = "" -%}
{%- else %}
{%- set k3s_role = 'agent' -%}
{%- set k3s_suffix = '-agent' -%}
{%- endif %}
k3s.{{ k3s_role }}.install:
cmd.run:
- name: curl -sfL https://get.k3s.io | sh -s -
- env:
- INSTALL_K3S_TYPE: {{ k3s_role }}
{%- if k3s_role == 'agent' %}
- K3S_URL: "https://{{ k3s_server }}:6443"
{%- endif %}
- INSTALL_K3S_SKIP_ENABLE: "true"
- INSTALL_K3S_SKIP_START: "true"
- K3S_TOKEN: {{ salt['pillar.get']('k3s:token', {}) }}
- unless:
# Run install on these failed conditions
# No binary
- ls /usr/local/bin/k3s
# Token changed/missing
- grep '{{ salt['pillar.get']('k3s:token', {}) }}' /etc/systemd/system/k3s{{ k3s_suffix }}.service.env
# Changed/missing server
{%- if k3s_role == 'agent' %}
- grep 'K3S_URL=https://{{ k3s_server }}:6443' /etc/systemd/system/k3s{{ k3s_suffix }}.service.env
{%- endif %}
- require:
- service: k3s.volume.mount
- service: k3s.kubelet.volume.mount
k3s.{{ k3s_role }}.running:
service.running:
- name: k3s{{ k3s_suffix }}
- enable: True
- require:
- cmd: k3s.{{ k3s_role }}.install
Workloads
The next step is to move workloads like homeassistant into this setup.
k3s allows to automatically deploy manifests located in /var/lib/rancher/server/manifests. We can deploy eg. mosquitto like the following:
homeassistant.mosquitto:
file.managed:
- name: /var/lib/rancher/k3s/server/manifests/mosquitto.yml
- source: salt://homeassistant/files/mosquitto.yml
- require:
- k3s.volume.mount
With mosquito.yml being:
---
apiVersion: v1
kind: Namespace
metadata:
name: homeassistant
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mosquitto
namespace: homeassistant
spec:
replicas: 1
selector:
matchLabels:
app: mosquitto
template:
metadata:
labels:
app: mosquitto
spec:
containers:
- name: mosquitto
image: docker.io/library/eclipse-mosquitto
resources:
requests:
memory: "64Mi"
cpu: "100m"
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 1883
imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: mosquitto
namespace: homeassistant
spec:
ports:
- name: mqtt
port: 1883
targetPort: 1883
protocol: TCP
selector:
app: mosquitto
Homeassistant is no different, except that we use a ConfigMap resource to store the configuration and define an Ingress resource to access it from the LAN:
---
apiVersion: v1
kind: Namespace
metadata:
name: homeassistant
---
apiVersion: v1
kind: ConfigMap
metadata:
name: homeassistant-config
namespace: homeassistant
data:
configuration.yaml: |
homeassistant:
auth_providers:
- type: homeassistant
- type: trusted_networks
trusted_networks:
- 192.168.178.0/24
- 10.0.0.0/8
- fd00::/8
allow_bypass_login: true
name: Home
latitude: xx.xxxx
longitude: xx.xxxx
elevation: xxx
unit_system: metric
time_zone: Europe/Berlin
frontend:
config:
http:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: homeassistant
namespace: homeassistant
spec:
replicas: 1
selector:
matchLabels:
app: homeassistant
template:
metadata:
labels:
app: homeassistant
spec:
containers:
- name: homeassistant
image: homeassistant/raspberrypi3-64-homeassistant:stable
volumeMounts:
- name: config-volume-configuration
mountPath: /config/configuration.yaml
subPath: configuration.yaml
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8123
initialDelaySeconds: 30
timeoutSeconds: 30
resources:
requests:
memory: "512Mi"
cpu: "100m"
limits:
memory: "1024Mi"
cpu: "500m"
ports:
- containerPort: 8123
protocol: TCP
imagePullPolicy: Always
volumes:
- name: config-volume-configuration
configMap:
name: homeassistant-config
items:
- key: configuration.yaml
path: configuration.yaml
---
apiVersion: v1
kind: Service
metadata:
name: homeassistant
namespace: homeassistant
spec:
selector:
app: homeassistant
ports:
- port: 8123
targetPort: 8123
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: homeassistant
namespace: homeassistant
annotations:
kubernetes.io/ingress.class: traefik
traefik.frontend.rule.type: PathPrefixStrip
spec:
rules:
- host: homeassistant.int.mydomain.com
http:
paths:
- path: /
backend:
serviceName: homeassistant
servicePort: 8123
Setting up Ingress was the most time consuming part. It took me a while to figure out how it was supposed to work, and customizing the Treafik Helm chart is not intuitive to me. While homeassistant was more straightforward as it is a simple HTTP behind SSL proxy service, the Kubernetes dashboard is already deployed with SSL inside the cluster. I am still figuring out how ingress.kubernetes.io/protocol: https, traefik.ingress.kubernetes.io/pass-tls-cert: "true" (oh, don’t forget the quotes!) or insecureSkipVerify work toghether and what is the best way to expose it to the LAN.
In a future post, I will describe the dashboards setup, and other improvements.