The lab from nothing, in one script, on GitHub's runners (K-L1, S7) #31

Merged
jcoffey-dev merged 2 commits from feat/lab-ci into main 2026-11-20 05:49:31 +00:00
8 changed files with 442 additions and 9 deletions

No files matched your search

+63
View File
@@ -0,0 +1,63 @@
# SPDX-FileCopyrightText: 2026 Coffey Labs LLC
# SPDX-License-Identifier: Apache-2.0
#
# The quick checks, on every pull request and on main: every script parses
# (bash -n, Python compile), and every YAML file and workflow loads. The
# end-to-end runs need Docker and kind, so they're on the GitHub replica
# (.github/workflows/e2e.yml).
#
# This file being here also keeps Gitea from running .github/workflows
# itself: it reads those only when .gitea/workflows doesn't exist, and their
# jobs would wait for GitHub's runners for ever.
name: ci
on:
push:
branches: [main]
pull_request:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
check:
runs-on: light
container:
image: mirror.gcr.io/library/debian:trixie-slim@sha256:a72b4433de1cf7d6bae0c4cee74d0547f58601a643b3375f070c51600d63b6b2 # trixie-slim
steps:
- name: Packages
run: |
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq --no-install-recommends ca-certificates git python3-minimal python3-yaml >/dev/null
- uses: coffey-labs/actions/checkout@fab0c4d45e0162963965f1555df27b7bed5e20ec
- name: Scripts parse
run: |
fail=0
while IFS= read -r -d '' f; do bash -n "$f" || { echo "::error file=$f::doesn't parse"; fail=1; }; done \
< <(find lab -name '*.sh' -print0)
while IFS= read -r -d '' f; do python3 -c 'import ast,sys; ast.parse(open(sys.argv[1]).read(), sys.argv[1])' "$f" \
|| { echo "::error file=$f::doesn't parse"; fail=1; }; done < <(find lab -name '*.py' -print0)
exit $fail
- name: YAML loads
run: |
python3 - <<'PY'
import pathlib, sys, yaml
bad = 0
for p in sorted(pathlib.Path(".").rglob("*.y*ml")):
if ".git" in p.parts:
continue
try:
list(yaml.safe_load_all(p.read_text()))
except yaml.YAMLError as e:
print(f"::error file={p}::{e}".replace("\n", " ")); bad = 1
sys.exit(bad)
PY
+60
View File
@@ -0,0 +1,60 @@
# SPDX-FileCopyrightText: 2026 Coffey Labs LLC
# SPDX-License-Identifier: Apache-2.0
#
# The lab end to end on a fresh GitHub runner, amd64 and arm64 (S7, K-L1):
# lab/tools.sh, then lab/e2e.sh with the chart from inbuxa-charts' main.
# Runs on the GitHub replica only (vars.BUILD_ON), where the runners are;
# Gitea's runners don't run it.
#
# The full matrix (three Kubernetes minors, nightly, chart PRs) is K-L8.
name: e2e
on:
workflow_dispatch:
# Every branch: the lab's PRs are on Gitea, its branches reach GitHub by
# the push mirror
push:
permissions:
contents: read
concurrency:
group: e2e-${{ github.ref }}
cancel-in-progress: true
jobs:
e2e:
if: ${{ vars.BUILD_ON == 'github' }}
strategy:
fail-fast: false
matrix:
runner: [ubuntu-24.04, ubuntu-24.04-arm]
runs-on: ${{ matrix.runner }}
timeout-minutes: 40
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: inbuxa-k8s-lab
persist-credentials: false
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: inbuxa/inbuxa-charts
# main, unless the repo variable CHART_REF names a chart branch
ref: ${{ vars.CHART_REF || 'main' }}
path: inbuxa-charts
persist-credentials: false
- name: Tools (pinned, checksummed)
run: |
inbuxa-k8s-lab/lab/tools.sh "$HOME/.local/bin"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: The runner
run: |
nproc; free -h | head -2; df -h / | tail -1
docker version --format 'docker {{.Server.Version}}'
- name: Lab end to end
run: inbuxa-k8s-lab/lab/e2e.sh
- name: Memory at the end
if: always()
run: |
free -h | head -2
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' || true
+1
View File
@@ -1 +1,2 @@
__pycache__/
lab/.policy-apply.err
+41 -6
View File
@@ -1,14 +1,49 @@
# inbuxa-k8s-lab
<p align="center">
<img src="./img/brand/inbuxa-lockup-light.svg" alt="inbuxa" height="140">
</p>
Test lab for inbuxa on Kubernetes: end-to-end, chaos, upgrade and soak tests.
<h3 align="center">
The test lab for inbuxa on Kubernetes
</h3>
`lab/up.sh` brings up a kind cluster with the data layer, Kyverno and the
policies. It needs the Helm repositories `cnpg`, `nats`, `metallb` and
`kyverno` (https://kyverno.github.io/kyverno/); the versions are pinned in
`lab/versions.env`.
<p align="center">
<a href="LICENSE"><img alt="License: Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-2dd4bf?style=flat-square"></a>
<a href="https://github.com/inbuxa/inbuxa-k8s-lab/actions/workflows/e2e.yml"><img alt="End to end" src="https://img.shields.io/github/actions/workflow/status/inbuxa/inbuxa-k8s-lab/e2e.yml?branch=main&label=e2e&style=flat-square"></a>
<a href="https://community.coffeylabs.org/c/inbuxa/5"><img alt="Forum: community.coffeylabs.org" src="https://img.shields.io/badge/forum-community.coffeylabs.org-0f766e?style=flat-square"></a>
<a href="https://discord.gg/nqcY4TKfAn"><img alt="Chat on Discord" src="https://img.shields.io/discord/1523538164084637797?label=discord&logo=discord&logoColor=white&color=5865f2&style=flat-square"></a>
</p>
---
> [!NOTE]
> Development happens on [git.coffeylabs.org/inbuxa/inbuxa-k8s-lab](https://git.coffeylabs.org/inbuxa/inbuxa-k8s-lab); the copy on GitHub is a read-only mirror, and where the end-to-end runs happen.
> Report issues at **[git.coffeylabs.org/inbuxa/inbuxa-k8s-lab/issues](https://git.coffeylabs.org/inbuxa/inbuxa-k8s-lab/issues)**, join discussions at **[community.coffeylabs.org](https://community.coffeylabs.org)**, or chat on **[Discord](https://discord.gg/nqcY4TKfAn)**.
Where [the inbuxa Helm chart](https://git.coffeylabs.org/inbuxa/inbuxa-charts)
is tested on real clusters: end to end, upgrades, failures and load. It's
for developing the chart, not for running mail; the chart's repository says
how to install it.
`lab/e2e.sh` is the whole lab from nothing: the cluster, the chart with the
images the next chart release names (inbuxa-charts `release/suite.yaml`),
seeded, then mail, the webmail and the console, timing each part. It wants
inbuxa-charts checked out next to this repository (or `CHART_REPO=`), and
kind, kubectl and Helm, which `lab/tools.sh` installs at pinned versions,
each checked against its published checksum. CI runs it on GitHub's amd64
and arm64 runners (`.github/workflows/e2e.yml`).
`lab/up.sh` is just the cluster: kind, Kyverno and the policies, MetalLB,
and the data layer (CloudNativePG, NATS, Garage). It adds the Helm
repositories it needs; the versions are pinned in `lab/versions.env`.
`lab/down.sh` removes it all.
The `inbuxa` namespace enforces Pod Security "restricted" and the policies in
`lab/policy/defaults.yaml`: a read-only root, no service-account token (except
the upgrade Job's), secrets never in the environment, and a deny-all
NetworkPolicy over every pod. `lab/policy/policy-test.sh` checks that the
chart passes them and that a bad value is refused.
## License
Free software under the [Apache License 2.0](LICENSE), and a Coffey Labs
project.
+52
View File
@@ -0,0 +1,52 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 400" width="1200" height="400">
<rect width="1200" height="400" fill="#ffffff"/>
<g transform="translate(193.84,41.32) scale(2.2956)"><g stroke="#1C4053" stroke-linecap="round" stroke-linejoin="round">
<g transform="translate(0 -12)">
<path d="M18 58 L32 60 M19 67 L32 65 M110 58 L96 60 M109 67 L96 65" stroke-width="3" fill="none"/>
<g transform="translate(64 62) scale(0.9) translate(-64 -60)">
<path d="M31 64 C30 50 31 38 34 27 Q36 20 42 23 L55 32 Q64 29.5 73 32 L86 23 Q92 20 94 27 C97 38 98 50 97 64 C96 83 82 92 64 92 C46 92 32 83 31 64 Z" fill="#F9A34C" stroke-width="5"/>
<path d="M38 30 L50 37.5 L40 45.5 Z M90 30 L78 37.5 L88 45.5 Z" fill="#F47F35" stroke="none"/>
<path d="M44 60 Q49.5 53 55 60 M73 60 Q78.5 53 84 60" stroke-width="4.2" fill="none"/>
<path d="M60.8 65.5 h6.4 l-3.2 3.6 z" fill="#1C4053" stroke-width="2"/>
<path d="M56 71.5 Q60 76.5 64 71.5 Q68 76.5 72 71.5" stroke-width="3.5" fill="none"/>
</g>
</g>
<rect x="12" y="73" width="104" height="51" rx="9" fill="#46CAC3" stroke-width="4.5"/>
<rect x="21" y="89" width="86" height="10" rx="2.5" fill="#1C4053" stroke="none"/>
<rect x="27" y="85.5" width="26" height="14" rx="2" fill="#FFFFFF" stroke-width="2.8"/>
<path d="M28.5 87 L40 93.5 L51.5 87" stroke-width="2.3" fill="none"/>
<circle cx="99" cy="94" r="2.5" fill="#9FF0E9" stroke="none"/>
<rect x="21" y="103" width="86" height="9" rx="2.5" fill="#1C4053" stroke="none"/>
<path d="M28 104.8 L31.8 107.5 L28 110.2 M34.8 110.2 H41" stroke="#9FF0E9" stroke-width="1.9" fill="none"/>
<circle cx="99" cy="107.5" r="2.5" fill="#9FF0E9" stroke="none"/>
<rect x="21" y="115.5" width="86" height="4.5" rx="2" fill="#2B8A86" stroke="none"/>
<circle cx="28" cy="117.75" r="1.6" fill="#F9A34C" stroke="none"/>
<circle cx="34" cy="117.75" r="1.6" fill="#9FF0E9" stroke="none"/>
<circle cx="40" cy="117.75" r="1.6" fill="#9FF0E9" stroke="none"/>
<g transform="translate(0 -12)">
<path d="M36.5 90 C36 81 40 76.5 46 76.5 C52 76.5 56 81 55.5 90 C55.5 94 51 96 46 96 C41 96 36.5 94 36.5 90 Z" fill="#F9A34C" stroke-width="4"/>
<path d="M91.5 90 C92 81 88 76.5 82 76.5 C76 76.5 72 81 72.5 90 C72.5 94 77 96 82 96 C87 96 91.5 94 91.5 90 Z" fill="#F9A34C" stroke-width="4"/>
<path d="M43 88.5 V93 M49 88.5 V93 M79 88.5 V93 M85 88.5 V93" stroke-width="2.4" fill="none"/>
</g>
</g></g>
<g fill="#1C4053">
<path d="M70 0V496H196V0ZM133 554Q99 554 75.5 576.0Q52 598 52 634Q52 670 75.5 692.0Q99 714 133 714Q168 714 191.0 692.0Q214 670 214 634Q214 598 191.0 576.0Q168 554 133 554Z" transform="translate(509.29,227.62) scale(0.148000,-0.148000)"/>
<path d="M70 0V496H194V431H212Q224 457 257.0 480.5Q290 504 357 504Q415 504 458.5 477.5Q502 451 526.0 404.5Q550 358 550 296V0H424V286Q424 342 396.5 370.0Q369 398 318 398Q260 398 228.0 359.5Q196 321 196 252V0Z" transform="translate(545.70,227.62) scale(0.148000,-0.148000)"/>
<path d="M368 -14Q301 -14 265.0 9.0Q229 32 212 60H194V0H70V700H196V439H214Q225 457 243.5 473.0Q262 489 292.5 499.5Q323 510 368 510Q428 510 479.0 480.5Q530 451 561.0 394.0Q592 337 592 256V240Q592 159 561.0 102.0Q530 45 479.0 15.5Q428 -14 368 -14ZM330 96Q388 96 427.0 133.5Q466 171 466 243V253Q466 325 427.5 362.5Q389 400 330 400Q272 400 233.0 362.5Q194 325 194 253V243Q194 171 233.0 133.5Q272 96 330 96Z" transform="translate(633.91,227.62) scale(0.148000,-0.148000)"/>
<path d="M259 -8Q201 -8 157.5 18.5Q114 45 90.0 92.0Q66 139 66 200V496H192V210Q192 154 219.5 126.0Q247 98 298 98Q356 98 388.0 136.5Q420 175 420 244V496H546V0H422V65H404Q392 40 359.0 16.0Q326 -8 259 -8Z" transform="translate(725.37,227.62) scale(0.148000,-0.148000)"/>
<path d="M26 0 206 250 28 496H174L287 331H305L418 496H564L386 250L566 0H418L305 167H287L174 0Z" transform="translate(813.58,227.62) scale(0.148000,-0.148000)"/>
<path d="M224 -14Q171 -14 129.0 4.5Q87 23 62.5 58.5Q38 94 38 145Q38 196 62.5 230.5Q87 265 130.5 282.5Q174 300 230 300H366V328Q366 363 344.0 385.5Q322 408 274 408Q227 408 204.0 386.5Q181 365 174 331L58 370Q70 408 96.5 439.5Q123 471 167.5 490.5Q212 510 276 510Q374 510 431.0 461.0Q488 412 488 319V134Q488 104 516 104H556V0H472Q435 0 411.0 18.0Q387 36 387 66V67H368Q364 55 350.0 35.5Q336 16 306.0 1.0Q276 -14 224 -14ZM246 88Q299 88 332.5 117.5Q366 147 366 196V206H239Q204 206 184.0 191.0Q164 176 164 149Q164 122 185.0 105.0Q206 88 246 88Z" transform="translate(898.24,227.62) scale(0.148000,-0.148000)"/>
</g>
<g fill="#1C4053">
<path d="M73 0V700H276L429 72H445L598 700H801V0H696V624H680L528 0H346L194 624H178V0Z" transform="translate(509.29,310.42) scale(0.026000,-0.026000)"/>
<path d="M21 0 221 700H411L610 0H499L453 165H179L133 0ZM206 265H426L324 633H308Z" transform="translate(537.22,310.42) scale(0.026000,-0.026000)"/>
<path d="M73 0V700H181V0Z" transform="translate(558.85,310.42) scale(0.026000,-0.026000)"/>
<path d="M73 0V700H181V98H518V0Z" transform="translate(570.65,310.42) scale(0.026000,-0.026000)"/>
<path d="M311 -14Q232 -14 171.0 14.0Q110 42 75.5 96.5Q41 151 41 229V254H148V229Q148 155 192.5 118.0Q237 81 311 81Q386 81 424.0 112.5Q462 144 462 194Q462 227 444.0 248.0Q426 269 393.0 281.5Q360 294 314 305L277 314Q210 330 160.5 353.5Q111 377 84.0 416.0Q57 455 57 516Q57 578 87.5 622.0Q118 666 172.0 690.0Q226 714 299 714Q372 714 429.5 689.0Q487 664 520.0 615.0Q553 566 553 492V456H446V492Q446 537 427.5 565.0Q409 593 376.0 606.0Q343 619 299 619Q235 619 199.5 592.5Q164 566 164 519Q164 487 179.5 466.5Q195 446 225.0 433.0Q255 420 300 410L337 401Q405 386 457.0 362.5Q509 339 539.0 299.0Q569 259 569 196Q569 133 537.5 86.0Q506 39 448.0 12.5Q390 -14 311 -14Z" transform="translate(601.83,310.42) scale(0.026000,-0.026000)"/>
<path d="M73 0V700H514V602H181V401H486V303H181V98H519V0Z" transform="translate(622.86,310.42) scale(0.026000,-0.026000)"/>
<path d="M73 0V700H367Q432 700 481.5 677.5Q531 655 558.5 613.0Q586 571 586 513V502Q586 436 554.5 396.5Q523 357 478 341V325Q516 323 539.0 299.5Q562 276 562 232V0H453V217Q453 244 439.0 260.0Q425 276 394 276H181V0ZM181 374H355Q413 374 445.0 403.5Q477 433 477 484V492Q477 543 445.5 572.5Q414 602 355 602H181Z" transform="translate(642.59,310.42) scale(0.026000,-0.026000)"/>
<path d="M216 0 21 700H133L303 68H316L485 700H597L403 0Z" transform="translate(664.23,310.42) scale(0.026000,-0.026000)"/>
<path d="M73 0V700H514V602H181V401H486V303H181V98H519V0Z" transform="translate(685.52,310.42) scale(0.026000,-0.026000)"/>
<path d="M73 0V700H367Q432 700 481.5 677.5Q531 655 558.5 613.0Q586 571 586 513V502Q586 436 554.5 396.5Q523 357 478 341V325Q516 323 539.0 299.5Q562 276 562 232V0H453V217Q453 244 439.0 260.0Q425 276 394 276H181V0ZM181 374H355Q413 374 445.0 403.5Q477 433 477 484V492Q477 543 445.5 572.5Q414 602 355 602H181Z" transform="translate(705.25,310.42) scale(0.026000,-0.026000)"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 6.7 KiB

Executable
+139
View File
@@ -0,0 +1,139 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: 2026 Coffey Labs LLC
# SPDX-License-Identifier: Apache-2.0
#
# The lab end to end, from nothing (K-L1, S7): the kind cluster and data
# layer (up.sh), then the chart with the released images the next chart
# release names (inbuxa-charts release/suite.yaml), seeded, then
#
# 1. mail: in over SMTP, found over IMAPS, through the mail Service
# 2. the webmail: signing in across both replicas
# 3. the console: signing in as its browser would
#
# and how long each part took. CI runs this on a fresh runner; on the bench
# it builds the lab from scratch too, so take the lab down first
# (lab/down.sh) unless you mean to reuse it.
#
# lab/e2e.sh # chart in ../inbuxa-charts
# CHART_REPO=path lab/e2e.sh
# KIND_NODE_IMAGE=kindest/node:v1.35.4 lab/e2e.sh
#
# Prints a summary, also to $GITHUB_STEP_SUMMARY when it's set, and on a
# failure what the pods and events were.
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
. "$here/versions.env"
CHART_REPO="${CHART_REPO:-$here/../../inbuxa-charts}"
CHART="$CHART_REPO/charts/inbuxa"
SUITE="$CHART_REPO/release/suite.yaml"
STATE="${STATE:-$HOME/.cache/inbuxa-lab}"
K=(kubectl --context "kind-$KIND_CLUSTER" -n inbuxa)
P="${PORTS:-18080}"
PIDS=()
t0=$SECONDS
SUMMARY=()
lap() { SUMMARY+=("| $1 | $(( SECONDS - ${2:-$t0} ))s |"); }
report() {
{ echo "| Part | Took |"; echo "|---|---|"; printf '%s\n' "${SUMMARY[@]}"; } > "$STATE/e2e-summary.md"
cat "$STATE/e2e-summary.md"
[ -n "${GITHUB_STEP_SUMMARY:-}" ] && { echo "### inbuxa lab, $(uname -m)"; cat "$STATE/e2e-summary.md"; } >> "$GITHUB_STEP_SUMMARY"
return 0
}
on_exit() {
rc=$?
for p in "${PIDS[@]}"; do kill "$p" 2>/dev/null || true; done
if [ "$rc" != 0 ]; then
SUMMARY+=("| **failed** | after $(( SECONDS - t0 ))s |")
echo "== failed: pods and events"
kubectl --context "kind-$KIND_CLUSTER" get pods -A -o wide 2>/dev/null || true
"${K[@]}" get events --sort-by=.lastTimestamp 2>/dev/null | tail -40 || true
"${K[@]}" logs -l app.kubernetes.io/component=server --tail=30 --prefix 2>/dev/null | grep -v '^\s*$' | tail -60 || true
fi
report
exit "$rc"
}
trap on_exit EXIT
pin() { awk -v k="$1:" '$1 == k {print $2}' "$SUITE"; }
SERVER="$(pin server)"; WEBMAIL="$(pin webmail)"; CONSOLE="$(pin console)"
[ -n "$SERVER" ] && [ -n "$WEBMAIL" ] && [ -n "$CONSOLE" ] || { echo "!! no images in $SUITE" >&2; exit 1; }
echo "== images: server $SERVER, webmail $WEBMAIL, console $CONSOLE"
mkdir -p "$STATE" && chmod 700 "$STATE"
echo "== the cluster and the data layer"
s=$SECONDS
"$here/up.sh"
lap "kind, Kyverno, MetalLB, CloudNativePG, NATS, Garage" "$s"
echo "== the seed's passwords and setup"
umask 077
new_password() { python3 -c 'import secrets; print(secrets.token_urlsafe(24))'; }
new_password > "$STATE/admin.password"
new_password > "$STATE/alice.password"
"${K[@]}" get secret inbuxa-s3 -o jsonpath='{.data.access-key}' | base64 -d > "$STATE/s3-access-key"
"${K[@]}" create secret generic inbuxa-seed \
--from-file=admin-password="$STATE/admin.password" \
--from-file=alice-password="$STATE/alice.password" \
--from-file=s3-access-key="$STATE/s3-access-key" --dry-run=client -o yaml | "${K[@]}" apply -f - >/dev/null
rm -f "$STATE/s3-access-key"
"${K[@]}" create configmap inbuxa-lab-seed --from-file="$here/chart/seed.json" --dry-run=client -o yaml \
| "${K[@]}" apply -f - >/dev/null
umask 022
echo "== the chart: seeded, two replicas of everything"
s=$SECONDS
helm --kube-context "kind-$KIND_CLUSTER" -n inbuxa upgrade --install inbuxa "$CHART" \
-f "$here/chart/values-lab.yaml" -f "$here/chart/values-frontends.yaml" \
--set server.image.repository=ghcr.io/inbuxa/inbuxa-server --set server.image.tag="$SERVER" \
--set server.image.pullPolicy=IfNotPresent \
--set webmail.image.repository=ghcr.io/inbuxa/inbuxa-webmail --set webmail.image.tag="$WEBMAIL" \
--set webmail.image.pullPolicy=IfNotPresent \
--set console.image.repository=ghcr.io/inbuxa/inbuxa-admin --set console.image.tag="$CONSOLE" \
--set console.image.pullPolicy=IfNotPresent \
--wait --timeout 15m >/dev/null
"${K[@]}" get pods --no-headers -o wide | awk '{print " " $1, $3, $7}'
lap "chart install (seed, server, webmail, console)" "$s"
echo "== 1. mail"
s=$SECONDS
lb=""
for _ in $(seq 1 30); do
lb="$("${K[@]}" get svc inbuxa-mail -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
[ -n "$lb" ] && break
sleep 2
done
[ -n "$lb" ] || { echo "!! the mail Service has no address" >&2; exit 1; }
ACCOUNT=[email protected] PASSWORD_FILE="$STATE/alice.password" timeout 150 python3 "$here/chart/mailtest.py" "$lb"
lap "mail: SMTP in, IMAPS found" "$s"
forward() { # <local port> <target>
if ss -ltn "sport = :$1" | grep -q LISTEN; then
echo "!! port $1 is in use: PORTS=<first of four free ports> $0" >&2
exit 1
fi
"${K[@]}" port-forward "$2" "$1:8080" >/dev/null 2>&1 &
PIDS+=($!)
}
mapfile -t webmail < <("${K[@]}" get pods -l app.kubernetes.io/component=webmail \
--field-selector status.phase=Running -o name | sort)
[ "${#webmail[@]}" -ge 2 ] || { echo "!! fewer than two webmail pods" >&2; exit 1; }
forward $((P + 1)) "${webmail[0]}"
forward $((P + 2)) "${webmail[1]}"
forward "$P" svc/inbuxa-http
forward $((P + 3)) svc/inbuxa-console
sleep 3
echo "== 2. webmail: start on ${webmail[0]#pod/}, finish on ${webmail[1]#pod/}"
s=$SECONDS
python3 "$here/chart/signin.py" "http://127.0.0.1:$((P + 1))" "http://127.0.0.1:$((P + 2))" \
"http://127.0.0.1:$P" [email protected] "$STATE/alice.password"
lap "webmail sign-in across replicas" "$s"
echo "== 3. console"
s=$SECONDS
python3 "$here/chart/console-signin.py" "http://127.0.0.1:$((P + 3))" "http://127.0.0.1:$P" \
https://admin.lab.test https://mail.lab.test [email protected] "$STATE/admin.password"
lap "console sign-in" "$s"
lap "**total**"
echo "== passed in $(( SECONDS - t0 ))s"
Executable
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: 2026 Coffey Labs LLC
# SPDX-License-Identifier: Apache-2.0
#
# The tools the lab runs with, at pinned versions, each checked against the
# project's published SHA-256 (K-L1): kind, kubectl and Helm, for linux/amd64
# or linux/arm64, into DIR (default ~/.local/bin).
#
# lab/tools.sh [DIR]
#
# To move a pin, change the version and both of its sums together.
set -euo pipefail
DIR="${1:-$HOME/.local/bin}"
case "$(uname -m)" in
x86_64) ARCH=amd64 ;;
aarch64|arm64) ARCH=arm64 ;;
*) echo "linux/amd64 or linux/arm64 only" >&2; exit 2 ;;
esac
KIND_VERSION=v0.33.0
KIND_SHA256_amd64=aee6151561422756b764a4ae28e7f44cda5af5a9eead3cc9985112b1de8d8e0d
KIND_SHA256_arm64=20022bee6cfcd5086cb7234d218e3454e6090022f2a8f55d1fa7fcf42c3867a2
KUBECTL_VERSION=v1.37.2
KUBECTL_SHA256_amd64=c04487523e7aa7d1493f7808174c899db93d7a9cf57531420cd76b9bd415f8f4
KUBECTL_SHA256_arm64=fcc16f63186659d9722014b7e02d5fdd2d4e669cb8c0ab66365d7d3b8ba03509
HELM_VERSION=v4.3.2
HELM_SHA256_amd64=d1c0278d46beab79582d7e50f5b3dc206944ba8e577cde618513fd07daa53983
HELM_SHA256_arm64=df31cb61680ef728f82c770e5280dd565617bee6b039d37e85c677288be4bbb0
mkdir -p "$DIR"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
sum() { local v="$1_SHA256_$ARCH"; echo "${!v}"; }
# fetch URL SHA256 FILE: download URL to $WORK/FILE and check its hash
fetch() {
curl -fsSL --retry 3 --proto '=https' -o "$WORK/$3" "$1"
echo "$2 $WORK/$3" | sha256sum -c --quiet -
}
fetch "https://github.com/kubernetes-sigs/kind/releases/download/$KIND_VERSION/kind-linux-$ARCH" "$(sum KIND)" kind
install -m 0755 "$WORK/kind" "$DIR/kind"
fetch "https://dl.k8s.io/release/$KUBECTL_VERSION/bin/linux/$ARCH/kubectl" "$(sum KUBECTL)" kubectl
install -m 0755 "$WORK/kubectl" "$DIR/kubectl"
fetch "https://get.helm.sh/helm-$HELM_VERSION-linux-$ARCH.tar.gz" "$(sum HELM)" helm.tgz
tar -xzf "$WORK/helm.tgz" -C "$WORK" "linux-$ARCH/helm"
install -m 0755 "$WORK/linux-$ARCH/helm" "$DIR/helm"
"$DIR/kind" version
"$DIR/kubectl" version --client | head -1
"$DIR/helm" version --short
+35 -3
View File
@@ -10,8 +10,18 @@ t0=$SECONDS
step() { printf '\n== %s (%ss)\n' "$1" "$((SECONDS - t0))"; }
rand() { head -c 24 /dev/urandom | base64 | tr -d '/+=' ; }
step "Helm repositories"
helm repo add cnpg https://cloudnative-pg.github.io/charts --force-update >/dev/null
helm repo add nats https://nats-io.github.io/k8s/helm/charts/ --force-update >/dev/null
helm repo add metallb https://metallb.github.io/metallb --force-update >/dev/null
helm repo add kyverno https://kyverno.github.io/kyverno/ --force-update >/dev/null
helm repo update >/dev/null
step "kind cluster"
kind get clusters 2>/dev/null | grep -qx "$KIND_CLUSTER" || kind create cluster --name "$KIND_CLUSTER" --config "$here/kind.yaml" --wait 120s
# KIND_NODE_IMAGE picks the Kubernetes version (CI's matrix); unset, kind's own
kind get clusters 2>/dev/null | grep -qx "$KIND_CLUSTER" ||
kind create cluster --name "$KIND_CLUSTER" --config "$here/kind.yaml" --wait 120s \
${KIND_NODE_IMAGE:+--image "$KIND_NODE_IMAGE"}
kubectl config use-context "kind-$KIND_CLUSTER" >/dev/null
for ns in data inbuxa; do kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f - >/dev/null; done
# inbuxa's pods run under Pod Security "restricted", enforced (K-C4), and
@@ -19,10 +29,25 @@ for ns in data inbuxa; do kubectl create namespace "$ns" --dry-run=client -o yam
kubectl label --overwrite namespace inbuxa inbuxa.org/policy=enforce \
pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/warn=restricted >/dev/null
step "MetalLB"
# LoadBalancer addresses from the top of kind's Docker network, which the
# host reaches: computed, since that network's subnet differs by machine
helm upgrade --install metallb metallb/metallb --version "$METALLB_CHART" -n metallb-system --create-namespace --wait >/dev/null
subnet=$(docker network inspect kind -f '{{range .IPAM.Config}}{{.Subnet}} {{end}}' | tr ' ' '\n' | grep -v ':' | head -1)
pool=$(python3 -c 'import ipaddress,sys; n=ipaddress.ip_network(sys.argv[1]); print(f"{n.broadcast_address-55}-{n.broadcast_address-5}")' "$subnet")
sed "s|172.19.255.200-172.19.255.250|$pool|" "$here/network/metallb-pool.yaml" | kubectl apply -f - >/dev/null
echo " LoadBalancer pool $pool ($subnet)"
step "Kyverno"
helm upgrade --install kyverno kyverno/kyverno --version "$KYVERNO_CHART" -n kyverno --create-namespace \
--set cleanupController.enabled=false --set reportsController.enabled=false --wait >/dev/null
kubectl apply -f "$here/policy/defaults.yaml" >/dev/null
# Helm's --wait returns before the webhook answers on a new cluster
for i in $(seq 1 30); do
kubectl apply -f "$here/policy/defaults.yaml" >/dev/null 2>"$here/.policy-apply.err" && break
[ "$i" = 30 ] && { cat "$here/.policy-apply.err" >&2; rm -f "$here/.policy-apply.err"; exit 1; }
sleep 3
done
rm -f "$here/.policy-apply.err"
step "CloudNativePG operator"
helm upgrade --install cnpg cnpg/cloudnative-pg --version "$CNPG_CHART" -n cnpg-system --create-namespace --wait >/dev/null
@@ -38,7 +63,14 @@ sed "s|GARAGE_IMAGE|$GARAGE_IMAGE|" "$here/data-layer/garage.yaml" | kubectl -n
kubectl -n data rollout status deploy/garage --timeout=180s >/dev/null
g() { kubectl -n data exec deploy/garage -- /garage "$@"; }
if ! g bucket info inbuxa-blobs >/dev/null 2>&1; then
node=$(g status 2>/dev/null | awk '/^[0-9a-f]{16}/{print $1; exit}')
# Ready to Kubernetes isn't ready to answer: wait for Garage to list itself
node=""
for _ in $(seq 1 30); do
node=$( (g status 2>/dev/null || true) | awk '/^[0-9a-f]{16}/{print $1; exit}')
[ -n "$node" ] && break
sleep 2
done
[ -n "$node" ] || { echo "!! Garage never listed its node" >&2; exit 1; }
g layout assign -z lab -c 2G "$node" >/dev/null
g layout apply --version 1 >/dev/null
g bucket create inbuxa-blobs >/dev/null