Obtain certificates, and wait for the one that matters
The public shape now works end to end: real ports, Caddy in front, and both programs that need certificates getting them from the same CA -- Caddy for the front ends over TLS-ALPN-01, the mail server for its own names over HTTP-01, which Caddy forwards on port 80. Proved in the lab against Pebble, with a DNS stub answering every name with the machine's own address, so no public name or public CA is involved: twenty checks, ending with IMAPS and submissions presenting a certificate for the mail host that verifies against the CA, and the webmail sending sign-in to the server as the first-party client the server registered. Two things the test found, both of which would have shipped: - The proxy fronted four of the server's five names. The server puts ua-auto-config in its own certificate too, so its challenge was never forwarded, one name failed, and the whole order failed with it -- leaving the mail ports on a self-signed certificate while everything else looked healthy. The list now matches what the server asks for. - Nothing waited for the certificate. An order that fails is not retried on its own and a restart does not start a new one, so the install declared itself finished over a self-signed certificate. It now waits, asks again every 45 seconds, and reports the issuer -- or says plainly that the server will keep trying once the domain resolves here, which is the ordinary case on a first install. --acme-directory and --acme-ca-root are what let a private CA be used: the root is added to the server image's own bundle and given to Caddy, because neither sees the other's trust store.
This commit is contained in:
@@ -43,7 +43,10 @@ the installer draft); the phases are there too.
|
||||
inbuxa install --local --domain example.test --install-deps --yes
|
||||
|
||||
is the shortest thing that works today: the whole suite on loopback, with no
|
||||
DNS and no certificates, on a machine that starts with nothing.
|
||||
DNS and no certificates, on a machine that starts with nothing. Without
|
||||
`--local` it takes the real ports, puts Caddy in front and obtains
|
||||
certificates -- which `e2e/cases/install-public.sh` proves against a private
|
||||
CA, with no internet and no public name involved.
|
||||
|
||||
## Building and testing
|
||||
|
||||
@@ -56,6 +59,7 @@ is tested on a throwaway virtual machine rather than on anybody's desk:
|
||||
e2e/vm/run.sh e2e/cases/survey.sh what it says about a machine
|
||||
e2e/vm/run.sh e2e/cases/deps.sh the offer, and taking it
|
||||
e2e/vm/run.sh e2e/cases/install-local.sh a whole suite, and signing in to it
|
||||
e2e/vm/run.sh e2e/cases/install-public.sh the same with real ports and certificates
|
||||
e2e/vm/down.sh remove it
|
||||
|
||||
Each case starts from a copy of the machine taken when it was new, so a run
|
||||
|
||||
@@ -46,6 +46,9 @@ install flags:
|
||||
--proxy WHICH caddy | snippets | none (default: caddy)
|
||||
--dir PATH where the installation lives; default: /var/lib/inbuxa
|
||||
--local loopback evaluation: no public ports, no certificates
|
||||
--acme-directory URL a private ACME CA, for testing the certificate path
|
||||
--acme-ca-root PATH that CA's root, which both the server and the proxy
|
||||
are made to trust
|
||||
--install-deps install what the chosen shapes need and this
|
||||
machine lacks, rather than refusing over it
|
||||
--dry-run print the plan and stop
|
||||
@@ -107,6 +110,8 @@ func install(args []string) int {
|
||||
fs.StringVar(&o.Dir, "dir", "", "")
|
||||
fs.BoolVar(&o.Local, "local", false, "")
|
||||
fs.BoolVar(&o.InstallDeps, "install-deps", false, "")
|
||||
fs.StringVar(&o.ACMEDirectory, "acme-directory", "", "")
|
||||
fs.StringVar(&o.ACMECARoot, "acme-ca-root", "", "")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return 2
|
||||
}
|
||||
@@ -164,6 +169,9 @@ func install(args []string) int {
|
||||
if res.WebmailURL != "" {
|
||||
fmt.Printf(" webmail %s\n", res.WebmailURL)
|
||||
}
|
||||
if res.Certificate != "" {
|
||||
fmt.Printf(" certificate issued by %s\n", res.Certificate)
|
||||
}
|
||||
if res.ZoneFile != "" {
|
||||
fmt.Printf(" dns records %s\n", res.ZoneFile)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/bin/bash
|
||||
# SPDX-FileCopyrightText: 2026 Coffey Labs
|
||||
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
#
|
||||
# The public shape, with no internet involved: Pebble stands in for Let's
|
||||
# Encrypt and a DNS stub answers every name with this machine's address, so
|
||||
# both Caddy and the mail server really obtain certificates, over the real
|
||||
# ports, through the real Caddyfile.
|
||||
#
|
||||
# --local proves the pieces talk to each other. This proves the part an
|
||||
# operator actually gets wrong: ports 25 and 443 held for real, two programs
|
||||
# asking the same CA for certificates for overlapping names, and a proxy in
|
||||
# front of all of it. The two are kept apart by challenge type -- Caddy uses
|
||||
# TLS-ALPN-01 on 443, the server HTTP-01 on 80, which Caddy forwards -- and
|
||||
# this is where that is checked.
|
||||
#
|
||||
# Run from the host with: e2e/vm/run.sh e2e/cases/install-public.sh
|
||||
set -uo pipefail
|
||||
|
||||
pass=0; fail=0
|
||||
ok() { echo " ok $*"; pass=$((pass+1)); }
|
||||
bad() { echo " FAIL $*"; fail=$((fail+1)); }
|
||||
|
||||
DOMAIN=lab.test
|
||||
MAIL=mx.lab.test # not "mail": proves the names follow --mail-host
|
||||
CONSOLE=console.lab.test
|
||||
WEBMAIL=webmail.lab.test
|
||||
DIR=/var/lib/inbuxa
|
||||
WORK=/tmp/lab
|
||||
LABNET=inbuxa-e2e
|
||||
LABSUBNET=172.31.254.0/24
|
||||
HOSTIP=172.31.254.1 # this machine, as the lab network sees it
|
||||
|
||||
rm -rf "$WORK"; mkdir -p "$WORK"
|
||||
|
||||
echo "==> what the installer needs, before the lab"
|
||||
/tmp/inbuxa deps --console container --webmail container --install >/dev/null 2>&1 || true
|
||||
docker version >/dev/null 2>&1 && ok "docker is usable" || { bad "no docker"; exit 1; }
|
||||
|
||||
echo
|
||||
echo "==> standing up a private CA and a DNS stub"
|
||||
docker network create --subnet "$LABSUBNET" "$LABNET" >/dev/null 2>&1
|
||||
# Pebble's own certificate names localhost and "pebble"; the server and Caddy
|
||||
# reach it at this machine's address from another network, so it gets one for
|
||||
# that address, signed by the test root that ships with it.
|
||||
docker create --name inbuxa-e2e-extract ghcr.io/letsencrypt/pebble:latest >/dev/null 2>&1
|
||||
docker cp inbuxa-e2e-extract:/test/certs "$WORK/pebble-certs" >/dev/null
|
||||
docker cp inbuxa-e2e-extract:/test/config/pebble-config.json "$WORK/pebble-config.json" >/dev/null
|
||||
docker rm inbuxa-e2e-extract >/dev/null
|
||||
openssl req -new -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -subj "/CN=pebble" \
|
||||
-keyout "$WORK/pebble-key.pem" -out "$WORK/pebble.csr" 2>/dev/null
|
||||
openssl x509 -req -in "$WORK/pebble.csr" -days 2 -CA "$WORK/pebble-certs/pebble.minica.pem" \
|
||||
-CAkey "$WORK/pebble-certs/pebble.minica.key.pem" -CAcreateserial \
|
||||
-extfile <(printf 'subjectAltName=IP:%s\nextendedKeyUsage=serverAuth\n' "$HOSTIP") \
|
||||
-out "$WORK/pebble-cert.pem" 2>/dev/null
|
||||
python3 - "$WORK/pebble-config.json" <<'PY'
|
||||
import json, sys
|
||||
c = json.load(open(sys.argv[1]))
|
||||
c["pebble"].update(httpPort=80, tlsPort=443,
|
||||
certificate="/work/pebble-cert.pem", privateKey="/work/pebble-key.pem")
|
||||
json.dump(c, open(sys.argv[1], "w"))
|
||||
PY
|
||||
chmod -R a+r "$WORK"
|
||||
docker run -d --name inbuxa-e2e-dns --network "$LABNET" --ip 172.31.254.3 \
|
||||
ghcr.io/letsencrypt/pebble-challtestsrv:latest \
|
||||
-defaultIPv4 "$HOSTIP" -defaultIPv6 "" -http01 "" -https01 "" -tlsalpn01 "" -doh "" >/dev/null
|
||||
# Nonce rejection off: Pebble refuses 5% of nonces on purpose, and the server
|
||||
# gives up an order on the first one rather than retrying.
|
||||
docker run -d --name inbuxa-e2e-pebble --network "$LABNET" --ip 172.31.254.2 \
|
||||
-p 14000:14000 -p 15000:15000 -v "$WORK:/work:ro" \
|
||||
-e PEBBLE_VA_NOSLEEP=1 -e PEBBLE_WFE_NONCEREJECT=0 \
|
||||
ghcr.io/letsencrypt/pebble:latest -config /work/pebble-config.json -dnsserver 172.31.254.3:8053 >/dev/null
|
||||
for _ in $(seq 1 30); do
|
||||
curl -sf --cacert "$WORK/pebble-certs/pebble.minica.pem" "https://$HOSTIP:14000/dir" >/dev/null && break
|
||||
sleep 1
|
||||
done
|
||||
curl -sf --cacert "$WORK/pebble-certs/pebble.minica.pem" "https://$HOSTIP:14000/dir" >/dev/null \
|
||||
&& ok "Pebble answers at https://$HOSTIP:14000/dir" || { bad "Pebble did not come up"; exit 1; }
|
||||
|
||||
echo
|
||||
echo "==> installing the public shape"
|
||||
OUT="$(/tmp/inbuxa install --domain "$DOMAIN" --mail-host "$MAIL" --console-host "$CONSOLE" \
|
||||
--webmail-host "$WEBMAIL" --acme-directory "https://$HOSTIP:14000/dir" \
|
||||
--acme-ca-root "$WORK/pebble-certs/pebble.minica.pem" --yes 2>&1)"; rc=$?
|
||||
echo "$OUT" | grep -v '^ |' | tail -22 | sed 's/^/ /'
|
||||
[ $rc -eq 0 ] && ok "installed (exit 0)" || bad "exit $rc"
|
||||
|
||||
echo
|
||||
echo "==> the ports an operator would expect"
|
||||
for p in 25 80 443 465 993 995 4190; do
|
||||
ss -ltn "sport = :$p" | grep -q LISTEN && ok "$p is listening" || bad "$p is not listening"
|
||||
done
|
||||
|
||||
# Caddy obtains its certificates in the background, so give the first
|
||||
# handshake for each name a little room.
|
||||
expect() {
|
||||
local want=$1 got=; shift
|
||||
for _ in $(seq 1 40); do
|
||||
got=$(curl -s -o /dev/null -w '%{http_code}' "$@" 2>/dev/null || true)
|
||||
[ "$got" = "$want" ] && return 0
|
||||
sleep 1
|
||||
done
|
||||
echo " got HTTP ${got:-nothing}, wanted $want" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
CA="$WORK/pebble-certs/pebble.minica.pem"
|
||||
curl -sf --cacert "$CA" "https://$HOSTIP:15000/roots/0" > "$WORK/root.pem"
|
||||
curl -sf --cacert "$CA" "https://$HOSTIP:15000/intermediates/0" > "$WORK/int.pem"
|
||||
cat "$WORK/int.pem" "$WORK/root.pem" > "$WORK/chain.pem"
|
||||
|
||||
echo
|
||||
echo "==> the front ends, over HTTPS, with certificates from the CA"
|
||||
expect 200 --cacert "$WORK/chain.pem" --resolve "$WEBMAIL:443:127.0.0.1" "https://$WEBMAIL/api/health" \
|
||||
&& ok "the webmail answers over HTTPS" || bad "the webmail does not answer over HTTPS"
|
||||
expect 200 --cacert "$WORK/chain.pem" --resolve "$CONSOLE:443:127.0.0.1" "https://$CONSOLE/" \
|
||||
&& ok "the console answers over HTTPS" || bad "the console does not answer over HTTPS"
|
||||
expect 308 --cacert "$WORK/chain.pem" --resolve "$WEBMAIL:80:127.0.0.1" "http://$WEBMAIL/" \
|
||||
&& ok "port 80 redirects" || bad "port 80 does not redirect"
|
||||
|
||||
echo
|
||||
echo "==> the mail server's own certificate, on the mail ports"
|
||||
grep -q "certificate issued by" <<<"$OUT" && ok "the install reported a certificate" || bad "the install reported no certificate"
|
||||
for port in 993 465; do
|
||||
out=$(echo | timeout 20 openssl s_client -connect "127.0.0.1:$port" -servername "$MAIL" \
|
||||
-verify_hostname "$MAIL" -CAfile "$WORK/chain.pem" 2>&1)
|
||||
grep -q "Verify return code: 0 (ok)" <<<"$out" \
|
||||
&& ok "$port presents a certificate for $MAIL, issued by the CA" \
|
||||
|| { bad "$port did not verify"; grep -E "Verify return code|subject=|issuer=" <<<"$out" | sed 's/^/ /'; }
|
||||
done
|
||||
|
||||
echo
|
||||
echo "==> sign-in goes to the mail server's own page, as a registered client"
|
||||
# Unlike --local, this shape has OAuth: the webmail refuses a password of its
|
||||
# own and sends the browser to the server, as the first-party client the
|
||||
# server registered for this URL. A 302 to the mail host is that working.
|
||||
USER=$(awk '/^first mailbox/ {print $3}' "$DIR/credentials.txt")
|
||||
LOC=$(curl -s -o /dev/null -w '%{redirect_url}' --cacert "$WORK/chain.pem" \
|
||||
--resolve "$WEBMAIL:443:127.0.0.1" --resolve "$MAIL:443:127.0.0.1" \
|
||||
"https://$WEBMAIL/api/auth/oauth/start?username=$USER&remember=1")
|
||||
grep -q "^https://$MAIL/" <<<"$LOC" && ok "the webmail sends sign-in to $MAIL" || bad "sign-in went to '${LOC:-nowhere}'"
|
||||
grep -q "client_id=ihasmail-inbuxa" <<<"$LOC" && ok "as the first-party client" || bad "no first-party client id in '$LOC'"
|
||||
CODE=$(curl -s -o /dev/null -w '%{http_code}' --cacert "$WORK/chain.pem" --resolve "$MAIL:443:127.0.0.1" "$LOC")
|
||||
[ "$CODE" = 200 ] && ok "and the server serves that page over its own certificate" || bad "the server answered $CODE for the sign-in page"
|
||||
|
||||
echo
|
||||
echo "==> and nothing was left behind that should not be"
|
||||
ENVOUT="$(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$(docker compose -f $DIR/compose.yaml ps -q server)")"
|
||||
grep -q "RECOVERY_ADMIN" <<<"$ENVOUT" && bad "the server still carries a recovery admin" || ok "no recovery admin on the server"
|
||||
|
||||
echo
|
||||
echo "==> $pass passed, $fail failed"
|
||||
[ "$fail" -eq 0 ]
|
||||
+72
-1
@@ -54,6 +54,7 @@ type Result struct {
|
||||
FirstUser string
|
||||
FirstPass string
|
||||
ZoneFile string
|
||||
Certificate string // what issued the mail server's certificate, when one arrived
|
||||
ConsoleURL string
|
||||
WebmailURL string
|
||||
ServerURL string
|
||||
@@ -130,6 +131,34 @@ func Run(ctx context.Context, p plan.Plan, log Log) (*Result, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// A private ACME CA has to be trusted by two programs that never see each
|
||||
// other's trust store: the mail server, which asks for its own
|
||||
// certificate over HTTP-01, and Caddy, which asks for the front ends'.
|
||||
// The server gets the image's own roots plus this one as a file it mounts
|
||||
// over its bundle; Caddy gets the root on its own.
|
||||
if o.ACMECARoot != "" {
|
||||
root, err := os.ReadFile(o.ACMECARoot)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading the ACME CA root: %w", err)
|
||||
}
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
bundle, err := docker.SystemCABundle(ctx, stack.ServerImage)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading the server image's CA bundle: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "ca-bundle.crt"), append(bundle, root...), 0o644); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "acme-ca-root.pem"), root, 0o644); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stack.CABundle = true
|
||||
stack.ACMECARoot = "acme-ca-root.pem"
|
||||
}
|
||||
stack.ACMEDirectory = o.ACMEDirectory
|
||||
|
||||
log.Step("writing the deployment into %s", dir)
|
||||
if _, err := compose.Write(dir, &stack); err != nil {
|
||||
return nil, err
|
||||
@@ -246,9 +275,21 @@ func Run(ctx context.Context, p plan.Plan, log Log) (*Result, error) {
|
||||
|
||||
if !o.Local {
|
||||
log.Step("turning on certificates")
|
||||
if _, err := srv.EnableACME(ctx, domainID, "", o.ACMEEmail); err != nil {
|
||||
if _, err := srv.EnableACME(ctx, domainID, o.ACMEDirectory, o.ACMEEmail); err != nil {
|
||||
return res, fmt.Errorf("enabling ACME: %w", err)
|
||||
}
|
||||
// An order that fails is not retried on its own, and a restart does
|
||||
// not start a new one: moving the domain to manual and back is what
|
||||
// does. So this waits, and asks again every so often, rather than
|
||||
// declaring the install finished over a self-signed certificate that
|
||||
// every mail client will refuse.
|
||||
if issuer := waitForCertificate(ctx, srv, log, domainID, o.MailHost, 3*time.Minute); issuer != "" {
|
||||
res.Certificate = issuer
|
||||
log.Info("certificate for %s issued by %s", o.MailHost, issuer)
|
||||
} else {
|
||||
log.Info("no certificate yet for %s: the server keeps trying, and will succeed once %s resolves to this machine and port 80 reaches it",
|
||||
o.MailHost, o.MailHost)
|
||||
}
|
||||
}
|
||||
|
||||
log.Step("creating the first account")
|
||||
@@ -276,6 +317,36 @@ func Run(ctx context.Context, p plan.Plan, log Log) (*Result, error) {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// waitForCertificate waits for the mail server's own certificate to arrive,
|
||||
// asking for a fresh order every 45 seconds. It returns the issuer, or ""
|
||||
// when none arrived in time -- which is not a failure: on a real install the
|
||||
// domain often does not point here yet, and the server goes on trying.
|
||||
func waitForCertificate(ctx context.Context, srv *jmap.Client, log Log, domainID, host string, limit time.Duration) string {
|
||||
deadline := time.Now().Add(limit)
|
||||
nextRetry := time.Now().Add(45 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
certs, err := srv.Certificates(ctx)
|
||||
if err == nil {
|
||||
for _, c := range certs {
|
||||
if c.SubjectAlternativeNames[host] && !strings.Contains(c.Issuer, "self signed") {
|
||||
return c.Issuer
|
||||
}
|
||||
}
|
||||
}
|
||||
if time.Now().After(nextRetry) {
|
||||
log.Info("asking for the certificate again")
|
||||
_ = srv.RetryCertificates(ctx, domainID)
|
||||
nextRetry = time.Now().Add(45 * time.Second)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ""
|
||||
case <-time.After(3 * time.Second):
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Verify is the last step: does what was installed actually answer?
|
||||
func Verify(ctx context.Context, res *Result, stackConsole, stackWebmail bool, log Log) []string {
|
||||
var problems []string
|
||||
|
||||
@@ -71,10 +71,17 @@ type Stack struct {
|
||||
CABundle bool
|
||||
}
|
||||
|
||||
// ServerNames is every name the server's certificate and the proxy need.
|
||||
// ServerNames is every name the server's certificate and the proxy need:
|
||||
// the mail host, and the four service names the server puts in its own
|
||||
// certificate by default.
|
||||
//
|
||||
// The list has to match what the server asks for exactly. Leaving
|
||||
// ua-auto-config out of the proxy meant its HTTP-01 challenge was not
|
||||
// forwarded, the whole order failed on that one name, and the mail ports
|
||||
// served a self-signed certificate while every other name validated.
|
||||
func (s Stack) ServerNames() []string {
|
||||
names := []string{s.MailHost}
|
||||
for _, n := range []string{"autoconfig", "autodiscover", "mta-sts"} {
|
||||
for _, n := range []string{"autoconfig", "autodiscover", "mta-sts", "ua-auto-config"} {
|
||||
names = append(names, n+"."+s.Domain)
|
||||
}
|
||||
return names
|
||||
|
||||
@@ -61,6 +61,12 @@ type Options struct {
|
||||
Proxy string // "caddy", "snippets", "none"
|
||||
Choices []Choice
|
||||
InstallDeps bool // resolve what is missing rather than refusing over it
|
||||
|
||||
// A private ACME CA, for testing the certificate path without asking a
|
||||
// public CA for certificates for names that are not ours. Empty means
|
||||
// Let's Encrypt, which is what an install should use.
|
||||
ACMEDirectory string
|
||||
ACMECARoot string
|
||||
}
|
||||
|
||||
// Shape returns what was chosen for a component.
|
||||
|
||||
Reference in New Issue
Block a user