diff --git a/packaging/local-ai/README.md b/packaging/local-ai/README.md new file mode 100644 index 0000000..6eec89b --- /dev/null +++ b/packaging/local-ai/README.md @@ -0,0 +1,67 @@ +# Local AI for spam filtering + +The model behind inbuxa's AI spam classifier, run on the same machine as the +mail server, as production runs it. Nothing here is used by the installer +yet: it's the reference for its bare-metal mode, and the record of what runs. + +The classifier itself is in inbuxa-server (the `ai-spam-classification` +spec). It's off until an administrator turns it on in inbuxa Admin, under +Settings › Spam Filter › Local AI. The model's opinion is one bounded signal +among many (at most +2 points by default), and a slow or missing model never +holds up mail. + +## What runs + +| Piece | What | +|---|---| +| Server | llama.cpp `b11160`, the CPU build `llama-b11160-bin-ubuntu-x64.tar.gz` (SHA-256 `48ece242…e435`), in `/opt/llama/b11160` | +| Model | Qwen3 4B Instruct 2507 (Apache-2.0), quantized to Q4_K_M: `/opt/llama/models/qwen3-4b-instruct-2507-Q4_K_M.gguf`, SHA-256 `0f5e5250018ea2e4384e8b440f3a51fd1fff31d6694906e63fca3b5abd8a9a28` | +| Service | [`inbuxa-llm.service`](inbuxa-llm.service): in the mail network namespace, on its loopback `127.0.0.1:8080` only, 4 cores and 8 GB at most, 4 slots of 4096 tokens | + +The mail server's model entry (`x:AiModel`) points at +`http://127.0.0.1:8080/v1/chat/completions` with the model name +`qwen3-4b-instruct-2507`, which is the service's `--alias`. The address is +the mail namespace's own loopback, so message text never leaves the machine. + +Measured on production (host1, CPU only): about 1.2 s per message once warm, +4 s for the first; about 4.3 GB of memory. + +## The model, built from source + +Qwen publishes no GGUF of this model, and the popular ones are third-party +conversions. [`build-model.sh`](build-model.sh) makes it from Qwen's official +weights instead: it checks each file against Hugging Face's published +checksums, converts with llama.cpp's own converter, quantizes to Q4_K_M, and +compares the result with production's SHA-256. + +```bash +packaging/local-ai/build-model.sh ~/ai-models +``` + +## Installing it by hand + +On a Debian 13 host whose mail server runs in the `mail` namespace +(`/run/netns/mail`, from `mail-netns.service`): + +```bash +apt-get install libgomp1 # llama.cpp's OpenMP runtime +mkdir -p /opt/llama/models +curl -fLO https://github.com/ggml-org/llama.cpp/releases/download/b11160/llama-b11160-bin-ubuntu-x64.tar.gz +echo "48ece24283876fc3401b737724008c03cbc4c7ba335b6c1aa2a7b6ce2d49e435 llama-b11160-bin-ubuntu-x64.tar.gz" | sha256sum -c +mkdir -p /opt/llama/b11160 && tar -C /opt/llama/b11160 -xzf llama-b11160-bin-ubuntu-x64.tar.gz +install -m 0644 qwen3-4b-instruct-2507-Q4_K_M.gguf /opt/llama/models/ +install -m 0644 packaging/local-ai/inbuxa-llm.service /etc/systemd/system/ +systemctl daemon-reload && systemctl enable --now inbuxa-llm +ip netns exec mail curl -s http://127.0.0.1:8080/health # {"status":"ok"} +``` + +A host without the mail namespace drops `NetworkNamespacePath`, +`After=mail-netns.service` and `Requires=mail-netns.service` from the unit; +the service then listens on the host's loopback, which is where a mail server +on that host reaches it. + +## Undoing it + +`systemctl disable --now inbuxa-llm`. The classifier then fails fast on every +message and mail flows as before. Turn the classifier off in inbuxa Admin to +stop it trying. diff --git a/packaging/local-ai/build-model.sh b/packaging/local-ai/build-model.sh new file mode 100755 index 0000000..9047ce0 --- /dev/null +++ b/packaging/local-ai/build-model.sh @@ -0,0 +1,84 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Coffey Labs +# SPDX-License-Identifier: AGPL-3.0-or-later +# +# Build the local AI spam classifier's model from Qwen's official weights, so +# the file that runs is one you made and can check, not a third party's. +# +# packaging/local-ai/build-model.sh WORKDIR +# +# Downloads Qwen/Qwen3-4B-Instruct-2507 (Apache-2.0) and checks every file +# against the checksums Hugging Face publishes, converts it to GGUF with +# llama.cpp's own converter, and quantizes it to Q4_K_M, the quantization the +# classifier was calibrated with (inbuxa-server's ai-spam-classification +# spec, "Calibration"). Needs docker, curl, python3 and about 20 GB of disk. +# +# Qwen publishes no GGUF of this model; the popular ones are third-party +# conversions. Built with llama.cpp b11160, the result is byte-identical to +# the model production runs, whose SHA-256 is EXPECTED below. +set -euo pipefail + +W="${1:?usage: build-model.sh WORKDIR}" +LLAMA=b11160 +LLAMA_BIN_SHA=48ece24283876fc3401b737724008c03cbc4c7ba335b6c1aa2a7b6ce2d49e435 +MODEL=Qwen/Qwen3-4B-Instruct-2507 +NAME=qwen3-4b-instruct-2507 +EXPECTED=0f5e5250018ea2e4384e8b440f3a51fd1fff31d6694906e63fca3b5abd8a9a28 + +mkdir -p "$W/$NAME" "$W/llama-bin" +cd "$W" + +echo "== $MODEL, checked against Hugging Face's checksums" +curl -sf "https://huggingface.co/api/models/$MODEL/tree/main" > tree.json +python3 - "$MODEL" "$NAME" <<'PY' +import hashlib, json, subprocess, sys +model, out = sys.argv[1], sys.argv[2] +for f in json.load(open('tree.json')): + path = f['path'] + if f.get('type') != 'file' or path.startswith('.'): + continue + dest = f'{out}/{path}' + subprocess.run(['curl', '-sfL', '-o', dest, + f'https://huggingface.co/{model}/resolve/main/{path}'], check=True) + oid = (f.get('lfs') or {}).get('oid') + if oid: + h = hashlib.sha256() + with open(dest, 'rb') as fh: + for block in iter(lambda: fh.read(1 << 22), b''): + h.update(block) + if h.hexdigest() != oid: + sys.exit(f'checksum mismatch: {path}') + print(f' ok {path}') +PY + +echo "== llama.cpp $LLAMA: converter source and quantizer" +curl -sfL -o "llama-src-$LLAMA.tar.gz" "https://github.com/ggml-org/llama.cpp/archive/refs/tags/$LLAMA.tar.gz" +tar -xzf "llama-src-$LLAMA.tar.gz" +curl -sfL -o "llama-$LLAMA-bin-ubuntu-x64.tar.gz" \ + "https://github.com/ggml-org/llama.cpp/releases/download/$LLAMA/llama-$LLAMA-bin-ubuntu-x64.tar.gz" +echo "$LLAMA_BIN_SHA llama-$LLAMA-bin-ubuntu-x64.tar.gz" | sha256sum -c +tar -C llama-bin -xzf "llama-$LLAMA-bin-ubuntu-x64.tar.gz" + +echo "== convert to GGUF (f16)" +# Runs as the container's root: the converter's dependencies look up the +# user's name, and a host uid with no passwd entry breaks that. +docker run --rm -v "$W":/w -w /w python:3.12-slim sh -c " + python -m venv /tmp/v && + /tmp/v/bin/pip install -q -r llama.cpp-$LLAMA/requirements/requirements-convert_hf_to_gguf.txt \ + --extra-index-url https://download.pytorch.org/whl/cpu >/dev/null && + /tmp/v/bin/python llama.cpp-$LLAMA/convert_hf_to_gguf.py $NAME --outtype f16 --outfile $NAME-f16.gguf && + chown $(id -u):$(id -g) $NAME-f16.gguf" + +echo "== quantize to Q4_K_M" +Q=$(find llama-bin -name llama-quantize -type f | head -1) +LD_LIBRARY_PATH="$(dirname "$Q")" "$Q" "$NAME-f16.gguf" "$NAME-Q4_K_M.gguf" Q4_K_M >/dev/null + +GOT=$(sha256sum "$NAME-Q4_K_M.gguf" | cut -d' ' -f1) +echo "== $W/$NAME-Q4_K_M.gguf" +echo " sha256 $GOT" +if [ "$GOT" = "$EXPECTED" ]; then + echo " matches the model production runs" +else + echo " differs from production's ($EXPECTED): a different llama.cpp or upstream file change" >&2 + exit 1 +fi diff --git a/packaging/local-ai/inbuxa-llm.service b/packaging/local-ai/inbuxa-llm.service new file mode 100644 index 0000000..3c71426 --- /dev/null +++ b/packaging/local-ai/inbuxa-llm.service @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: 2026 Coffey Labs +# SPDX-License-Identifier: AGPL-3.0-or-later +# +# The local language model for inbuxa's AI spam classifier: llama.cpp's +# server with Qwen3 4B Instruct 2507 (Apache-2.0), converted from Qwen's +# official weights and quantized to Q4_K_M. It runs in the mail network +# namespace and listens only on its loopback, 127.0.0.1:8080, where the mail +# server's x:AiModel "local" points. Nothing leaves the machine. +# +# Bounded so it can never starve the mail server: 4 cores, 8 GB. Four slots +# match the classifier's default of four requests in flight (inbuxa:AiLimits +# maxConcurrentCalls), each with a 4096-token context. +[Unit] +Description=inbuxa local AI model (llama.cpp, Qwen3 4B Instruct 2507) +After=mail-netns.service +Requires=mail-netns.service + +[Service] +NetworkNamespacePath=/run/netns/mail +Environment=LD_LIBRARY_PATH=/opt/llama/b11160/llama-b11160 +ExecStart=/opt/llama/b11160/llama-b11160/llama-server \ + --model /opt/llama/models/qwen3-4b-instruct-2507-Q4_K_M.gguf \ + --alias qwen3-4b-instruct-2507 \ + --host 127.0.0.1 --port 8080 \ + --threads 4 --parallel 4 --ctx-size 16384 \ + --no-webui +DynamicUser=yes +CPUQuota=400% +MemoryMax=8G +Nice=10 +Restart=on-failure +RestartSec=10 +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +NoNewPrivileges=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictSUIDSGID=yes +LockPersonality=yes +CapabilityBoundingSet= +SystemCallArchitectures=native + +[Install] +WantedBy=multi-user.target