From 8d475e2b075d0f6b746c50ba6db79e46bf53132d Mon Sep 17 00:00:00 2001 From: John Coffey Date: Sun, 30 Aug 2026 13:56:12 -0700 Subject: [PATCH] Refuse to save a script we only partly read The transport fix stops the truncation that caused #76, but the save path had no answer for a baseline that arrives incomplete. It is neither unknown nor empty, so every existing guard passes it through: it parses into a shorter rule list that looks exactly like a script with fewer rules, and saving writes that back over the real one. Check the script against the shape the generator emits instead. Every rule comment parses, every enabled rule has an if and a closed body under it, every block ends with a blank line. Structural rather than a re-serialize-and-compare, so a script written by an older version whose serializer differed is still editable. The rule editor reports a short script as unreadable rather than showing the rules that happened to parse, since a list that looks complete over a script that is not is the most dangerous thing it could offer. A cut at the end of a complete rule block is still a valid shorter script and cannot be told apart from one; that residual is the proxy's to cover. --- KNOWN-ISSUES.md | 2 +- web/src/lib/sieve.ts | 94 +++++++++++++++++++ .../store/__tests__/sieve-overwrite.test.ts | 88 +++++++++++++++++ web/src/store/sieve.ts | 29 ++++-- web/src/views/mail/FilterFromMessage.tsx | 14 +-- web/src/views/settings/FiltersSettings.tsx | 15 ++- 6 files changed, 227 insertions(+), 15 deletions(-) diff --git a/KNOWN-ISSUES.md b/KNOWN-ISSUES.md index 4152fd4..cfde003 100644 --- a/KNOWN-ISSUES.md +++ b/KNOWN-ISSUES.md @@ -21,7 +21,7 @@ works the same way — and dropped where 0.15 was the whole subject. Support for 0.15 was removed on 2026-08-26; the last release that runs on it is tagged [`stalwart-0.15-support`](https://github.com/LINUXexpert-org/ihasmail/releases/tag/stalwart-0.15-support). -- **A compressing hop in front of Stalwart truncated every blob download, and nothing said so.** Node decompresses a gzip response before the code ever sees the body, but leaves the `content-length` header describing the *compressed* bytes. The blob proxy copied that header onto the longer body it forwarded, so the browser stopped reading exactly that many bytes in and called the download complete. Reported on [#76](https://github.com/LINUXexpert-org/ihasmail/issues/76) against a Coolify deployment, where Traefik's compress middleware only engages above 1 KiB: filter rules one and two were fine and the third pushed the script past the threshold, after which it came back cut off mid-rule — 384 bytes of a 1.3 KB script. The size threshold is what made it look like a race. This is the *second* cause behind that issue, and the first fix did not touch it: a truncated script is neither unknown nor empty, so the "refuse to save from a baseline we could not read" guard never fired — the script parsed, just with rules missing, and the next save wrote the short version back over the real one. Every blob download shared the fault, not just Sieve: message source, vCards, signature HTML, attachments being forwarded, and the `settings.json` sync. Settings degraded honestly by luck rather than design — a truncated file fails `JSON.parse`, which is caught and leaves the local cache in charge — so it stopped syncing between devices instead of being overwritten. The proxy now asks upstream for `identity` and, for a hop that compresses anyway, forwards no length at all rather than one describing different bytes. The image proxy is unaffected: it uses `node:http` directly, sends no `accept-encoding`, and never decompresses. +- **A compressing hop in front of Stalwart truncated every blob download, and nothing said so.** Node decompresses a gzip response before the code ever sees the body, but leaves the `content-length` header describing the *compressed* bytes. The blob proxy copied that header onto the longer body it forwarded, so the browser stopped reading exactly that many bytes in and called the download complete. Reported on [#76](https://github.com/LINUXexpert-org/ihasmail/issues/76) against a Coolify deployment, where Traefik's compress middleware only engages above 1 KiB: filter rules one and two were fine and the third pushed the script past the threshold, after which it came back cut off mid-rule — 384 bytes of a 1.3 KB script. The size threshold is what made it look like a race. This is the *second* cause behind that issue, and the first fix did not touch it: a truncated script is neither unknown nor empty, so the "refuse to save from a baseline we could not read" guard never fired — the script parsed, just with rules missing, and the next save wrote the short version back over the real one. Every blob download shared the fault, not just Sieve: message source, vCards, signature HTML, attachments being forwarded, and the `settings.json` sync. Settings degraded honestly by luck rather than design — a truncated file fails `JSON.parse`, which is caught and leaves the local cache in charge — so it stopped syncing between devices instead of being overwritten. The proxy now asks upstream for `identity` and, for a hop that compresses anyway, forwards no length at all rather than one describing different bytes. The image proxy is unaffected: it uses `node:http` directly, sends no `accept-encoding`, and never decompresses. The save path no longer trusts the transport either: a script is now checked for completeness against the shape the generator emits — every `# rule:` comment parses, every enabled rule has an `if` and a closed body below it, every block ends with a blank line — and saving refuses on anything short, as does the rule editor, which reports the script as unreadable rather than showing the rules that happened to parse. The check is structural rather than a re-serialize-and-compare, so a script written by an older version with a different serializer is still editable; refusing over a changed byte would be the worse bug. It catches a cut at every offset except the end of a complete rule block, which is a legitimately shorter script and indistinguishable from one in the bytes alone — that residual is what the proxy fix covers. - **Delete all spam destroys, and does not pass through Deleted Items** — this is the point of the feature and the thing worth checking on a real server, since a folder that empties into another folder has solved nothing. `Email/set destroy`, walked a page at a time so it survives `maxObjectsInSet` the way emptying Deleted Items already had to. **Confirmed live on 0.16.19 (2026-08-26)**: Junk Mail emptied and Deleted Items stayed empty afterwards. There is no undo, which is why all three entry points share one dialog that says so. Only Deleted Items and Junk Mail can be emptied this way, enforced in the store rather than only hidden in the menus. - **Sharing a mail folder is accepted and does nothing.** `Mailbox/set` with a `shareWith` map is applied, `Mailbox/get` reads it back, and the folder never appears for the account it was shared with — **confirmed live on 0.16.19 (2026-08-27)** with a folder shared read-only to another account on the same server, which never saw it. Stalwart's own sharing documentation lists calendars, address books and file storage; mail folders are not among them. Nothing reports a failure at any point, which is the whole problem: the share is stored, so a client that trusts what it reads back shows it as live for ever. The entry point is withdrawn. A folder that is *already* shared still offers **Stop sharing**, because a share nobody can see is exactly the one you want to be able to clear, and there is no other way to. File sharing is unaffected and works end to end. diff --git a/web/src/lib/sieve.ts b/web/src/lib/sieve.ts index 5c0417f..f70176c 100644 --- a/web/src/lib/sieve.ts +++ b/web/src/lib/sieve.ts @@ -174,6 +174,100 @@ export function rulesToSieve(rules: SieveRule[]): string { return lines.join("\n"); } +/** + * Why this script must not be rewritten from the rules parsed out of it, or + * null when rewriting it is safe. + * + * Saving replaces the whole script with a fresh serialization of the rules read + * out of it, so whatever was not read is deleted. `sieveToRules` cannot raise + * the alarm by itself: it skips what it does not recognise, so a script cut off + * partway through parses cleanly into a shorter list and looks exactly like one + * that genuinely has fewer rules. That is the shape of the loss in #76 -- a + * truncated download, a plausible parse, and a save that wrote the short + * version back over the real one. The transport fault behind it is fixed in the + * blob proxy; this is the check that makes the save path refuse regardless of + * how the content came to be short. + * + * The tests are structural rather than an equality check against + * `rulesToSieve(sieveToRules(content))`. A script written by an older version + * whose serializer differed in some detail is intact, and refusing to let + * anyone edit their rules over a changed byte would be the worse bug. + */ +export function scriptDamage(content: string): string | null { + // Not one of ours: genuinely empty, or hand-written. Both mean something + // else and are answered elsewhere. + if (!content.includes("# rule:") && !content.includes(SCRIPT_HEADER)) { + // Unless it is one of ours cut off inside its own first line, which reads + // as a very short hand-written script -- and that reading is the one that + // offers to replace it. + const head = content.replace(/\n+$/, ""); + if (head !== "" && SCRIPT_HEADER.startsWith(head)) return "breaks off inside its first line"; + return null; + } + // Every generated script ends with a newline, so a body that stops mid-line + // stopped early. The rest of the walk covers the cuts that land on one. + if (!content.endsWith("\n")) return "stops in the middle of a line"; + + const lines = content.replace(/\r\n/g, "\n").split("\n"); + let i = 0; + let hadRequire = false; + if (lines[0] === SCRIPT_HEADER) { + i = 1; + hadRequire = lines[i]?.startsWith("require ") ?? false; + if (hadRequire) i++; + if (lines[i] !== "") return "breaks off in its opening lines"; + i++; + } else { + // Header edited away but the rule comments kept. Still ours to walk. + i = lines.findIndex((l) => l.startsWith("# rule:")); + } + + // Walk the shape rulesToSieve emits, one rule block at a time. Deliberately + // structural: the condition and action lines are read only for their + // presence, so a serializer that words them differently is still intact. + let seen = 0; + const cut = (r: SieveRule, what: string) => `has a rule in it (“${r.name}”) ${what}`; + while (i < lines.length) { + const line = lines[i]!; + if (!line.startsWith("# rule:")) return "has a stray line where a rule should start"; + let rule: SieveRule | null = null; + try { + const parsed = JSON.parse(line.slice(7)) as SieveRule; + if (parsed && typeof parsed === "object" && Array.isArray(parsed.tests) && Array.isArray(parsed.actions)) rule = parsed; + } catch { + /* reported below */ + } + if (!rule) return "has a rule in it that breaks off unfinished"; + seen++; + i++; + // `!r.enabled` is how rulesToSieve chooses the branch, so match it exactly + // rather than testing for `=== false`. + if (rule.enabled) { + if (!lines[i]?.startsWith("if ")) return cut(rule, "with nothing below it"); + i++; + if (lines[i] !== "{") return cut(rule, "whose body never opens"); + i++; + while (i < lines.length && lines[i] !== "}") { + if (lines[i] === "") return cut(rule, "whose body breaks off"); + i++; + } + if (i >= lines.length) return cut(rule, "whose body never closes"); + i++; + } else { + if (!lines[i]?.startsWith("# (disabled) ")) return cut(rule, "with nothing below it"); + i++; + } + // Each block is followed by a blank line, the last one included: it is the + // empty final element left by the trailing newline. + if (lines[i] !== "") return cut(rule, "that runs into what follows it"); + i++; + } + // A require line is written only for rules that need it, so one standing over + // no rules at all means the rules it was written for are gone. + if (hadRequire && seen === 0) return "breaks off before the first rule"; + return null; +} + /** Returns rules if the script was generated by ihasmail, else null (raw script). */ export function sieveToRules(script: string): SieveRule[] | null { if (!script.includes("# rule:")) return script.trim() === "" || script.includes(SCRIPT_HEADER) ? [] : null; diff --git a/web/src/store/__tests__/sieve-overwrite.test.ts b/web/src/store/__tests__/sieve-overwrite.test.ts index 12917e9..29a7797 100644 --- a/web/src/store/__tests__/sieve-overwrite.test.ts +++ b/web/src/store/__tests__/sieve-overwrite.test.ts @@ -83,3 +83,91 @@ describe("reloading", () => { expect(useSieve.getState().rules().rules).toHaveLength(3); }); }); + +/** + * Issue #76, second round. The transport fault is fixed in the blob proxy, but + * the save path had no answer for a script that arrives *partly* read: it is + * neither unknown nor empty, so the guards above all pass it through. It parses + * into a shorter rule list that looks exactly like a script with fewer rules, + * and saving writes that shorter version back over the real one. + * + * These pin the third state: read, but not all of it. + */ +describe("a script that was only partly read", () => { + /** Cut at 384 bytes, the way a compressing hop cut the reporter's script. */ + const truncate = (content: string, at: number) => content.slice(0, at); + const full = rulesToSieve(threeRules); + + it("reports its rules as unknown rather than handing back the ones that parsed", () => { + useSieve.setState({ contents: { s1: truncate(full, 384) } }); + const { rules, loaded, damage } = useSieve.getState().rules(); + expect(rules).toBeNull(); + expect(loaded).toBe(true); + expect(damage).toBeTruthy(); + }); + + it("refuses to save over the part it never saw", async () => { + useSieve.setState({ contents: { s1: truncate(full, 384) } }); + await expect(useSieve.getState().saveRules([newRule({ name: "New" })])).rejects.toThrow(/overwrite the rest of it/i); + }); + + it("catches a cut at every offset through the script, not just a lucky one", () => { + // The offsets that cannot be caught are the ends of complete rule blocks: + // each is a valid shorter script and nothing in the bytes says otherwise. + // That is the residual the proxy fix covers and this check cannot. + const safe = new Set(); + for (let n = 0; n <= threeRules.length; n++) safe.add(rulesToSieve(threeRules.slice(0, n)).length); + let missed = 0; + for (let at = 1; at < full.length; at++) { + useSieve.setState({ contents: { s1: truncate(full, at) } }); + const { damage } = useSieve.getState().rules(); + if (!damage && !safe.has(at)) missed++; + } + expect(missed).toBe(0); + }); + + it("leaves an intact script alone at every length it can legitimately have", () => { + for (let n = 0; n <= threeRules.length; n++) { + useSieve.setState({ contents: { s1: rulesToSieve(threeRules.slice(0, n)) } }); + const { rules, damage } = useSieve.getState().rules(); + expect(damage).toBeNull(); + expect(rules).toHaveLength(n); + } + }); + + it("leaves the shapes a rule can take alone — disabled, many actions, extensions", () => { + // A false positive here costs someone the use of the rules editor, so the + // walk has to pass everything rulesToSieve can legitimately produce. + const varied = [ + newRule({ name: "Disabled", enabled: false }), + newRule({ name: "Many actions", actions: [{ type: "fileinto", mailbox: "A" }, { type: "markread" }, { type: "flag" }, { type: "stop" }] }), + newRule({ name: "Two tests", join: "anyof", tests: [{ type: "body", op: "contains", value: "x" }, { type: "size", op: "over", value: 1024 }] }), + newRule({ name: "No actions at all", actions: [] }), + newRule({ name: "Quotes \" and \\ backslash" }), + ]; + useSieve.setState({ contents: { s1: rulesToSieve(varied) } }); + const { rules, damage } = useSieve.getState().rules(); + expect(damage).toBeNull(); + expect(rules).toHaveLength(varied.length); + }); + + it("catches a cut at every offset through that script too", () => { + const varied = [newRule({ name: "Disabled", enabled: false }), newRule({ name: "Live" }), newRule({ name: "Also off", enabled: false })]; + const full = rulesToSieve(varied); + const safe = new Set(); + for (let n = 0; n <= varied.length; n++) safe.add(rulesToSieve(varied.slice(0, n)).length); + let missed = 0; + for (let at = 1; at < full.length; at++) { + useSieve.setState({ contents: { s1: full.slice(0, at) } }); + if (!useSieve.getState().rules().damage && !safe.has(at)) missed++; + } + expect(missed).toBe(0); + }); + + it("does not call a hand-written script damaged", () => { + useSieve.setState({ contents: { s1: 'require ["fileinto"];\nif header :contains "from" "x" { fileinto "X"; }' } }); + const { rules, damage } = useSieve.getState().rules(); + expect(damage).toBeNull(); + expect(rules).toBeNull(); // hand-written, which is a different refusal + }); +}); diff --git a/web/src/store/sieve.ts b/web/src/store/sieve.ts index 7a2b5f3..37efa8d 100644 --- a/web/src/store/sieve.ts +++ b/web/src/store/sieve.ts @@ -1,7 +1,7 @@ import { create } from "zustand"; import { CAP, client, setErrorMessage } from "@/jmap/client"; import type { GetResponse, Id, SetResponse, SieveScript } from "@/jmap/types"; -import { rulesToSieve, sieveToRules, type SieveRule } from "@/lib/sieve"; +import { rulesToSieve, scriptDamage, sieveToRules, type SieveRule } from "@/lib/sieve"; import { useSession } from "./session"; export const IHASMAIL_SCRIPT = "ihasmail"; @@ -19,7 +19,7 @@ interface SieveState { getContent(id: Id): Promise; /** Rules derived from the "ihasmail" script (null = the active script is hand-written). */ /** `loaded` distinguishes "this script is hand-written" from "we could not read it". */ - rules(): { script: SieveScript | null; rules: SieveRule[] | null; content: string; loaded: boolean }; + rules(): { script: SieveScript | null; rules: SieveRule[] | null; content: string; loaded: boolean; damage: string | null }; saveRules(rules: SieveRule[]): Promise; saveScript(id: Id | null, name: string, content: string, activate: boolean): Promise; activate(id: Id | null): Promise; @@ -91,14 +91,19 @@ export const useSieve = create((set, get) => ({ rules() { const { scripts, contents } = get(); const script = scripts.find((s) => s.name === IHASMAIL_SCRIPT) ?? scripts.find((s) => s.isActive) ?? null; - if (!script) return { script: null, rules: [], content: "", loaded: true }; + if (!script) return { script: null, rules: [], content: "", loaded: true, damage: null }; const content = contents[script.id]; // Not loaded, or the fetch failed. `null` means "cannot say", which every // caller already treats as "do not edit this script" -- as opposed to `[]`, // which means "this script genuinely has no rules" and invites a save that // would overwrite whatever is really in it. - if (content === undefined) return { script, rules: null, content: "", loaded: false }; - return { script, rules: sieveToRules(content), content, loaded: true }; + if (content === undefined) return { script, rules: null, content: "", loaded: false, damage: null }; + // Read, but not all of it. Showing the rules that did parse would be the + // most dangerous thing available: a short list that looks complete, over a + // script that is not. Say "cannot say" here too. + const damage = scriptDamage(content); + if (damage) return { script, rules: null, content, loaded: true, damage }; + return { script, rules: sieveToRules(content), content, loaded: true, damage: null }; }, async saveRules(rules) { @@ -106,8 +111,18 @@ export const useSieve = create((set, get) => ({ // The last line of defence. Writing rules replaces the whole script, so // doing it from a baseline we never managed to read deletes whatever was // there. Refusing is recoverable; overwriting is not. - if (existing && get().contents[existing.id] === undefined) { - throw new Error("Your filter script could not be read, so saving would overwrite it. Reload and try again."); + if (existing) { + const content = get().contents[existing.id]; + if (content === undefined) { + throw new Error("Your filter script could not be read, so saving would overwrite it. Reload and try again."); + } + // Read in full is a separate question from read at all, and the answer + // that cost rules in #76 was "partly". A baseline missing its tail writes + // out just as confidently as one missing entirely. + const damage = scriptDamage(content); + if (damage) { + throw new Error(`Your filter script ${damage}, so saving would overwrite the rest of it. Reload and try again.`); + } } await get().saveScript(existing?.id ?? null, IHASMAIL_SCRIPT, rulesToSieve(rules), true); }, diff --git a/web/src/views/mail/FilterFromMessage.tsx b/web/src/views/mail/FilterFromMessage.tsx index e293470..46fa8ba 100644 --- a/web/src/views/mail/FilterFromMessage.tsx +++ b/web/src/views/mail/FilterFromMessage.tsx @@ -33,17 +33,19 @@ export function FilterFromMessageDialog({ email, mailboxId, onClose }: { email: } if (!ready) return ; - const { rules, loaded } = sieve.rules(); + const { rules, loaded, damage } = sieve.rules(); if (rules === null) { return ( Close}> {/* - Two different situations, and telling them apart matters: one is - permanent and one is a reload away. Saying "written by hand" when the - script merely failed to fetch sends someone looking for a problem - they do not have. + Three different situations, and telling them apart matters: one is + permanent and two are a reload away. Saying "written by hand" when the + script merely failed to fetch -- or arrived in part -- sends someone + looking for a problem they do not have. */} - {loaded ? ( + {damage ? ( +

Your filter script {damage}, so only part of it arrived. Adding a rule would write that part back over the whole thing. Reload the page and try again.

+ ) : loaded ? (

Your active Sieve script was written by hand, so rules can't be added automatically. Open Settings → Filters & rules to edit the script or switch to managed rules.

) : (

Your filter script couldn't be read just now, so adding a rule would risk overwriting it. Reload the page and try again.

diff --git a/web/src/views/settings/FiltersSettings.tsx b/web/src/views/settings/FiltersSettings.tsx index e5ff16f..828173f 100644 --- a/web/src/views/settings/FiltersSettings.tsx +++ b/web/src/views/settings/FiltersSettings.tsx @@ -45,7 +45,7 @@ const RULE_MIME = "application/x-ihasmail-sieve-rule"; function RulesEditor() { const sieve = useSieve(); - const { script, rules, content } = sieve.rules(); + const { script, rules, content, damage } = sieve.rules(); const [local, setLocal] = useState(null); const [editing, setEditing] = useState(null); const [saving, setSaving] = useState(false); @@ -80,6 +80,19 @@ function RulesEditor() { } }; + // Before the hand-written branch: a script that arrived in part is not a + // script someone chose to write themselves, and the way out of it is a reload + // rather than the "start with rules" button below, which would write over it. + if (damage) { + return ( +
+
Only part of your filter script arrived.
+

It {damage}, so the rules in it can't be shown or edited — saving what did arrive would write it back over the rest. Reload the page to try again. Your rules are still on the server; nothing here has changed them.

+ +
+ ); + } + if (rules === null) { return (