Field Notes · kept as written, during the work

Potluck — The Field Notes

1,042 lines written during the build by six hands — the Orchestrator and five lane agents — appended, never edited. This is the raw material behind the guide and the post-mortem.

Back to the guide: /tutorials/potluck · These notes are append-only and unpolished by design — every surprise, wrong turn, and pushback, recorded at the moment it happened.

_Append-only, timestamped, unpolished. Every surprise, pushback, and reality-vs-plan discrepancy lands here as it happens — these become the brass callouts in Field Guide No. 4. All six windows write here; append at the bottom, never rewrite someone else's note._

Pre-seeded slots the guide is already waiting on:

What the Orchestrator challenged in PLAN.md before contracts committed
The actual Instagram export structure vs. the plan's expectation
Eval scores per prompt revision; caption patterns that fooled early versions
The waterfall simplification (recipe-scrapers is Python → two stages)

---

2026-09-03 — scaffold

Repo bootstrapped from FrameWork's lead session: Next.js shell, Drizzle config, worktree script, prompt pack. Contracts deliberately NOT pre-written — that's the Orchestrator window's first act.
Plan-level deviations from the original RecipeVault doc are recorded in PLAN.md §0 (name, two-stage waterfall, local-only v1, Flow B deferred).

2026-09-03 — Phase 0, Orchestrator (contracts)

Context loss is real. The Orchestrator window lost its conversation between the scaffold commit and the contracts. Nothing was on disk, so the work restarted from PLAN.md. Lesson for the guide: the plan file and STATUS.md are the memory; anything decided in chat and not written down did not happen. From here every decision lands in PLAN.md §0 the moment it is made.
Alignment check vs. the original RecipeVault doc: the repo's PLAN.md is a faithful evolution. Every divergence (name, Drizzle+npm, two-stage waterfall, Sonnet 5, shared password, drain script instead of a worker, Flow B + PWA deferred) was already recorded in §0. No silent drift.
"Temperature 0" met the API. The plan specified temperature 0 for the extraction call. claude-sonnet-5 rejects temperature/top_p/top_k with a 400. Determinism now comes from structured outputs against the Zod schema in the contracts, which is arguably the better tool for the job anyway — the plan was written against an older API surface. Also learned: the API takes a PDF directly as a document block (no beta, 600 pages), which may delete the render-pages-to-images step from the PDF branch. Left as Extract's call.
int4range met Drizzle. No native range type, and the house rule is never hand-write migration SQL. PageRange becomes two int columns.
package.json was a collision waiting to happen. Four prompts promised scripts (db:seed, drain, test, eval) that would each have meant a different agent editing package.json — against the one-writer rule. Fixed by pre-declaring every script in the contracts commit and making the file Orchestrator-owned. Same for vitest.config.ts.
Worktrees share history, not node_modules or .env.local. Obvious in hindsight; scripts/worktrees.sh up now copies the env file and runs npm install per tree. Would have cost each agent its first ten minutes.
Pushed back on the extraction prompt's framing. It named "web page text or Instagram captions" but PDF Pass 2 reuses it. Added cookbook-page guidance (ignore page numbers/running headers; extract the first complete recipe) rather than leaving the model to guess.
Added pending_imports.raw_source so a retry re-extracts from stored text. The plan only kept raw text on the recipe row, which meant a no_recipe outcome had nowhere to keep the caption — contradicting "never silently drop a save".

2026-09-03 — Phase 1, QA (fixtures, eval harness, seam tests)

First tests landed: 180 (148 run, 32 guarded-skip) under npm run check. passWithNoTests: true in vitest.config.mts can come off now.
npm run eval can never call Claude. package.json has "eval": "SKIP_API_TESTS=1 vitest run tests/eval", but contracts/api.md's scripts table says vitest run tests/eval, and 13 of 16 fixtures need the API (every caption; JSON-LD-absent/malformed/non-recipe HTML). As wired, the gate number for Extract's merge would come from 3 JSON-LD fixtures only. An inline env assignment in the script cannot be overridden from the shell, so there is no workaround without editing package.json. (Request below.)
Vitest 5's default reporter hides console output from passing tests. The eval report is written to process.stdout directly so a plain npm run eval shows it; the token-zone scan summary still only appears under --reporter=verbose. Worth a line in the guide: "your test printed nothing" is a reporter setting, not a bug.
FixtureMeta is a TS interface, not a Zod schema, so QA defines FixtureMetaSchema (strict) in tests/fixtures/loader.ts with a compile-time both-ways assignability check against the contract. If Extract wants to validate meta.json too, it belongs in contracts.
PDF fixtures are not runnable in the contracted format. The harness would have to turn input.pdf into PdfPage[] — page splitting is Ingest's job with a library nobody has picked yet. No pdf fixtures shipped. Proposal: a pdf fixture carries pages.json (PdfPage[] without image bytes) next to input.pdf, so the harness can call extractFromPdfPages offline and the PDF itself stays as the audit copy. Needs an Orchestrator decision.
The token-zone write check is transitive by design. The static scan walks every import reachable from app/s/**; any reachable module with a Drizzle .insert/.update/.delete( chain or raw INSERT/UPDATE/DELETE SQL fails it. Practical consequence for UI: keep read helpers in modules that contain no writes (app/lib/read.ts vs app/lib/admin.ts, or similar). Surfacing now so it is a design choice, not a merge-day surprise.
Server-level seam tests need a running app. /s/:token pages are server components and route handlers assume Next's request context, so invariants 1–5 at the route level are driven over HTTP against POTLUCK_BASE_URL with ADMIN_PASSWORD for the cookie; database-level checks (UNIQUE, partial index, status CHECK, nullable columns) need only DATABASE_URL. Vitest does not load .env.local, so those variables must be exported in the shell for the tests to light up. The drain test is opt-in (RUN_DRAIN_TEST=1) because at 4/min it takes ~2 minutes.
Drain scope assumption. The drain test queues source_type: 'web' rows pointing at a local fixture server. The Ingest prompt describes the drain as fetching "public post pages" (Instagram). If the drain only picks up instagram rows, this test will time out — Ingest should confirm the drain processes any queued row, which is what PLAN.md §4 step 4 implies.
Eval rubric (tests/eval/scoring.ts): recipe_found is a gate (mismatch = 0 for the fixture); then title 20 · ingredient_count 10 · ingredient_overlap 25 (F1 over fuzzy item matches) · step_count 15 · route 15 · metadata 15 (servings/prep/cook). Quantities, units, notes, tags, description and step wording are deliberately unscored — they vary legitimately and would turn the number into a style test. Stage (jsonld/claude) is reported, not scored. The confidence values in expected.json are QA's priors; route agreement will be the noisiest component, and PLAN.md §0 already expects thresholds to be tuned after the first real run. Harness self-check: `EVAL_EXTRACTOR=@/tests/eval/echo-extractor npm run eval` scores 16/16 with one fixture deliberately degraded.
Fixture counts: 10 captions (hashtag wall, emoji dividers, link-in-bio, implied steps, German/English metric, Spanish/English bilingual, giveaway, vague quantities → needs_review band, follow-for-more banner, restaurant review), 6 HTML (JSON-LD simple, @graph + HowToSection, @type array + string instructions, absent, malformed, valid-but-no-Recipe), 1 Instagram export. All handwritten and scrubbed — no live fetches.

Requests for Orchestrator

package.json: change eval to vitest run tests/eval (no SKIP_API_TESTS) so the merge-gate number includes Claude fixtures, and optionally add "eval:offline": "SKIP_API_TESTS=1 vitest run tests/eval".
vitest.config.mts: drop passWithNoTests (first tests landed).
Decide the pdf fixture format (proposal above: pages.json sidecar).
.env.local.example: document optional POTLUCK_BASE_URL (running dev server) for the server-level seam tests; and decide whether vitest.config.mts should load .env.local so DATABASE_URL reaches the DB-level tests under npm run check on a machine with Postgres.

2026-09-03 — build method changed: one window, five subagents

The plan assumed Tommy drives five Claude Code windows by hand. Tommy's call after Phase 0: "I do not want to try to control 5 agents." The Orchestrator now spawns each lane as a background subagent pointed at its worktree, and Tommy coordinates with the Orchestrator alone. Worktrees, branches, lanes, gates, and merge order are unchanged — only who types the opening prompt. Worth teaching: the contract-first structure is what made this swap free; nothing about the plan depended on humans in the loop per lane.
Sequencing consequence: Ingest and UI need the real Drizzle schema and DB client on main before they can write anything real, so they start after the Schema merge (wave 2). Schema, Extract, QA run now (wave 1).
Predicted merge wrinkle: every agent appends to this file on its own branch, so every merge will conflict here. Resolution rule: keep every note, in branch-merge order. Never drop a note to resolve a conflict.

2026-09-04 — Phase 1, Schema agent

drizzle-kit parameterizes sql\${value}\`` inside CHECK DDL. First db:generate emitted CHECK (source_type IN ($1, $2, $3, $4)) — bound placeholders in a migration file, which Postgres would reject at migrate time. Literals inside a CHECK have to go through sql.raw(...). Caught by reading the generated SQL before running it; worth a callout because the generate step reports success either way.
Enum columns are text + CHECK, not pgEnum. Both derive from the contract const arrays, so one list still drives everything. Native enums were rejected because adding a value later means ALTER TYPE ... ADD VALUE in a hand-edited migration, which the house rule forbids; a CHECK is just a regenerated diff.
PLAN.md §2 leaves FK delete behaviour unstated. Decided: ingredients, steps, recipe_tags, and share_links.recipe_id cascade (a recipe-scope link dies with its recipe, so DELETE /api/recipes/:id can succeed); pending_imports.recipe_id is SET NULL (queue rows are never deleted — api.md invariant — so they must outlive the recipe); the two import_files FKs are plain (files are never deleted by the pipeline).
Added CHECKs the contracts imply but §2 doesn't state: share link recipe_id IS NULL exactly when scope = 'collection'; both halves of a split page range null together and from <= to; confidence in 0..1; tags.name lowercase. Each is a doc comment in contracts/types.ts turned into a constraint so the DB, not the reviewer, enforces it.
The Drizzle row is not the contract row. RecipeRow carries source_page_from/to and Date; contract Recipe carries source_pages: PageRange and ISO strings. Somebody needs a row → contract mapper before RecipeWithParts can be served. See request below. A compile-time key-drift guard at the bottom of core/db/schema.ts fails typecheck if the contracts grow a field the schema doesn't have.
Neither drizzle-kit nor tsx loads .env.local. Every DB script worked only because drizzle.config.ts and core/db/client.ts fall back to postgres://localhost:5432/potluck. Fine for v1 local-only; the deploy milestone will need node --env-file or an explicit dotenv load.
Seed share tokens are deterministic (printed at the end of db:seed) so QA and UI can hit /s/<token> without querying first. Real tokens come from crypto.randomBytes per api.md; nothing in the seed pretends otherwise.

Requests for Orchestrator (Schema)

Decide who owns the row → contract mapper (RecipeRow → Recipe: join source_page_from/to into source_pages, Date → ISO string; same for PendingImportRow). Schema can add core/db/map.ts in Phase 1 if you'd rather Ingest and UI not each write one.
Optional convenience script: "db:reset": "dropdb --if-exists potluck && createdb potluck && drizzle-kit migrate && tsx core/db/seed.ts". Not needed for the gate; it is what I ran by hand.

2026-09-04 — QA after the Schema merge

DB-level seam tests now run for real under npm run check (no env needed on a machine with the local potluck db migrated): the QA env helper mirrors core/db/client.ts's fallback to postgres://localhost:5432/potluck. Every write happens in a Drizzle transaction that is rolled back (savepoints around statements expected to fail), so the seed is untouched. Set DATABASE_URL="" to force a skip. Request 4 in the list above (load .env.local in vitest) is therefore moot for DATABASE_URL; it still applies to POTLUCK_BASE_URL/ADMIN_PASSWORD for the server-level tests.
**Invariant 2 ("status only moves forward") cannot be proven at the DB level** — there is no trigger, the CHECK only bounds the value set. The runtime proof (observed status sequence over HTTP) waits for Ingest. What runs now: the CHECK, the default queued, and every row at rest obeying invariant 3 (failed/no_recipe carry error, no_recipe carries raw_source, done carries recipe_id + raw_source, terminal rows are stamped) — which doubles as a check that the seed obeys the contract.
Seed token length. core/db/seed.ts says its tokens are "43-char base64url-shaped"; they are 45 chars. Harmless for the DB (text column), but if UI validates /s/:token by exact length before the lookup, the seeded family link 404s. QA's at-rest check tests charset + the 128-bit floor, not exact length. Schema or UI should pick one.
PDF fixtures unblocked by the PdfLoader contract. With Extract exporting loadPdfPages, the harness loads input.pdf through it and runs extractFromPdfPages over the full range — so the pages.json sidecar proposal above is withdrawn (request 3 closed). First pdf fixture landed: a one-page text-layer PDF assembled by hand (no PDF library needed to write it; verified to parse with pdfjs-dist, 946 chars of text). 17 fixtures total: 10 caption · 6 html · 1 pdf.
Deleting a recipe keeps its queue row (recipe_id → SET NULL) and kills its recipe-scope share links (CASCADE) but not collection links — both now asserted. Matches "the pipeline never deletes a queue row".

2026-09-04 — Schema follow-up: mappers

npx vitest run core/db finds nothing. A CLI path filter narrows the root include (tests/**/*.test.ts); it does not widen it. The mapper tests run through a Schema-owned core/db/vitest.config.mts instead: npx vitest run --config core/db/vitest.config.mts. They are therefore not part of npm run check yet.
Insert helpers are per-table (toRecipePageColumns, toImportPageColumns) on top of the neutral fromPageRange, because the split columns have different names on recipes (source_page_*) and pending_imports (page_*). Ingest spreads the helper's result into .values().

Requests for Orchestrator (Schema, round 2)

Add "core//__tests__//*.test.ts" to test.include in the root vitest.config.mts so the mapper tests join npm run check. Then core/db/vitest.config.mts can be deleted.

2026-09-03 — Phase 2, Extract agent

Structured outputs, as promised — with one surprise in the SDK. @anthropic-ai/sdk 0.123 has client.messages.parse + zodOutputFormat, so the extraction call is grammar-constrained to ExtractionResultSchema and there is no JSON scraping. But the SDK's transformJSONSchema strips everything the API grammar doesn't support into the property *description* — and that includes enum. The unit enum reaches the model as a description hint, not a hard constraint; the contract's .catch(null) is the backstop. Live smoke test: units came back normalized (tbsp, tsp, lb, clove) anyway — the system prompt lists them too. Same story for max(5) tags, min(1) strings, and 0–1 confidence: Zod validates them client-side after the grammar, so a violation throws from parse and the one repair retry re-sends the input with the validation issues appended. Never triggered across six live calls.
cache_control on the system prompt is a no-op today. The extraction prompt is ~450 tokens; Sonnet 5's minimum cacheable prefix is well above that, so the cache silently doesn't engage. Left in place per the contract (harmless, and it starts paying the day the prompt grows or few-shot examples land). Worth knowing before anyone stares at cache_read_input_tokens: 0 and thinks something is broken.
**PDF library: pdfjs-dist (Mozilla PDF.js, Apache-2.0), one dependency for both halves of the PDF branch.** Per-page text runs with positions — needed, because pdfjs emits word-level fragments and the lines have to be rebuilt by baseline before the prompt sees them — and rasterizing scanned pages to PNG through its optional @napi-rs/canvas peer (prebuilt binaries, no node-gyp). Rejected: pdf-parse (text only, old pdfjs inside), unpdf (a wrapper over pdfjs — a middleman), mupdf (WASM, does both, but AGPL), pdf-lib (writes PDFs, doesn't read text). Node needs the legacy/build/pdf.mjs entry.
Scans: page images, not PDF document blocks. The API does take a PDF as a base64 document block, and it looked like it would delete the render step. It doesn't fit the contract: PdfPage carries text-or-PNG per page and extractFromPdfPages gets a page *range*, so a document block would mean slicing a new PDF per segment (another library) or re-sending the whole file for every recipe in a forty-recipe scan. Page images keep Pass 1 and Pass 2 on one code path (pagesToBlocks), keep text and scanned pages mixable in one call, and cost the same tokens. Segmentation windows at 80 pages per call to stay under the per-request image cap for big scanned cookbooks.
Scanned pages have no raw text to be sacred about. raw_source for a vision segment labels each scanned page `[scanned image: no text layer; see the stored PDF]; the file under storage/pdfs/` is the source of record. Ingest should not treat that marker as extractable text on retry.
Bug caught by my own fixture, not by Claude: the JSON-LD path split every string on commas (right for keywords: "a, b, c", wrong for "3 cloves garlic, smashed", which became two ingredients). Ingredient strings now split on newlines only.
Instagram pages via extractFromHtml: the caption lives in og:description shaped `"1,204 likes, 33 comments - handle on March 3, 2026: "…"; the handle is parsed out as source_author`. Handwritten fixture only — no live Instagram fetch anywhere in tests.
Runner quirks: the root vitest config only includes tests/** (QA's lane), so Extract's smoke tests run via npx vitest run --config core/extract/vitest.config.mts. And tsx compiles .ts as CommonJS here (no "type": "module"), so top-level await needs .mts — trivia that cost ten minutes.
Live smoke (6 calls, all green): handwritten smashed-potatoes caption → auto_save (confidence 0.82) with 7 ingredients / 6 steps, hashtags and "follow for more" ignored; "link in bio" caption → recipe_found: false; 3-page text PDF → segmentation found exactly the two recipes (skipped the contents page) and Pass 2 got servings/prep/cook right; 1-page scan → vision read all 8 ingredients. ~7 s per call.

Requests for Orchestrator (Extract)

Add the PDF dependency on main (Extract developed against it with --no-save; package.json untouched): npm install pdfjs-dist@^6.3.289 (@napi-rs/canvas comes along as its optional dependency — nothing else.)
next.config.mjs: add serverExternalPackages: ["pdfjs-dist", "@napi-rs/canvas"] so the route handlers load pdfjs from node_modules instead of trying to bundle its worker and the native canvas binary.
Contract addition proposal: core/extract/index.ts also exports loadPdfPages(bytes, opts?) → { pages: PdfPage[], page_count, has_text_layer }. Ingest needs exactly those three things after a PDF upload and should not grow its own pdfjs code; suggest adding it to the Extractor surface (or blessing the named export in contracts/api.md).
The eval harness should run Extract's offline fixtures too; the two tiny PDFs under core/extract/__tests__/fixtures/ are free for QA to copy into fixtures/pdf/.

2026-09-04 — Phase 2 gate, Orchestrator

First real eval run on main: 93.6% over 17 fixtures (PASS ≥ 90). caption 90.0% (10), html 98.5% (6), pdf 100% (1). All three JSON-LD fixtures hit stage 1 and scored 100; every non-recipe fixture routed to no_recipe correctly. Total run 101 s, 14 live calls.
What lost points, in order: step granularity (Claude splits "mix, then bake" into more steps than the fixture author did — 3 fixtures), title embellishment ("Shakshuka" vs "Shakshuka for a Grey Sunday" — the model keeps the caption's full title, the fixture wanted the dish), one route miss (vague-quantities chili: model confidence 0.45 → no_recipe, fixture expected 0.62 → needs_review). No prompt change yet; the rubric's confidence priors are QA's guesses and the first thing to recalibrate is the fixture, not the model.
Vitest now loads .env.local via loadEnv in the root config with no prefix filter, so npm run eval and the live seam tests find ANTHROPIC_API_KEY / DATABASE_URL / ADMIN_PASSWORD without shell exports. Closes the note from both Schema and QA.
eval runs live; eval:offline is the free variant. The previous wiring made the gate number unreachable (QA caught it).

2026-09-04 — Phase 1, UI agent

**Next 16 renamed middleware to proxy.ts, and it must sit at the repo root — not under app/. That's outside UI's stated lane (app/, components/**), but Next allows exactly one such file at exactly that path, so UI owns proxy.ts by necessity. The cookie check it calls lives in app/lib/auth.ts; the proxy is a thin matcher over it. Proxy runs on the Node runtime here, so node:crypto HMAC verification works inside it without an edge-safe rewrite.
Two DB modules, split by intent, not by table. app/lib/queries.ts is SELECT-only and is the sole @/core/db consumer the token zone (app/s/**) imports; app/lib/mutations.ts holds every admin write and imports queries, never the reverse. QA's static check is transitive, so "readable zone imports only a read-only module" has to be true of the whole import graph, not just the page file. Naming the boundary makes the grep trivial.
Search is four EXISTS subqueries OR'd together, not a join. ILIKE over title/description is a plain WHERE; ingredient item and tag name go through EXISTS (SELECT 1 …) so a recipe with three matching ingredients is one row, not three. LIKE metacharacters (% _ \) are escaped, so searching for "100%" doesn't match everything.
Recipe-scope tokens are checked in the page, not the proxy. /s/** is deliberately outside the proxy matcher: token resolution needs the DB, and a wrong recipe id under a recipe-scope token returns the same 404 as a bad token. Nothing tells a guesser whether the token or the id was wrong.
Ingredient tick-off is CSS-only. <label><input type=checkbox> plus :has(:checked) strikes the line; no state, no JS, nothing persisted, so the readable zone stays write-free while still feeling like a cooking app. Hidden in print.
"Keep screen on" uses the Screen Wake Lock API and renders nothing where unsupported. The phone propped against the flour canister is the whole point of the readable zone; a screen that sleeps mid-step is the most common failure.
The editor preserves raw_text on edited rows. If the admin fixes "2 cup" → "2 cups" on an extracted ingredient, the line as written stays as extracted. Only hand-added rows get a composed raw_text (server-side). Raw source is shown under a <details> on the editor so review happens next to the evidence.
Login is a plain form + server action; the JSON route is for tools. Both call the same checkPassword / issueAdminCookie. Same for share links: the page uses server actions, the API routes use the same mutations.ts functions. One code path per behavior, two entry points.
Cookie age is enforced server-side (30 days) independent of maxAge. The signed issued_at is what's checked; the browser's expiry is a convenience, not a control.
Revoked links can be restored from /admin/share. Not in api.md, but ShareLinkPatch.revoked is a boolean, so { revoked: false } is already legal; a Restore button costs one bound server action and saves a mis-tap from being permanent.
Import forms degrade to "not on this branch yet" on a 404. The dashboard fetches Ingest's /api/imports/* routes, which may not exist in a given worktree; a 404 renders as a calm notice, not a stack trace, so each lane's page is reviewable before the merge gate.
Trap: curl -w '%{http_code}' appended to a JSON capture file broke my own smoke test's JSON.parse; the resulting empty token produced 308 trailing-slash redirects that looked like an auth bug. Not a Potluck issue, but it cost ten minutes; keep status codes out of body captures.
Shared local DB is shared. Other lanes' smoke tests added recipes and share links to potluck while UI was walking it (4 recipes became 6 mid-run). npm run db:seed re-asserts the seed rows but doesn't remove strangers. Fine for Phase 1; worth a per-worktree DB at the QA gate.

Requests for Orchestrator (UI)

proxy.ts at the repo root is UI-owned (see above). Please record that in the lane map / STATUS.md so nobody else edits it, and consider adding it to the "one writer per file" list explicitly.
api.md says POST /api/auth/logout is admin-only; the proxy exempts all of /api/auth/*, so the route checks the cookie itself and 401s without one. If you'd rather logout be idempotent-and-open (204 always), say so and it's a one-line change.
next build typegen writes .next/types and next-env.d.ts; both are already gitignored/committed as scaffolded. No package.json changes requested. ESLint is not installed, so the eslint-disable comments on the two <img> tags are documentation only — happy to drop them.
Vitest only includes tests/ and core//__tests__; UI has no unit tests (the readable zone is exercised by curl against the seeded DB and by next build). If QA wants UI helpers covered, app/lib/auth.ts (verifyAdminCookie/issueAdminCookie) and app/lib/validate.ts are pure and DOM-free; add app//__tests__//*.test.ts to the include.

Ingest — Phase 1 (2026-09-04)

Worktree potluck-ingest, branch agent/ingest. Developed against agent/extract merged locally (only core/extract/** + field notes); the Orchestrator's merge order (extract before ingest) makes that a no-op later.

Instagram export structure, actual vs. PLAN.md §4. The plan says "a saved_saved_media array; each entry carries the post title/handle and a string_map_data object containing the post href and saved timestamp". That matches what Meta ships today, with one detail the plan leaves out: the string_map_data key is the display string "Saved on" — a localized label, not a stable identifier. The parser therefore takes the first entry under string_map_data that has an href and ignores the key name entirely (fixture has a "Guardado el" case). Entries with no href anywhere (deleted accounts show up like that) count as skipped. The fixture at core/ingest/__tests__/fixtures/saved_posts.json is handwritten from that shape; the contract schema (InstagramSavedExportSchema) is what makes a shuffle fail loud.
Tracking params defeat the UNIQUE index. Exported hrefs carry ?igsh=…; a pasted reel link carries ?igshid=… or ?utm_source=ig. normalizeUrl strips igsh, igshid, utm_*, fbclid and the fragment before insert, so the same post saved twice really is one row.
What raw_source holds, per row kind. URL rows: the *entire fetched HTML* at fetch time — so a retry re-runs the JSON-LD stage too, not just Claude. The recipe's own raw_source is whatever Extract hands back (the JSON-LD block for Stage 1, the page text / caption for Stage 2). PDF rows: the segment's page text after extraction; a retry re-reads the file from storage/ instead, because scanned segments need the rendered images and the file is the source of record (Extract's note agrees).
Retry with nothing stored re-fetches. api.md says retry "re-runs extraction against the row's stored raw_source (no re-fetch)". A row that failed *at* fetch has no raw_source; the only useful retry is another fetch, so that is what happens. Rows that fetched but failed later never re-fetch. Flagging in case QA's invariant test counts fetches.
Duplicate of an already-saved recipe still returns a queue id. ImportAcceptedResponse requires pending_import_id; a URL whose recipe exists but has no queue row (seeded, or pre-queue) gets a done row linked to that recipe. No work is created, and the review UI sees it.
no_recipe rows carry an error string ("no recipe found in the source (confidence 0.05)") even though nothing failed — otherwise the review list shows a bare status with no hint why.
Instagram login walls are failed, not no_recipe. Anonymous fetches of instagram.com sometimes return the login page with no og:description. Ingest checks for that tag before calling Extract and fails the row with "Instagram returned no caption (login wall, private, or deleted post)", keeping the HTML. Treating it as no_recipe would hide the real problem (throttle harder, or the post is private).
PDF with no segments still leaves a row. If Pass 1 returns {recipes: []} or throws, one pdf row spanning pages 1–N lands in no_recipe / failed with the page text, so an upload never produces *nothing*. These rows are inserted already terminal (not walked through queued → fetching), which is the one place a row does not visibly move forward.
PDF route responds before rendering. loadPdfPages is called twice: once with renderScans: false for page_count / has_text_layer (fast, drives the 202 body), then in the background with rendering for the pipeline. A 300-page scan would otherwise hold the upload response for minutes.
Contract drift, small: PdfLoader in the contracts types the option as render_scans; Extract's loadPdfPages takes renderScans. Ingest calls what exists. One of the two should move.
undici hides the reason. A DNS or connection failure surfaces as Error: fetch failed with the real ENOTFOUND / ECONNREFUSED in err.cause. The first drain run stored a row with error "fetch failed: fetch failed"; core/ingest/fetch.ts now appends the cause.
The seed's queued row fails on drain. npm run drain picks the oldest queued row first, which on a seeded DB is the seed's https://recipes.example.com/sheet-pan-chicken-thighs/ — DNS fails, row goes failed (correctly; restored by hand after my run). Harmless, but a fresh db:seed followed by drain will always log one failure first.
Drain pacing, measured. Interval is 60000 / DRAIN_REQUESTS_PER_MINUTE from the *start* of the previous request plus DRAIN_JITTER_MS jitter, so starts are ≥ 17 s apart (< 3.6/min). It walks every queued row regardless of source_type (QA's seam test queues web rows), exits 0 when empty, and --limit N stops early. Env comes from process.loadEnvFile(".env.local") (Node ≥ 21.7; we are on 24).
Route handlers under Next 16: params is a Promise (awaited), plain Request + Response.json throughout, runtime = "nodejs" + dynamic = "force-dynamic" on each. pdfjs ran fine inside the upload route with the serverExternalPackages line already on main.
Admin auth, interim. core/ingest/auth.ts implements the cookie scheme from api.md: value <issued_at>.<hmac>, HMAC-SHA256 keyed by SHA-256(ADMIN_PASSWORD) — no extra salt. signAdminCookie() is exported so curl smokes and QA's seam tests can mint one. UI should either import this or implement byte-identically; swapping the import is a one-line change in five route files.
Live smoke (route-level, one Claude call set): next dev on :3131, every route hit with a minted cookie — url 202/202-duplicate/400/400/401, instagram 200 {queued:3,duplicates:1,skipped:1} and 400 on a non-export, pdf 202 {page_count:3,has_text_layer:true} and 400 on a non-PDF, retry 404/409/202, list 400 on a bad status. Background: cookbook-text.pdf → segments p2–2 "Grandma's Buttermilk Pancakes" and p3–3 "Weeknight Tomato Soup", both done, 8 ingredients / 4 steps each, confidence 0.9. The lemon-bars HTML fixture served from localhost:8787 went through npm run drain --limit 1 → done on the JSON-LD stage (no Claude call), 6 ingredients, 3 steps, 3 tags, pending_imports.raw_source = the HTML.

Requests for Orchestrator (Ingest)

Align PdfLoader option naming in contracts/types.ts (render_scans) with Extract's renderScans, or ask Extract to accept the snake_case key.
Decide where the admin cookie helper lives once UI lands: either UI imports core/ingest/auth.ts, or Ingest swaps its five requireAdmin imports for UI's. Both use SHA-256(ADMIN_PASSWORD) as the HMAC key with no salt — if UI picks HKDF or a salt, tell me and I match.
api.md: consider wording retry as "re-runs from stored raw_source when present; re-fetches only when the row failed before anything was captured", which is what ships.
Seed's queued row could point at a localhost URL (or be dropped) so a demo npm run drain does not open with a DNS failure.

2026-09-04 — Phase 5 integration pass, Orchestrator

Merge order bent, harmlessly. UI finished before Ingest, and the two lanes share no files, so UI merged first (Phase 4 before Phase 3). The plan's order was about dependencies, not ceremony; when the dependency isn't there, the order isn't either. Ingest's branch had merged agent/extract locally for development, which git treated as a no-op once Extract was on main.
Four lanes, four field-notes conflicts, zero code conflicts. Every merge conflicted only on this append-only file, exactly as predicted, and every resolution was "delete the three marker lines". Contracts did their job: not one line of TypeScript collided across five agents.
Two independent cookie helpers, one scheme, compatible on first try. Ingest wrote core/ingest/auth.ts and UI wrote app/lib/auth.ts from the same three lines in api.md; cookies minted by either verify in both. UI's adds a 30-day max age. Left both in place: the proxy guards Ingest's routes anyway, and the in-route check is defense in depth.
The only integration failure was a test racing a test. QA's invariant-4 check counts every table before and after hammering a share token; in the same run, the queue-invariants file was POSTing a real URL through the live server. fileParallelism: false in the root vitest config fixes it; the suite is under a second so nothing is lost. The app had no bug — the token zone wrote nothing.
Live seam run on main: 39 passed, 1 skipped (the opt-in drain timing test). npm run check: 238 tests. next build: every route listed, one tracing warning from the PDF readFile (harmless locally; the deploy milestone should pass an explicit outputFileTracingIncludes or move storage off disk).
Contract drift, one field: PdfLoader said render_scans; Extract and Ingest both used renderScans. The implementation was right and the contract moved, because a function option isn't a wire field. First time the contracts were edited after an agent started, and it was a rename.
Clock check for the guide: agent work time across all five lanes was roughly two hours. Wall clock was about seventeen, almost all of it agents parked behind the account's session rate limit. The build method scales in agents, not in hours — until the limit.

Ingest — first household bug (2026-09-04)

"file is not a PDF" on a real PDF. looksLikePdf demanded %PDF- at byte 0; the spec and Acrobat accept it anywhere in the first 1024 bytes, and scanner output / some exporters put junk (or a BOM) first. Now a 1024-byte window search — and then the real file arrived and that was not enough either. **Moms Recipes.pdf was an RTFD bundle with the PDF inside:** first bytes rtfd (macOS TextEdit's rich-text container), %PDF- at byte offset 16384, 63 scanned pages, 1.9 MB. Final approach: search the *whole* buffer for the first %PDF- (uploads are ≤ PDF_MAX_BYTES, so that is cheap), slice from there, hand the slice to pdfjs, and store the slice — not the container — under storage/pdfs/. pdfjs opened the raw container and the slice alike (63 pages both ways), and 16 / 512 / 1000 / 20480 bytes of synthetic junk, so the slice is for a clean file on disk rather than for pdfjs's sake. The 400 for a file with no header anywhere carries the first 8 bytes as hex, the declared MIME type, and the size, so the next report is diagnosable from the error alone. Tutorial callout: the first real household upload was not a PDF at all by its magic bytes, and a byte-0 check would have bounced it with a message nobody could act on.

2026-09-04 — Household test, first two findings (Orchestrator)

"Moms Recipes.pdf" is not a PDF. First bytes are rtfd: a macOS TextEdit rich-text container with the PDF embedded at byte 16384. Sliced out, it opens fine — 63 pages, no text layer, a scanned cookbook headed for the vision path. The magic-byte check that rejected it was correct and useless. Lesson: let the PDF library be the judge; a cheap header check should search the whole file, not the first bytes.
The Instagram export is HTML, and that's better. Meta's "Download your information" defaults to HTML; the plan assumed JSON. The HTML saved_posts.html carries a Caption row per post (1602 of 2141 in Tommy's export, up to ~1300 chars), plus Username and Name. The JSON format has only links. So the throttled page-scrape the plan built the drain around is unnecessary for three quarters of the backfill: extract straight from the export. The contract grew InstagramSavedEntry and pending_imports.source_author; Ingest normalizes both formats.
Collections are not categories. saved_collections.html names are mostly accounts ("Jordan Berg", "Camera Confidence for Women 40+"), not "dinner" / "desserts". Parsed, not applied as tags. Most of a real saved feed isn't recipes at all; the no_recipe list is going to be long, and that is the design working, not failing.
Cost check before the backfill: ~1600 caption extractions on Sonnet 5 at roughly 1–2K input tokens each is on the order of ten to twenty dollars. Say so before pressing drain.

Ingest — the actual Instagram export structure (2026-09-04)

This is the promised "what the export really looks like" note. PLAN.md §4 said: *"select Saved only, JSON format. File arrives as saved/saved_posts.json … a saved_saved_media array; each entry carries the post title/handle and a string_map_data object containing the post href and saved timestamp."* Tommy's real export (your_instagram_activity/saved/, generated 2026-09-04) is:

HTML, not JSON — and the HTML carries the captions. saved_posts.html (5.4 MB) is one <table> per post: a URL cell holding the post link, a Caption row with the full caption (entity-encoded: ', @ for @, &, <), rarely a Title row, a nested Owner section with Name (display) and Username (handle) — and, when the poster has a website, a *second* URL row in plain-text form inside Owner. A <div class="_3-94 _a6-o">Sep 03, 2026 1:47 pm</div> footer carries the save time (local, no zone). Some posts also carry a Hashtags section (with its own Name label as a <div>, not a <td>) and 18 have a Brand partner section. The JSON format, which the plan designed for, has no caption at all — just href + timestamp.
Numbers from the real file (through the shipped parser, 21 ms): 2143 unique post URLs (2012 reels, 129 posts, 2 IGTV), **2089 with a caption** (median 241 chars, max 2198), 0 skipped, 0 undecoded entities. saved_collections.html (4.9 MB): 20 collections covering 1905 of those URLs — each collection is Name / Type / Privacy / Update time and then a Media section that *re-embeds the same post tables*, captions included.
The scanner bug that only the real file could show. My first pass treated every URL-labeled row as a new post. The Owner section's plain URL</td><td>https://their-site… row (1731 of them) reset the current post, so only 430 of 2161 usernames attached. Only the URL<div><a href> form is a post now; the fixture carries a website row to keep it that way.
What this does to the plan. The throttled drain existed to fetch captions from instagram.com, politely, over an afternoon. With the HTML export, 2089 of 2143 rows are born with raw_source = caption and source_author = username: processImport claims them straight into extracting and calls extractFromText — zero fetches, and the drain skips the pacing wait for them. The throttle now matters only for the 54 caption-less rows (and JSON exports, and pasted URLs). The backfill went from "an afternoon of polite scraping" to "however fast Claude answers 2089 prompts".
Detection is by content, not filename. {/[ → JSON; an <html/<table/<body tag in the first 4 KB → HTML. Both normalize to InstagramSavedEntry[], deduped by normalized URL within the file; a re-upload that now carries a caption for a still-queued, never-fetched row fills it in rather than being a pure no-op.
Collections are mostly not tags. The 20 collection names in this export are largely account names and "Cool"; per api.md they are parsed onto the entry but not applied as tags in v1.

Requests for Orchestrator (Ingest, round 2)

None open: with_caption + format landed on ImportInstagramResponse and Schema's pending_imports.source_author merged while this was being built (both picked up by merging main twice mid-task). One heads-up for the drain's tutorial paragraph: with the HTML export the pacing wait is skipped for caption rows, so "a few requests per minute" now describes only the caption-less minority.

2026-09-04 — Household test, round two (Orchestrator)

The real test uploads went through in one try after the RTFD and HTML fixes: Moms Recipes.pdf → 202, 63 pages, no text layer; saved_posts.html + saved_collections.html → 2143 queued, 2089 with captions, in about a second.
Then the drain said no_recipe five times in a row, correctly. The first captioned rows were photography reels. Tommy: "those are all of my saved reels, not just recipes, there should be one set that is only recipes." There is — a collection literally named Recipes, one of twenty (College, Dirtbikes, Gambling, Tinfoil Hat…). The plan designed the backfill around "your saved posts"; the real unit is the collection. Import became two-step: preview (collections with counts, writes nothing) → queue only the checked ones.
The scanned cookbook rendered 63 blank pages. pdfjs in Node: Jbig2Error: JBig2 failed to initialize. The scan is JBIG2-encoded and pdfjs 5+ decodes that with WASM that has to be located explicitly under Node. Vision was handed 63 white rectangles and answered "no recipes", which is the honest answer to that question. Two lessons: a decode warning in a library log must become a hard failure on the queue row, and "no recipes in 63 pages" should never be reachable silently — a blank-render detector now sits in front of the vision call.
Survey error, mine: I first read saved_collections.html's Name rows as collection names and concluded "collections are people". Most Name rows are the *poster's* display name inside each media entry; the collection-level Name is the one followed by Type/Privacy. Ingest's parser had it right; my grep didn't. Recorded because the guide should show the orchestrator being wrong too.

2026-09-04 — UI follow-up: Instagram import is two-step

A real saved feed is not a recipe list. Tommy's export: 2,143 saved posts, one collection that is recipes. Queueing everything would have drained for hours (DRAIN_REQUESTS_PER_MINUTE = 4) over mostly not-food. So choosing the file(s) on /admin now POSTs them to /api/imports/instagram/preview (parse only, writes nothing) and renders format, total, with-caption count, and a checkbox list of collections with counts; the real POST carries one only_collections field per tick. Nothing is ticked by default — the safe default for a queue that costs API calls is "queue nothing until told".
The preview response is the contract's InstagramPreviewResponse; the UI holds no parsed export state of its own, just the File objects and a Set<string> of ticked names, and re-sends the same files on submit so the server parses once per step and the client never touches the HTML.
"Queue up to N posts" is an upper bound: a post can sit in several collections, and the server dedupes. Saying "up to" beats a number that is wrong by a few.
A 404 from the preview route (branch without Ingest's work yet) renders as a notice plus the old one-step "Queue everything" button, so the page still works in every worktree.

Ingest — collection filter (2026-09-04)

The drain's first five captioned posts were all no_recipe — because a saved feed is *everything ever saved* (for Tommy: mostly photography), and only one collection is recipes. Importing all 2143 would have spent 2089 Claude calls to find a few dozen recipes. The import is now two-step: POST /api/imports/instagram/preview parses both files and returns per-collection counts of posts present in the export (writes nothing); the real import takes repeated only_collections=<name> and queues only those, counting the rest in skipped. Names match exactly after entity decoding and trim. With no filter it behaves as before.
The collections parser keys on the collection-level Name row (the one followed by a Type row); the per-media Name (poster display name) inside the embedded post tables is skipped. The real file has 20 collection tables under 19 distinct names — two collections share a name, which a name-keyed Map silently collapsed (second overwrote first; 7 posts fell out). They are merged now, since the filter is name-based. Preview on the real export: 19 names, 1905 of 2143 posts in at least one, 238 uncollected.
The multipart reading is shared (core/ingest/instagramForm.ts) so the preview and the import cannot drift on what file / collections mean.

2026-09-04 — First real backfill numbers (Orchestrator)

Preview → filter → drain worked end to end on the real export. Preview: 2143 posts, 2089 with captions, 19 collection names. Filter Recipes: 102 queued, 101 with captions, in 143 ms. Drain of the first 12 captioned rows: **5 recipes saved (4 auto, 1 needs_review at 0.55), 7 no_recipe.** Each caption extraction took 2–10 s with no fetch.
**The hit rate inside a recipe collection is about 40%, and that is the video-transcription signal the plan promised.** The misses are reels whose caption is a hook ("you NEED this", macros, "save for later") with the recipe only spoken on screen. The no_recipe list is now a concrete v2 backlog with the raw captions attached, exactly the "never silently drop a save" payoff.
Extraction quality on the hits was good: titles, authors (from the export's Username row), 4–11 ingredients, 4–5 steps, confidences 0.80–0.90. The 0.55 one is a legitimate review case (a wrap with vague quantities).
One caption-less row took the fetch path and hit Instagram's wall: 719 KB of page HTML, recipe_found: false at 0.95. Correct outcome, wrong label — Ingest already classifies login walls as failed, but this page returned enough HTML to look like content. Fine for v1; the fetch path is now the exception (54 of 2143), not the plan's main road.

2026-09-04 — Extract: the household scan that came back "no recipes"

63 pages of Mom's cookbook, 63 blank PNGs, zero errors. The scan is JBIG2-encoded (the bilevel codec scanners and macOS Preview default to). pdfjs-dist 5+/6 decodes JBIG2 and JPEG 2000 in WebAssembly that it has to be *told* how to find — in a browser that's document.baseURI, in Node it is nothing, so every image "failed to initialize", every page rendered flat white, and vision dutifully reported no recipe on all of them. The only signal was a console.log warning nobody reads in a fire-and-forget job. Fix: getDocument({ wasmUrl: <pdfjs-dist>/wasm/ }) (trailing slash mandatory), resolved from the installed package at runtime; fonts and CMaps get the same treatment while we're there. Page 1 now renders a handwritten "Cola Cake" card at 1212×1568.
Silent blank is the failure mode to design against. Two guards now: the loader flags a page that *paints images yet renders one flat color* as undecodable (a genuinely empty page paints nothing and is just blank; real scans of empty pages carry noise and pass as readable), and a file with no readable page throws PdfReadError with the codec hint. segmentPdf / extractFromPdfPages likewise refuse a page set with no text and no image (ExtractionError kind empty_input) — the row fails with a reason instead of joining the couldn't-parse list. Default is "mark", not "throw", because one dud page in a 63-page scan should not sink the other 62; Ingest reads undecodable_pages and decides.
stopAtErrors: true does not surface image-decode failures — the render promise resolves fine over an undecoded image. Detection has to be after the fact, from the pixels, which is why the guard is heuristic.
No offline JBIG2 encoder on the machine, so the fixture is a hand-built PDF whose XObject is /JBIG2Decode garbage: the WASM decoder runs, fails, and the page goes flat — the exact pre-fix symptom, now caught by a test.
Request for Orchestrator/Ingest: after loadPdfPages, treat undecodable_pages.length > 0 as worth surfacing (row error or a warning on the file), and pass PdfReadError.message straight through as the failed reason — it is written for that.

2026-09-04 — UI: browse features (course groups, facets, nutrition)

"A thousand keywords is not a filter." Tommy's first reaction to the family link with ~90 real recipes: the tag list was noise. The collection page now has exactly four top-level controls — course chips (which are also the grouping), an author select, "under N minutes", and calories/protein only when facets.has_nutrition — plus a second, smaller "Diet & method" chip row that shows ONLY tags from PREFERRED_TAGS. Freeform tags stay searchable through q and visible on the recipe page; they are never offered as a list. Cards show at most two tags, preferred first.
**Facets are computed over the whole collection, not the current filter.** A chip's count means "how many exist", and a control never disappears because the current filter emptied it. Four small GROUP BYs; fine at family scale. Authors are top 20 by count; when truncated the select carries a disabled "more… search by name above" row, and the currently selected author is always present as an option even if it fell outside the top 20, so the URL state is never invisible.
The URL is the state. Every filter is a query-string key (RecipeFilters ⇄ app/lib/filters.ts, shared by the page and GET /api/recipes), so /s/<token>?course=dinner&max_minutes=30 is a shareable "what's for dinner tonight" link. Selects auto-submit through a 12-line client component; without JS the same form still submits via the Search button.
course=other matches null too. Null course renders under "Everything else" (last, per COURSES), so filtering by that chip has to find the nulls as well or the chip count and the result count disagree.
"Under N minutes" uses coalesce(total, prep+cook) with NULL when neither prep nor cook is set — a recipe with no timing never matches a time filter, rather than matching every one because 0 ≤ N.
Group headers hide when one course is selected (api.md), and also when only one group exists — a lone "Dinner" header over the whole list is just noise.
Nutrition is only ever what the source states. The editor's four inputs say so in the helper text; the recipe page prints a single "Per serving: …" line and only the values present. Grams accept one decimal; calories are integers, matching the contract.
Gate note: on this branch tsc is clean in app/, components/, and proxy.ts; the remaining 6 errors are Extract/Ingest/QA fixtures that haven't grown course/nutrition fields yet. npm run check will be green for UI as soon as those lanes rebase.

2026-09-04 — Extract: browse features (course + nutrition)

Schema growth was free on the Claude side. The five new fields (course, calories, protein_g, carbs_g, fat_g) ride through zodOutputFormat(ExtractionResultSchema) untouched: all five land in the wire schema's required as anyOf [type, null], so the grammar forces the key to appear; .default(null) becomes a description note (the SDK strips it like every other non-grammar keyword) and .catch(null) on course turns an off-list value into null at parse time. No post-processing, no wire-schema fork. Live macro caption ("160-calorie protein cheesecake… 24g protein per slice"): `course: dessert, calories: 160, protein_g: 24, carbs_g: 9, fat_g: 3`, servings "8", auto_save.
JSON-LD: recipeCategory → course by keyword family (breakfast/brunch, dessert/sweets/cakes…, drinks/cocktails, side dish, appetizer→snack, lunch, main/entrée/supper→dinner), else null — not "other"; "other" is the model's explicit judgement, null means the publisher didn't say. nutrition.calories / proteinContent / carbohydrateContent / fatContent parse the leading number ("350 calories", "1,250 kcal", "12.5 g"); calories rounded to an integer to satisfy the schema.
Tags addendum (PREFERRED_TAGS): every stage's suggested_tags now pass through one normalizer (core/extract/tags.ts): lowercase, # and hyphen/underscore cleanup, an alias table onto the vocabulary ("high-protein"/"protein" → high protein, "gf" → gluten free, "crockpot" → slow cooker, "sheet-pan" → one pan, "bbq" → grill, "30-minute meals" → quick…), a single-vocabulary-word-inside-a-phrase rule ("gluten free recipes" → gluten free), dedupe, and a cap of five with preferred tags kept ahead of freeform. "one pot" deliberately stays freeform — a pot is not a pan and the facet would lie. JSON-LD reads keywords before category/cuisine because that is where publishers put the diet/method words. Zod's .toLowerCase() on the schema already lowercased; the normalizer is what makes "Gluten-Free" and "gluten free" one facet.

Ingest — browse features: nutrition fields, retry-in-place, `--retry-done` (2026-09-04)

persistExtractionOutcome now writes course, calories, protein_g, carbs_g, fat_g from the result. Nothing else about the insert changed.
Retry from done re-extracts in place. The day's 30-odd recipes were saved before those fields existed; re-importing them would have made twins (or bounced on the source_url UNIQUE). retryImport now accepts done and keeps recipe_id on the row; persistExtractionOutcome sees that id and does UPDATE recipes … ; DELETE + re-INSERT ingredients, steps and recipe_tags in one transaction. Same id → share links, URLs and anything the household bookmarked survive. If the re-run comes back no_recipe, the row says so (error notes it) and the recipe is left untouched rather than deleted — a worse extraction should never erase a good one.
npm run drain -- --retry-done walks every done instagram row that has a caption, retries and re-processes each in place, no fetch and no pacing (--limit N still caps it). One command backfills the new fields.
api.md deviation: it still says retry is "allowed from failed and no_recipe"; the routes now accept done as well. Suggest updating the sentence to "any terminal status; done updates the recipe in place".

2026-09-04 — Mom's cookbook, extracted (Orchestrator)

63 scanned pages → 55 segments → 46 recipes, 9 no_recipe, 1 needs_review. Handwritten index cards on lined paper, read by vision after the JBIG2 fix: Cola Cake, Italian Cream Cake, 125 Year Old Walnut Pound Cake, Buttermilk Fried Chicken, Chicken and Dumplings, Hush Puppies (the one flagged for review, at 0.72). Confidences 0.72–0.90, 3–17 ingredients each. ~10 s per page, roughly 55 vision calls plus one 63-image segmentation call. The plan's "OCR-quality scans of handwritten cards will be the low-confidence tail" was right about the band and wrong about the outcome: nearly all auto-saved.
Segmentation was the surprise. One call over 63 page images found 55 recipe boundaries, including two-page recipes (13–14, 16–17, 19–20, 24–25, 41–42, 43–44, 47–48) and paired items ("Cola Cake" p1, "Icing For Cola Cake" p2). The nine no_recipe pages are worth a look: either blank/photo pages or cards the segmenter split wrong.
Instagram Recipes collection, complete: 102 → 43 recipes, 59 no_recipe. Total in the box after one afternoon: 89 real recipes plus the 4 seeds. The no_recipe list (68 rows) is the v2 backlog.

2026-09-04 — Version skew between the drain and the dev server (Orchestrator)

Nine legible recipe cards came back no_recipe at confidence 0.00 — Mommy's Apple Pie, Chili-Beans, Meat Loaf, Lasagna… The blank-render signature, hours after the JBIG2 fix was merged. Cause: my `npm run drain --limit 100` had been running since before the fix landed. tsx loads modules once; the drain kept the old loadPdfPages in memory, saw the cookbook's fresh queued segments in the shared queue, claimed nine of them, rendered them white, and asked vision. The hot-reloaded dev server processed the other 46 with the new code. Same queue, two processes, two versions.
Fix was cheap (re-queue nine rows, drain again with current code: 9/9 saved, 0.75–0.90) but the lesson is structural: a long-running worker and a hot-reloading server sharing one queue will disagree about what "the code" is. Restart the drain after any merge that touches Extract, or have the drain check the git HEAD it started on. Worth a callout: the queue's claim semantics prevented double-processing exactly as designed, and still produced wrong answers.

2026-09-04 — QA, browse-features round (course + per-serving nutrition)

Existing expected.json files validate unchanged against the new ExtractionResultSchema (course/nutrition default null) — 138 loader assertions green before any fixture was touched. Three fixtures added: caption/macros-protein-pancakes (per-serving macros, breakfast cue), caption/macros-cottage-cheese-brownies (per-piece macros, dessert cue), html/jsonld-nutrition-granola-bars (recipeCategory + NutritionInformation, Stage 1 only). 20 fixtures now: 12 caption · 7 html · 1 pdf.
Rubric extension is additive. metadata (weight 15) gains a course term (exact) and one term per expected nutrition value (within 10%) ONLY when the fixture expects them, so the 17 older fixtures are scored exactly as before. Nutrition stated where a fixture expects null is reported as an informational diff, not scored ("never estimate" is a prompt rule; the eval should not punish a correct value the fixture author didn't record). Course is not reported that way — the prompt always classifies it.
First real headline: 96.3% over 20 fixtures (npm run eval now calls Claude — request 1 from Phase 1 was applied on main; eval:offline keeps the JSON-LD-only run). Captions already come back with course and macros from the shared prompt (the two macro captions score 98.1 / 95.3, every stated number exact); the JSON-LD path does not emit course/nutrition yet, so jsonld-nutrition-granola-bars scores 90.6 and will move once Extract's browse change lands. pdf/text-layer-lemon-loaf scores 100 through loadPdfPages → extractFromPdfPages.
Captions that fooled the current prompt (for the eval-per-revision slot): follow-for-more-cookies — 11 steps for 6 paragraphs and prep 80 (the 1-hour chill folded into prep); metric-units-shakshuka — 8 steps for 5 (the German method and the English recap were partly both kept); bilingual-arroz-con-pollo — title kept "de mi Abuela"; `vague-quantities- family-chili — confidence 0.45 → no_recipe` where the fixture expects the needs_review band (0.62). That last one is the threshold question PLAN.md §0 anticipated: a real recipe with vague quantities is being dropped to the "couldn't parse" list instead of the review queue. Worth a look before tuning REVIEW_MIN — the prompt may simply be under-confident on measurement-free family recipes.
done queue rows without a recipe. The household's real 63-page PDF run left done rows (pages 31, 37) with recipe_id null and no error. Because pending_imports.recipe_id is ON DELETE SET NULL, this is indistinguishable at rest from "extracted, then deleted in the admin UI" — which is presumably what happened. The at-rest invariant-3 check now requires raw_source on every done row but tolerates a null recipe_id. If the Orchestrator wants the queue to remember *why* a done row has no recipe, that needs a contract decision (e.g. UI sets `error: "deleted by user" on delete, or a recipe_deleted_at` column).
Scanned-page rows carry a 66-char raw_source. For vision-path pages the "raw source" is the page image, and the queue row stores a short placeholder. Fine for the audit trail as long as the PDF stays on disk; noting it because "raw source is sacred" reads differently for scans.
New seam test tests/seams/recipes-filters.test.ts: GET /api/recipes with course=dessert&max_minutes=30 must equal the client-side filter of the unfiltered listing (using the contract's total→prep+cook fallback); facets course/author/tag counts must sum against that listing and has_nutrition must match the data. Guarded on POTLUCK_BASE_URL + ADMIN_PASSWORD; it inserts nothing. The contract does not say whether null-course recipes count under "other" in facets.courses (the page groups them there) — the test accepts either and logs which.

2026-09-05 — Single-agent mode (Orchestrator)

Tommy stopped the multi-agent build: "you burned thru 92% of my tokens in about 10 minutes… your agents are fighting more than we are accomplishing." The last two "bugs" I had dispatched three agents to chase were recipes he had deleted on purpose. Every agent resume reloads 150–300K tokens of context; five lanes × one message per finding is how a session allowance vanishes. From here the Orchestrator writes the small things itself. Honest tutorial callout: contract-first multi-agent is for the *build*; for the *tuning* loop it is the wrong tool.
Steps 1–3 by hand, ~2 hours, under 1K lines: course backfill (drain re-runs in place), estimated nutrition (estimateNutrition, flagged and labeled "est."), photos from each post's own og:image (stored, not linked — Instagram's image URLs are signed and expire), and a free DuckDuckGo lookup for the cookbook's photos, each credited to its page.
Claude web search was the wrong tool for photos: two calls cost 13¢ because each search returns ~30K tokens of page content. DuckDuckGo's HTML endpoint plus the existing og:image fetcher finds the same pages for nothing. Measured, then swapped.
The $5 key ran out at Pecan Pie. The nutrition script logged the 400, stopped, and left the row untouched — "never silently drop" paid off one more time. npm run usage now exists so the next dollar is visible before it is spent.
A wait loop that greps for its own command line waits forever. until ! pgrep -f drain.mjs matched the shell running it. Two background jobs sat idle for an hour. Trivia, but it cost real time.

2026-09-05 — Evening, UI and the traps (Orchestrator)

"It kind of sucks" → screenshots first. Headless Chrome rendered the family link at 414 px and everything was clipped on the right. Two fixes later it was still clipped, on the login page too. Chrome on macOS will not open a window narrower than ~500 px; the screenshot was cropped, the CSS was fine. Measure the tool before trusting the measurement.
Restyle in one commit: framelogic.ai's palette (near-black warm ground, sand accent, off-white type; dark only, print inverts) and its three faces via next/font; photo-first 4:3 tiles in a 2/3/4-column grid; hero photo first on the recipe page; nutrition as a table that fits a phone; source as "Moms Recipes, page 24" on every card, searchable, and a "Source" facet. Tommy: "This is looking much better."
Two stale-state bugs that looked like code bugs: the editor refused /recipe-images/… because the field was type="url" (browser-side, the server never minded); and the API rejected course/nutrition keys because the dev server had been running since before the schema changed. Neither needed a line of logic.
Credit that never arrived. Tommy bought API credit; the key kept saying "balance too low" for 40 minutes. The key authenticates, so the org it belongs to is not the org that got the money. Left for tomorrow with the fix written down.
Instagram profile pages are a wall. Fetched a creator's reels page with a browser UA: HTTP 200, 660 KB, zero post links, four "log in"s. The app's per-post fetch works because single post pages still expose og:description; listings do not. Offered batch-paste and website-sitemap import instead of anything that logs in.

2026-09-05 — Tommy's verdict on the multi-agent build

In Tommy's words: "I consider the multi-agent my first breakdown. I do not consider it a success." Recorded as said, because the guide is not honest otherwise.
What it cost him: 92% of a Claude Code session allowance in about ten minutes of fan-out, ~17 hours of wall clock for ~2 hours of work while agents sat behind the rate limit, three agents dispatched after two "bugs" that were his own deletions, and a day where he could not tell what was going on. His counterfactual: "with one fable agent I could have completed this last night." The single-agent day that followed shipped nutrition, photos, source labels, a full restyle, and the fixes, with a clean gate, in roughly the same session budget the fan-out burned in minutes.
What the Orchestrator would defend: contracts-first meant five agents wrote ~7K lines with zero code conflicts, and each lane's report was accurate and testable. What it would not defend: treating every finding as a dispatch, never checking cost per agent resume, and letting the method run past the point where the work needed it.
The lesson the guide should carry, in one line: multi-agent is a build tool, not an iteration tool; the moment a human is testing and reacting, fold back to one agent.

← Back to the guide