From aa0eba3457272432211e3b5bb200a5526d3abc27 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cau=C3=AA=20Faleiros?= Date: Fri, 18 Sep 2026 18:33:28 -0300 Subject: [PATCH] docs: track outstanding work in ROADMAP.md Findings from the 2026-09-18 audit, ordered by block, each with its acceptance criterion and audit id. Block 0 is closed; the remaining blocks record security, architecture, scale and maintenance work, including the decisions that need a product answer before any code. Co-Authored-By: Claude Opus 5 --- ROADMAP.md | 402 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 402 insertions(+) create mode 100644 ROADMAP.md diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..a60d57a --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,402 @@ +# DTF System — Working Roadmap + +> Internal engineering tracker. Not a client document, not a promise sheet. +> The client-facing narrative lives in `output/pdf/dtf-plano-producao-e-roadmap.pdf` +> and in the weekly report (`Relatorio-Semana-1-DTF.docx`). +> +> Update the **Current step** line and the item status every time something moves. +> Add new findings at the bottom of the relevant block rather than rewriting history. + +**Current step:** Block 0 closed on 2026-09-18 (0.1–0.6 done and verified against a +live stack). Next: Block 1 needs client inputs for 1.1/1.2, so Block 2 (2.1, 2.2, +2.3) and 5.1 are the ones that can start immediately. + +**Last audit:** 2026-09-18, full read of `local/`, `dtf-site.html`, `deploy/`, +`.gitea/`, docs and legacy prototypes. Findings below carry their audit IDs. + +| Status | Meaning | +|---|---| +| `[ ]` | not started | +| `[~]` | in progress | +| `[x]` | done and verified | +| `[?]` | blocked on a decision (product or client), not on code | + +--- + +## Block 0 · Broken right now + +Nothing in this block is optional. Until it is closed, the system cannot be +demonstrated, and the week-1 claims cannot be defended. + +### `[x]` 0.1 — API returns 500 on every session, login and registration `(F1)` + +`local/auth.py:46` calls `os.environ` and the module never imports `os`. +Reproduced: `NameError: name 'os' is not defined`. `/api/session` calls +`new_session()` whenever there is no cookie, so the Site checkout bridge, cart +recovery and the customer portal all fail on first visit. Introduced in `e3e37f6`. + +- Add `import os` to `local/auth.py`. +- Move the `COOKIE_SECURE` read to a module constant so it is evaluated once. +- **Accept:** a fresh browser hits the Site and `/api/session` returns 200 with a + `cart_scope`; `local/smoke_test.py` and `local/workflow_test.py` pass. + +### `[x]` 0.2 — "Arquivo por metro" is unreachable in practice + +Two independent causes, both from 2026-09-18 commits. Verified in a browser. + +**a. Every entry point hard-routes to loose artwork** (`483a083`) + +| Control | Currently opens | +|---|---| +| Nav "Impressão DTF" | `avulsa` | +| "DTF Têxtil · 57 cm" | `avulsa` | +| "DTF UV · 28,5 cm" | `uv` | +| Hero "Enviar minha arte" | `avulsa` | + +Only the price card reaches `file`. Revert the three `data-modo-cta` attributes +so navigation links land on the product chooser, not on a product. + +**b. Dropping a PNG/JPG in `file` mode silently switches the order** `(F25)` + +`dtf-site.html` `sel()` — from `abrir('file')`, dropping `imagem-teste.jpg` gives +`modo: "avulsa"`, header "Artes avulsas", price `R$ 29,90/m`. Meanwhile the same +screen says *"Arraste suas folhas montadas · PNG, JPG ou PDF"*, marks +*"PNG, JPG ou PDF · a partir de R$ 14,90"* as the recommended path, and sets +`input.accept=".png,.jpg,.jpeg,.pdf"`. The page invites the drop and then +reprices the order 50% higher. + +The escape hatch `#imagemComoFolha` sits above the drop zone as small text inside +an informational notice, defaults to unchecked, and must be ticked *before* the +drop. After the switch fires, `pintaModo()` sets `trocarParaAvulsa.hidden = true`, +so the checkbox disappears and there is no way back in place. + +Net effect: the only formats that survive `file` mode are PDF/TIFF/PSD/AI/CDR — +the path the UI itself marks as the worse option. A mixed drop (JPG + PDF) is +rejected and accepts nothing. + +- Make the ready-sheet choice an explicit two-option control **inside** the drop + area, styled like the existing `.cam` selector — not a checkbox in a notice. +- Stop advertising PNG/JPG in the by-metre drop zone while rejecting them. +- When a switch does happen, show the price change and offer one-click undo. +- **Accept:** a customer can complete a by-metre order with a PNG from any entry + point, and no product/price change ever happens without a visible confirmation. + +### `[x]` 0.3 — Ready-sheet declaration is unverified and worth money `(F22 related)` + +Ticking `#imagemComoFolha` is an honour-system claim that moves the price from +R$ 29,90/m to R$ 19,90/m (R$ 14,90 with a good grade). `medirFolha` then derives +sheet height purely from aspect ratio × 57 cm — the original bug, now opt-in. + +Measured with `imagem-teste.jpg` (466 × 659 px): + +| Route | System behaviour | Billed | +|---|---|---| +| Ticked | treated as a 57 × 80,6 cm mounted sheet | 1 m × R$ 19,90 = **R$ 19,90** | +| Not ticked | 20 cm wide, packs to 28,3 cm of film | 1 m × R$ 29,90 = **R$ 29,90** | + +- Validate the claim: declared width must be ≈ film width (57 / 28,5 cm) at a + plausible DPI before the sheet model is accepted. +- Re-check server-side in `/api/operator/quotes/{id}/approve` before pricing. +- **Accept:** a small single artwork declared as a ready sheet is rejected with a + clear message; a genuine 57 cm sheet passes; the operator sees the verdict. + +### `[x]` 0.4 — Documented local startup fails `(F2)` + +`LOCAL_SETUP.md` says `docker compose up --build` with no `.env`. +`docker compose --env-file .env.example config` exits 1: `R2_ENDPOINT`, +`R2_ACCESS_KEY_ID`, `SITE_DOMAIN`, `KANBAN_DOMAIN` missing. `docker-compose.yml` +became a production/R2 stack in `e3e37f6`; `.env.example` is still the MinIO one +and there is no MinIO service left. + +- Decide: keep one production compose and add `compose.local.yaml` with MinIO, or + restore a local default. Recommend the former. +- **Accept:** a clean clone reaches a working Site + Kanban with the documented + command, and `LOCAL_SETUP.md` matches what actually runs. + +### `[x]` 0.5 — App DB role shares the admin password `(F3)` + +`docker-compose.yml:65` sets `APP_DB_PASSWORD: ${POSTGRES_PASSWORD}` — the same +value as `dtf_admin`. `bootstrap.py` grants the app role DML-only and then hands +it a credential that also logs in as the owner. Anyone reading the API container +env has admin on the database. + +- **Accept:** distinct secrets; connecting as `dtf_app` with the admin password fails. + +### `[x]` 0.6 — A paid order showed the customer nothing (found while closing Block 0) + +`#checkoutStatus` and `#checkoutActions` lived inside `#carr`, and the success path +in `checkout.js` clears the cart (`pedido=[]; limpaPaineis()`) before writing the +confirmation — `.carr{display:none}` then hid the panel holding it. Present since +the first commit; it only surfaced once 0.1 made a payment reachable at all. +Both elements now sit in their own always-visible `.checkout` container. + +### How Block 0 was verified + +A live stack (`compose.local.yaml`), then the full suite: + +| Check | Result | +|---|---| +| `/api/session` on a cold browser | 200 with `cart_scope` + session cookie | +| smoke · workflow · security · scanning | pass | +| retention · runtime security (in-container) | pass | +| `artwork_browser_test.mjs` | pass, updated to the new declared-product behaviour | +| `browser_test.mjs` end-to-end | pass — upload → quote → operator approval → paid order → all Kanban states | +| 4 CI unit tests | pass | + +Ports 8090/8091/8010 were used; 8080 was held by an unrelated preview server. + +--- + +## Block 1 · Week 2 — committed to the client + +From the report already sent. These are dated promises, not backlog. + +- `[ ]` 1.1 — Mercado Pago transparent checkout, signed and idempotent webhooks. + Requires production credentials + webhook access. Payment must never be created + before the freight amount is final. +- `[ ]` 1.2 — Real freight quotation. **Blocked on client inputs** (see + `PRODUCTION_INPUTS.md`): source platform, credentials, origin CEP, services, + packaging weight/dimensions per length, subsidy policy. +- `[ ]` 1.3 — Idempotent Tiny/Olist order creation with order-number traceability. + Confirm endpoints, tag behaviour and rate limits first. +- `[ ]` 1.4 — Final print-file generation (see 3.2 — this is the same problem). +- `[ ]` 1.5 — Main Kanban production states consolidated. +- `[ ]` 1.6 — **Block 0.2 + 0.3**, promised as "início da próxima semana". + +`[!]` The production compose currently blocks `dev_paid` (`ENVIRONMENT != 'local'`) +and ships only fake adapters, so the deployed system cannot take an order at all. +1.1 is what unblocks it. + +--- + +## Block 2 · Security — before any public exposure + +### `[ ]` 2.1 — Rate limiting and audit logs are blind to the client `(F5)` + +uvicorn runs without trusted proxy headers (`forwarded_allow_ips` defaults to +`127.0.0.1`; nginx is a different container IP), so `request.client.host` is nginx +for every request. `rate_limit('auth-source', ...)` at 60/15min becomes a single +global bucket — **60 failed logins lock out every customer** — and every `audit()` +record has no attacker IP. + +- Set `--proxy-headers` with `FORWARDED_ALLOW_IPS` scoped to the nginx service, or + read `X-Forwarded-For` explicitly at the edge. +- **Accept:** two clients on different IPs have independent buckets; audit rows + carry the real IP. + +### `[ ]` 2.2 — The public site throttles itself `(F6)` + +`local/app.py:115` — `rate_limit('guest-sessions', ENVIRONMENT, 120, 900)` is keyed +on the environment name: 120 new visitors per 15 minutes **site-wide** (~8/min). +Normal traffic 429s. Key per source IP (after 2.1) and raise the ceiling. + +### `[ ]` 2.3 — `deploy/stack.yaml` cannot boot `(F7)` + +It passes `DATABASE_URL_FILE`, `AWS_ACCESS_KEY_ID_FILE`, `OPERATOR_PASSWORD_FILE`, +`OPERATOR_USER`. The code reads `DATABASE_URL`, `AWS_ACCESS_KEY_ID`, +`OPERATOR_PASSWORD`, `OPERATOR_EMAIL`, and no `_FILE` loader exists. +`app.py:58` does `os.environ['OPERATOR_PASSWORD']` → `KeyError` → 500 instead of 503. + +- Implement `local/secrets.py` (the preflight already expects it) reading `*_FILE` + with env fallback. Reconcile `OPERATOR_USER` vs `OPERATOR_EMAIL`. +- **Accept:** the stack renders and boots against Swarm secrets; missing operator + config yields 503, not 500. + +### `[ ]` 2.4 — The documented release gate does not exist `(F8)` + +`PORTAINER.md` and `SECURITY_REPORT.md` claim the workflow runs the full isolated +suite, Trivy HIGH/CRITICAL image gates, secret scanning and the source preflight +before calling Portainer. `.gitea/workflows/deploy.yml` runs `py_compile` plus four +unit tests, then builds, pushes `latest` and calls the webhook **unconditionally**. +`deploy/production_preflight.py` is never invoked — only its unit test runs. + +- Either implement the gate or correct both documents. Do not leave the gap. + +### `[ ]` 2.5 — The preflight has silently decayed `(F9)` + +It blocks by string-matching source. **4 of 6 markers are dead** after the R2 +refactor: `'This runtime only supports APP_ENV=local'`, +`'Only local S3 storage is supported'`, `"allowed_hosts=['localhost', '127.0.0.1']"`, +`"'environment': 'local'"`. String gates weaken without failing. + +- Replace marker matching with behavioural assertions (import the module, assert + the adapter classes in use). + +### `[ ]` 2.6 — Base images are not pinned `(F10)` + +Dockerfiles default to mutable `python:3.12-slim` / `nginx:1.28-alpine`, the +workflow passes no digest build-args, and `--pull` makes builds non-reproducible — +while `PORTAINER.md` documents digest-pinned immutable bases. + +### `[ ]` 2.7 — pdf.js loaded from CDN without integrity `(F11)` + +3.11.174 from `cdnjs`, no SRI, and CSP allows the whole host for `script-src` **and** +`worker-src`. Vendor the asset or pin `integrity` and narrow the CSP to the exact path. + +### `[ ]` 2.8 — Single shared operator credential `(F12)` + +One `OPERATOR_EMAIL`/`OPERATOR_PASSWORD` for the whole factory; `movements.operator` +records the same name for everyone. The meeting asked for traceability, and the old +`kanban/main.py` explicitly designed separation of duties (Mayana classifies, +Thales/Alexandre authorise). Needs real per-person accounts with roles. + +### `[ ]` 2.9 — No TLS in the stack `(F13)` + +Ports publish plain HTTP on 18080/18081 while `COOKIE_SECURE: "true"` — cookies are +silently dropped unless something external terminates TLS. Nothing in the repo +provisions certificates; `TAREFAS.md` A2 still lists it as pending. + +### `[ ]` 2.10 — No email verification, no password recovery `(F14)` + +A locked-out customer has no path back, and registration accepts any CNPJ without +proving control of the e-mail. Needs a transactional mail provider — **client input**. + +### `[ ]` 2.11 — LGPD `(F15)` + +CNPJ, phone and e-mail are kept indefinitely in `accounts.profile` and +`orders.snapshot`. Artwork has a 30-day policy; personal data has none, and there is +no privacy notice, consent record or deletion path. + +--- + +## Block 3 · Architecture — needs a decision before code + +### `[?]` 3.1 — Manual quote approval contradicts the 24h business case `(F16)` + +Payment requires `quotes.approved`, set only by an authenticated operator. The +meeting's premise was that the 17h30 order waiting until 5am is what costs the +money. As built, a 2am order still waits for a person. `CONTEXT.md` frames this as a +temporary development trust boundary — the risk is that it silently becomes the +delivered model. + +**Decide:** what makes a quote auto-approvable (mode, metre range, grade floor, +returning customer), and what still routes to a human. + +### `[?]` 3.2 — Billable metres are computed in the customer's browser `(F17, F18)` + +For loose artwork, `metros` comes from `desenhaMontagem`/`encaixar` — a canvas +alpha-mask packer running client-side. The server never recomputes it. +`passoDe()`/`CELULAS_MAX` coarsen the grid for large sheets and image decoding +differs by browser, so **the same cart can price differently on different devices**. +The code comments reference "o motor do servidor"; that engine does not exist. + +Worse, the layout the customer is quoted on is never produced — operators upload +final files by hand, so billed metres ≠ printed metres and a designer redoes work +the site already did. + +**Decide:** port the packer to the server as the pricing authority and the print-file +generator, with the browser as preview only. This is the single largest gap between +what was promised in the meeting and what exists. + +### `[?]` 3.3 — The 5 GB problem is unsolved `(F19)` + +Transport accepts 5 GiB; `SCAN_MAX_BYTES` / ClamAV `StreamMaxLength` release only +≤ 128 MiB. Files above that are quarantined permanently with no path forward. This +is exactly the risk Jorge raised in the meeting. + +**Decide:** raise the scan ceiling with a resource/timeout design, or define an +explicit large-file path (staged scan, sampled scan, operator override with audit). + +### `[ ]` 3.4 — Upload throughput `(F20)` + +8 MiB parts, strictly sequential in `local/static/upload.js:21`, one presign +round-trip per part → ~640 sequential API calls for a 5 GB file, through an nginx +`limit_req` of 20r/s. Add parallelism (4–6 in flight) and batch presigning. + +### `[ ]` 3.5 — Payment ordering `(F27)` + +`dev_paid` charges before persisting the order and passes no idempotency key. +Harmless with `FakePayment`; with Mercado Pago that ordering is how you get double +charges. Fix as part of 1.1. + +--- + +## Block 4 · Scale and performance + +- `[ ]` 4.1 — Missing indexes `(F23)`. `local/schema.sql` indexes only + `uploads(owner)`. Add `orders(owner)`, `quotes(owner)`, `order_files(order_id)`, + `movements(order_id)`, and a partial index on + `outbox(available_at) WHERE delivered_at IS NULL` — the worker polls that table + every second and it only grows. +- `[ ]` 4.2 — `/api/operator/board` is unpaginated `(F24)`: every order ever, plus a + per-quote subquery each. Fine at 10 orders, not at 200/day. +- `[ ]` 4.3 — Scan throughput `(F21)`: one `scan_loop` thread, `worker` at + `replicas: 1`, ClamAV `MaxThreads 2`, browser gives up after 150s. +- `[ ]` 4.4 — Quality grade fallback `(F22)`: when `carregarImagem` fails, + `px(f)=Math.sqrt(f.size/1024)*95` stands — a DPI inferred from **file size in + bytes** — and it drives up to a 25% discount. Fail closed instead. +- `[ ]` 4.5 — Dead config `(F28)`: `stack.yaml` sets `CLAMD_HOST: scanner`, + `scanning.py` hardcodes `'scanner'`. + +--- + +## Block 5 · Hygiene and maintenance + +- `[ ]` 5.1 — CI coverage `(F33)`. The smoke, workflow, security, retention and + browser suites exist under `local/` and would have caught 0.1 — none run in CI. + Add a compose-backed job. **Highest leverage item in this block.** +- `[ ]` 5.2 — Remove or archive the dead prototypes `(F30)`: `portal/`, `kanban/`, + `agente/`, root `schema.sql` (~1,500 lines describing an abandoned model). Several + expose unauthenticated endpoints taking the acting user from the request body + (`/api/puxar`, `/api/devolver`). +- `[ ]` 5.3 — Root `requirements.txt` is the prototype's `(F31)`: wildcard pins, + unused `sqlmodel`/`pyvips`/`qrcode`/`pillow`, next to the hash-locked + `local/requirements.lock`. +- `[ ]` 5.4 — `pip==26.2.1` pinned as a runtime dependency `(F32)` — pip ships inside + the read-only production image. +- `[ ]` 5.5 — Doc drift `(F34)`. `README.md`, `CONTEXT.md`, `LOCAL_SETUP.md` and + `SECURITY_REPORT.md` describe a MinIO localhost stack, an API with "no external + network route", a `operator` / `local-operator-only` login the email-validated + model rejects, and a release gate — none match the current tree. +- `[ ]` 5.6 — `dtf-site.html` is 2,889 lines with commercial rules, the packing + engine, PDF analysis, UI and checkout inline `(F29)`. Split at least the pricing + table and the packer so 3.2 has somewhere to land. +- `[ ]` 5.7 — Commercial rules duplicated between `FAIXAS` (JS) and `TIERS` (Python) + `(F26)`. `test_pricing` guards parity; generate one from the other instead. +- `[ ]` 5.8 — The Site promises retention the system does not honour. The cart aside + still reads *"O arquivo fica guardado por 90 dias e o histórico do pedido por 12 + meses"*, while `CONTEXT.md` and the implemented retention are 30 days maximum. + This is a customer-facing commercial promise, so correct the copy or the policy — + do not leave them disagreeing. Found 2026-09-18 while closing Block 0. + +--- + +## Done + +### Week 1 — infrastructure, uploads, service base + +- `[x]` Compose stack: Site, Kanban, API, PostgreSQL, worker, ClamAV, R2/MinIO storage. +- `[x]` Gitea + Portainer publication path; web service startup and API rollout fixed. +- `[x]` Database passwords with special characters handled via discrete libpq fields. +- `[x]` Kanban e-mail/password login; missing operator config no longer breaks stack boot. +- `[x]` Direct resumable multipart browser → private object storage. +- `[x]` Quarantine + ClamAV gate: only `clean` files can be quoted, paid, downloaded or queued. +- `[x]` Retention worker: incomplete 1d, rejected 3d, originals 7d after approval, finals 30d. +- `[x]` Server-side pricing authority with parity test against the Site JavaScript (4,444 cases). +- `[x]` Loose-artwork packing flow: per-file width, copies, rotate, mirror, live 57 cm preview, + 5 mm gap, ruler and watermark preserved; metres follow packed height. +- `[x]` Rotation/mirror applied to packing masks; stale renders no longer overwrite a newer preview. +- `[x]` Hash-locked Python dependencies; Trivy reports in `output/security/`. + +### Block 0 — 2026-09-18 + +- `[x]` `import os` restored in `local/auth.py`; `COOKIE_SECURE` is now a single + module constant shared with `local/app.py`. +- `[x]` Navigation links no longer preselect a product (`data-modo-cta` removed). +- `[x]` Ready sheet vs loose artwork is an explicit, priced, reversible selector + (`#tipoEnvio`), shown in all four modes and locked once a file is attached. + The product is the declaration; `sel()` no longer switches anything silently. +- `[x]` `medirFolha` returns `dpiFolha`; an image that cannot span the film width + at `DPI_RECUSA` is refused as a sheet, with one click to send it as loose artwork. +- `[x]` `pintaCaminhos` scoped to `#caminhos .cam` — its global `.cam` selector was + clobbering the new control. +- `[x]` `compose.local.yaml` restored (MinIO, fake providers, builds from source). +- `[x]` `APP_DB_PASSWORD` separated from `POSTGRES_PASSWORD`, with a `bootstrap.py` + guard that refuses identical credentials in both configuration forms. +- `[x]` Checkout confirmation moved out of the panel the success path hides. + +### Reporting + +- `[x]` Week-1 client report (`Relatorio-Semana-1-DTF.docx`), corrected 2026-09-18 to + remove the inaccurate "Arquivo por metro permanece separado, com seleção explícita" + claim and the internal commit reference.