dtf-system/ROADMAP.md

DTF System — Working Roadmap

Internal engineering tracker. Not a client document, not a promise sheet. The client-facing narrative lives in output/pdf/dtf-plano-producao-e-roadmap.pdf and in the weekly report (Relatorio-Semana-1-DTF.docx).

Update the Current step line and the item status every time something moves. Add new findings at the bottom of the relevant block rather than rewriting history.

Current step: Block 0 closed, plus 2.1, 2.2, 2.3, 2.5 and 5.1. Next: 2.4 (the release gate the docs describe but the workflow never ran — partly addressed by 5.1), then 2.6/2.7. Block 1 still waits on client inputs for 1.1/1.2.

Last audit: 2026-09-18, full read of local/, dtf-site.html, deploy/, .gitea/, docs and legacy prototypes. Findings below carry their audit IDs.

Status	Meaning
[ ]	not started
[~]	in progress
[x]	done and verified
[?]	blocked on a decision (product or client), not on code

---

Block 0 · Broken right now

Nothing in this block is optional. Until it is closed, the system cannot be demonstrated, and the week-1 claims cannot be defended.

[x] 0.1 — API returns 500 on every session, login and registration (F1)

local/auth.py:46 calls os.environ and the module never imports os. Reproduced: NameError: name 'os' is not defined. /api/session calls new_session() whenever there is no cookie, so the Site checkout bridge, cart recovery and the customer portal all fail on first visit. Introduced in e3e37f6.

• Add import os to local/auth.py.
• Move the COOKIE_SECURE read to a module constant so it is evaluated once.
• Accept: a fresh browser hits the Site and /api/session returns 200 with a cart_scope; local/smoke_test.py and local/workflow_test.py pass.

[x] 0.2 — "Arquivo por metro" is unreachable in practice

Two independent causes, both from 2026-09-18 commits. Verified in a browser.

a. Every entry point hard-routes to loose artwork (483a083)

Control	Currently opens
Nav "Impressão DTF"	avulsa
"DTF Têxtil · 57 cm"	avulsa
"DTF UV · 28,5 cm"	uv
Hero "Enviar minha arte"	avulsa

Only the price card reaches file. Revert the three data-modo-cta attributes so navigation links land on the product chooser, not on a product.

b. Dropping a PNG/JPG in file mode silently switches the order (F25)

dtf-site.html sel() — from abrir('file'), dropping imagem-teste.jpg gives modo: "avulsa", header "Artes avulsas", price R$ 29,90/m. Meanwhile the same screen says "Arraste suas folhas montadas · PNG, JPG ou PDF", marks "PNG, JPG ou PDF · a partir de R$ 14,90" as the recommended path, and sets input.accept=".png,.jpg,.jpeg,.pdf". The page invites the drop and then reprices the order 50% higher.

The escape hatch #imagemComoFolha sits above the drop zone as small text inside an informational notice, defaults to unchecked, and must be ticked before the drop. After the switch fires, pintaModo() sets trocarParaAvulsa.hidden = true, so the checkbox disappears and there is no way back in place.

Net effect: the only formats that survive file mode are PDF/TIFF/PSD/AI/CDR — the path the UI itself marks as the worse option. A mixed drop (JPG + PDF) is rejected and accepts nothing.

• Make the ready-sheet choice an explicit two-option control inside the drop area, styled like the existing .cam selector — not a checkbox in a notice.
• Stop advertising PNG/JPG in the by-metre drop zone while rejecting them.
• When a switch does happen, show the price change and offer one-click undo.
• Accept: a customer can complete a by-metre order with a PNG from any entry point, and no product/price change ever happens without a visible confirmation.

[x] 0.3 — Ready-sheet declaration is unverified and worth money (F22 related)

Ticking #imagemComoFolha is an honour-system claim that moves the price from R$ 29,90/m to R$ 19,90/m (R$ 14,90 with a good grade). medirFolha then derives sheet height purely from aspect ratio × 57 cm — the original bug, now opt-in.

Measured with imagem-teste.jpg (466 × 659 px):

Route	System behaviour	Billed
Ticked	treated as a 57 × 80,6 cm mounted sheet	1 m × R$ 19,90 = R$ 19,90
Not ticked	20 cm wide, packs to 28,3 cm of film	1 m × R$ 29,90 = R$ 29,90

• Validate the claim: declared width must be ≈ film width (57 / 28,5 cm) at a plausible DPI before the sheet model is accepted.
• Re-check server-side in /api/operator/quotes/{id}/approve before pricing.
• Accept: a small single artwork declared as a ready sheet is rejected with a clear message; a genuine 57 cm sheet passes; the operator sees the verdict.

[x] 0.4 — Documented local startup fails (F2)

LOCAL_SETUP.md says docker compose up --build with no .env. docker compose --env-file .env.example config exits 1: R2_ENDPOINT, R2_ACCESS_KEY_ID, SITE_DOMAIN, KANBAN_DOMAIN missing. docker-compose.yml became a production/R2 stack in e3e37f6; .env.example is still the MinIO one and there is no MinIO service left.

• Decide: keep one production compose and add compose.local.yaml with MinIO, or restore a local default. Recommend the former.
• Accept: a clean clone reaches a working Site + Kanban with the documented command, and LOCAL_SETUP.md matches what actually runs.

[x] 0.5 — App DB role shares the admin password (F3)

docker-compose.yml:65 sets APP_DB_PASSWORD: ${POSTGRES_PASSWORD} — the same value as dtf_admin. bootstrap.py grants the app role DML-only and then hands it a credential that also logs in as the owner. Anyone reading the API container env has admin on the database.

• Accept: distinct secrets; connecting as dtf_app with the admin password fails.

[x] 0.6 — A paid order showed the customer nothing (found while closing Block 0)

#checkoutStatus and #checkoutActions lived inside #carr, and the success path in checkout.js clears the cart (pedido=[]; limpaPaineis()) before writing the confirmation — .carr{display:none} then hid the panel holding it. Present since the first commit; it only surfaced once 0.1 made a payment reachable at all. Both elements now sit in their own always-visible .checkout container.

How Block 0 was verified

A live stack (compose.local.yaml), then the full suite:

Check	Result
/api/session on a cold browser	200 with cart_scope + session cookie
smoke · workflow · security · scanning	pass
retention · runtime security (in-container)	pass
artwork_browser_test.mjs	pass, updated to the new declared-product behaviour
browser_test.mjs end-to-end	pass — upload → quote → operator approval → paid order → all Kanban states
4 CI unit tests	pass

Ports 8090/8091/8010 were used; 8080 was held by an unrelated preview server.

---

Block 1 · Week 2 — committed to the client

From the report already sent. These are dated promises, not backlog.

• [ ] 1.1 — Mercado Pago transparent checkout, signed and idempotent webhooks. Requires production credentials + webhook access. Payment must never be created before the freight amount is final.
• [ ] 1.2 — Real freight quotation. Blocked on client inputs (see PRODUCTION_INPUTS.md): source platform, credentials, origin CEP, services, packaging weight/dimensions per length, subsidy policy.
• [ ] 1.3 — Idempotent Tiny/Olist order creation with order-number traceability. Confirm endpoints, tag behaviour and rate limits first.
• [ ] 1.4 — Final print-file generation (see 3.2 — this is the same problem).
• [ ] 1.5 — Main Kanban production states consolidated.
• [ ] 1.6 — Block 0.2 + 0.3, promised as "início da próxima semana".

[!] The production compose currently blocks dev_paid (ENVIRONMENT != 'local') and ships only fake adapters, so the deployed system cannot take an order at all. 1.1 is what unblocks it.

---

Block 2 · Security — before any public exposure

[x] 2.1 — Rate limiting and audit logs are blind to the client (F5)

uvicorn runs without trusted proxy headers (forwarded_allow_ips defaults to 127.0.0.1; nginx is a different container IP), so request.client.host is nginx for every request. rate_limit('auth-source', ...) at 60/15min becomes a single global bucket — 60 failed logins lock out every customer — and every audit() record has no attacker IP.

• Set --proxy-headers with FORWARDED_ALLOW_IPS scoped to the nginx service, or read X-Forwarded-For explicitly at the edge.
• Accept: two clients on different IPs have independent buckets; audit rows carry the real IP.

[x] 2.2 — The public site throttles itself (F6)

local/app.py:115 — rate_limit('guest-sessions', ENVIRONMENT, 120, 900) is keyed on the environment name: 120 new visitors per 15 minutes site-wide (~8/min). Normal traffic 429s. Key per source IP (after 2.1) and raise the ceiling.

[x] 2.3 — deploy/stack.yaml cannot boot (F7)

It passes DATABASE_URL_FILE, AWS_ACCESS_KEY_ID_FILE, OPERATOR_PASSWORD_FILE, OPERATOR_USER. The code reads DATABASE_URL, AWS_ACCESS_KEY_ID, OPERATOR_PASSWORD, OPERATOR_EMAIL, and no _FILE loader exists. app.py:58 does os.environ['OPERATOR_PASSWORD'] → KeyError → 500 instead of 503.

• Implement local/secrets.py (the preflight already expects it) reading *_FILE with env fallback. Reconcile OPERATOR_USER vs OPERATOR_EMAIL.
• Accept: the stack renders and boots against Swarm secrets; missing operator config yields 503, not 500.

[ ] 2.4 — The documented release gate does not exist (F8)

PORTAINER.md and SECURITY_REPORT.md claim the workflow runs the full isolated suite, Trivy HIGH/CRITICAL image gates, secret scanning and the source preflight before calling Portainer. .gitea/workflows/deploy.yml runs py_compile plus four unit tests, then builds, pushes latest and calls the webhook unconditionally. deploy/production_preflight.py is never invoked — only its unit test runs.

• Either implement the gate or correct both documents. Do not leave the gap.

[x] 2.5 — The preflight has silently decayed (F9)

It blocks by string-matching source. 4 of 6 markers are dead after the R2 refactor: 'This runtime only supports APP_ENV=local', 'Only local S3 storage is supported', "allowed_hosts=['localhost', '127.0.0.1']", "'environment': 'local'". String gates weaken without failing.

• Replace marker matching with behavioural assertions (import the module, assert the adapter classes in use).

[ ] 2.6 — Base images are not pinned (F10)

Dockerfiles default to mutable python:3.12-slim / nginx:1.28-alpine, the workflow passes no digest build-args, and --pull makes builds non-reproducible — while PORTAINER.md documents digest-pinned immutable bases.

[ ] 2.7 — pdf.js loaded from CDN without integrity (F11)

3.11.174 from cdnjs, no SRI, and CSP allows the whole host for script-src and worker-src. Vendor the asset or pin integrity and narrow the CSP to the exact path.

[ ] 2.8 — Single shared operator credential (F12)

One OPERATOR_EMAIL/OPERATOR_PASSWORD for the whole factory; movements.operator records the same name for everyone. The meeting asked for traceability, and the old kanban/main.py explicitly designed separation of duties (Mayana classifies, Thales/Alexandre authorise). Needs real per-person accounts with roles.

[ ] 2.9 — No TLS in the stack (F13)

Ports publish plain HTTP on 18080/18081 while COOKIE_SECURE: "true" — cookies are silently dropped unless something external terminates TLS. Nothing in the repo provisions certificates; TAREFAS.md A2 still lists it as pending.

[ ] 2.10 — No email verification, no password recovery (F14)

A locked-out customer has no path back, and registration accepts any CNPJ without proving control of the e-mail. Needs a transactional mail provider — client input.

[ ] 2.11 — LGPD (F15)

CNPJ, phone and e-mail are kept indefinitely in accounts.profile and orders.snapshot. Artwork has a 30-day policy; personal data has none, and there is no privacy notice, consent record or deletion path.

---

Block 3 · Architecture — needs a decision before code

[?] 3.1 — Manual quote approval contradicts the 24h business case (F16)

Payment requires quotes.approved, set only by an authenticated operator. The meeting's premise was that the 17h30 order waiting until 5am is what costs the money. As built, a 2am order still waits for a person. CONTEXT.md frames this as a temporary development trust boundary — the risk is that it silently becomes the delivered model.

Decide: what makes a quote auto-approvable (mode, metre range, grade floor, returning customer), and what still routes to a human.

[?] 3.2 — Billable metres are computed in the customer's browser (F17, F18)

For loose artwork, metros comes from desenhaMontagem/encaixar — a canvas alpha-mask packer running client-side. The server never recomputes it. passoDe()/CELULAS_MAX coarsen the grid for large sheets and image decoding differs by browser, so the same cart can price differently on different devices. The code comments reference "o motor do servidor"; that engine does not exist.

Worse, the layout the customer is quoted on is never produced — operators upload final files by hand, so billed metres ≠ printed metres and a designer redoes work the site already did.

Decide: port the packer to the server as the pricing authority and the print-file generator, with the browser as preview only. This is the single largest gap between what was promised in the meeting and what exists.

[?] 3.3 — The 5 GB problem is unsolved (F19)

Transport accepts 5 GiB; SCAN_MAX_BYTES / ClamAV StreamMaxLength release only ≤ 128 MiB. Files above that are quarantined permanently with no path forward. This is exactly the risk Jorge raised in the meeting.

Decide: raise the scan ceiling with a resource/timeout design, or define an explicit large-file path (staged scan, sampled scan, operator override with audit).

[ ] 3.4 — Upload throughput (F20)

8 MiB parts, strictly sequential in local/static/upload.js:21, one presign round-trip per part → ~640 sequential API calls for a 5 GB file, through an nginx limit_req of 20r/s. Add parallelism (4–6 in flight) and batch presigning.

[ ] 3.5 — Payment ordering (F27)

dev_paid charges before persisting the order and passes no idempotency key. Harmless with FakePayment; with Mercado Pago that ordering is how you get double charges. Fix as part of 1.1.

---

Block 4 · Scale and performance

• [ ] 4.1 — Missing indexes (F23). local/schema.sql indexes only uploads(owner). Add orders(owner), quotes(owner), order_files(order_id), movements(order_id), and a partial index on outbox(available_at) WHERE delivered_at IS NULL — the worker polls that table every second and it only grows.
• [ ] 4.2 — /api/operator/board is unpaginated (F24): every order ever, plus a per-quote subquery each. Fine at 10 orders, not at 200/day.
• [ ] 4.3 — Scan throughput (F21): one scan_loop thread, worker at replicas: 1, ClamAV MaxThreads 2, browser gives up after 150s.
• [ ] 4.4 — Quality grade fallback (F22): when carregarImagem fails, px(f)=Math.sqrt(f.size/1024)*95 stands — a DPI inferred from file size in bytes — and it drives up to a 25% discount. Fail closed instead.
• [ ] 4.5 — Dead config (F28): stack.yaml sets CLAMD_HOST: scanner, scanning.py hardcodes 'scanner'.

---

Block 5 · Hygiene and maintenance

• [x] 5.1 — CI coverage (F33). An integration job now builds the localhost stack and runs smoke, workflow, security, scanning, retention, runtime security and both browser suites; publish-and-deploy depends on it. Verified by reintroducing the 0.1 defect: py_compile and the unit tests still passed while smoke_test failed on /session, blocking the release. Browser tests skip with a warning when the runner has no Chrome — install google-chrome-stable on the runner (or set CHROME_BIN) to make them gate as well.
• [ ] 5.2 — Remove or archive the dead prototypes (F30): portal/, kanban/, agente/, root schema.sql (~1,500 lines describing an abandoned model). Several expose unauthenticated endpoints taking the acting user from the request body (/api/puxar, /api/devolver).
• [ ] 5.3 — Root requirements.txt is the prototype's (F31): wildcard pins, unused sqlmodel/pyvips/qrcode/pillow, next to the hash-locked local/requirements.lock.
• [ ] 5.4 — pip==26.2.1 pinned as a runtime dependency (F32) — pip ships inside the read-only production image.
• [ ] 5.5 — Doc drift (F34). README.md, CONTEXT.md, LOCAL_SETUP.md and SECURITY_REPORT.md describe a MinIO localhost stack, an API with "no external network route", a operator / local-operator-only login the email-validated model rejects, and a release gate — none match the current tree.
• [ ] 5.6 — dtf-site.html is 2,889 lines with commercial rules, the packing engine, PDF analysis, UI and checkout inline (F29). Split at least the pricing table and the packer so 3.2 has somewhere to land.
• [ ] 5.7 — Commercial rules duplicated between FAIXAS (JS) and TIERS (Python) (F26). test_pricing guards parity; generate one from the other instead.
• [ ] 5.8 — The Site promises retention the system does not honour. The cart aside still reads "O arquivo fica guardado por 90 dias e o histórico do pedido por 12 meses", while CONTEXT.md and the implemented retention are 30 days maximum. This is a customer-facing commercial promise, so correct the copy or the policy — do not leave them disagreeing. Found 2026-09-18 while closing Block 0.

---

Done

Week 1 — infrastructure, uploads, service base

• [x] Compose stack: Site, Kanban, API, PostgreSQL, worker, ClamAV, R2/MinIO storage.
• [x] Gitea + Portainer publication path; web service startup and API rollout fixed.
• [x] Database passwords with special characters handled via discrete libpq fields.
• [x] Kanban e-mail/password login; missing operator config no longer breaks stack boot.
• [x] Direct resumable multipart browser → private object storage.
• [x] Quarantine + ClamAV gate: only clean files can be quoted, paid, downloaded or queued.
• [x] Retention worker: incomplete 1d, rejected 3d, originals 7d after approval, finals 30d.
• [x] Server-side pricing authority with parity test against the Site JavaScript (4,444 cases).
• [x] Loose-artwork packing flow: per-file width, copies, rotate, mirror, live 57 cm preview, 5 mm gap, ruler and watermark preserved; metres follow packed height.
• [x] Rotation/mirror applied to packing masks; stale renders no longer overwrite a newer preview.
• [x] Hash-locked Python dependencies; Trivy reports in output/security/.

Block 0 — 2026-09-18

• [x] import os restored in local/auth.py; COOKIE_SECURE is now a single module constant shared with local/app.py.
• [x] Navigation links no longer preselect a product (data-modo-cta removed).
• [x] Ready sheet vs loose artwork is an explicit, priced, reversible selector (#tipoEnvio), shown in all four modes and locked once a file is attached. The product is the declaration; sel() no longer switches anything silently.
• [x] medirFolha returns dpiFolha; an image that cannot span the film width at DPI_RECUSA is refused as a sheet, with one click to send it as loose artwork.
• [x] pintaCaminhos scoped to #caminhos .cam — its global .cam selector was clobbering the new control.
• [x] compose.local.yaml restored (MinIO, fake providers, builds from source).
• [x] APP_DB_PASSWORD separated from POSTGRES_PASSWORD, with a bootstrap.py guard that refuses identical credentials in both configuration forms.
• [x] Checkout confirmation moved out of the panel the success path hides.

Block 2 and CI — 2026-09-18

• [x] The web gateway overwrites X-Forwarded-For with the peer address instead of appending to it, and client_ip() resolves the requester for rate-limit buckets and security events. Verified: a forged 203.0.113.99 never reaches the audit trail.
• [x] Guest sessions are limited per source. The first attempt used 30/IP, which the new regression caught as too tight for shared NAT — recreating the original fault in a narrower form — so the ceiling is 240 per 15 minutes, overridable with GUEST_SESSION_LIMIT.
• [x] Security events now carry the source address (operator_login_failed, customer_login_failed, cross_origin_rejected, http_security_event).
• [x] CI runs the integration suites against a real stack before publishing.

2026-09-21

• [x] 2.3 — local/secrets.py resolves every <NAME>_FILE into <NAME> from the API, worker and bootstrap entrypoints, failing closed on an unreadable or empty secret and on a value supplied both ways. Verified by booting the API, the worker and bootstrap with credentials supplied only as mounted files, including a password containing :/?#[]&=+$ ,%. OPERATOR_USER in the stack became OPERATOR_EMAIL, which is what the runtime reads.
• [x] 2.5 — The gate now loads local/secrets.py and makes it resolve every secret deploy/stack.yaml declares, plus asserts it fails closed. Verified against a no-op loader (11 blockers) and one that swallows a missing file (1 blocker); only the real implementation passes. The four marker strings that stopped matching when R2 support landed were removed; the two describing real blockers stay, so the gate still refuses a release while the payment and messaging adapters are fake.

Reporting

• [x] Week-1 client report (Relatorio-Semana-1-DTF.docx), corrected 2026-09-18 to remove the inaccurate "Arquivo por metro permanece separado, com seleção explícita" claim and the internal commit reference.