first commit

LOCAL_SETUP.md

@@ -0,0 +1,366 @@
+# DTF local development
+
+Read `CONTEXT.md` first. The current runtime lives in `local/`; older portal,
+Kanban, agent, requirements, and SQL files are historical prototypes.
+
+## Start
+
+Install Docker Engine/Desktop with Compose v2 or later. Docker must be running
+and your account must have permission to use it. The initial build downloads
+public container images and Python packages; running integrations are local.
+
+From this repository directory:
+
+```bash
+docker compose up --build
+```
+
+No `.env` is required: Compose has the same disposable defaults as `.env.example`.
+To customize, copy `.env.example` to `.env` and edit it. Never use real credentials.
+For a detached start with readiness verification:
+
+```bash
+docker compose up --build -d --wait
+docker compose ps
+```
+
+| Component | Local URL / port | Purpose |
+|---|---|---|
+| Site DTF | http://localhost:8080 | Existing Site with local checkout bridge |
+| Customer portal | http://localhost:8080/portal.html | Local account, order history, corrections and files |
+| Kanban | http://localhost:8081 | Quote review, orders, movement, original downloads |
+| API health | http://localhost:8000/health | Checks PostgreSQL and MinIO |
+| MinIO S3 | http://localhost:9000 | Direct signed multipart uploads and downloads |
+| MinIO console | http://localhost:9001 | Inspect private local storage |
+| PostgreSQL | `db:5432`, internal only | Shared persistent data |
+| ClamAV | `scanner:3310`, internal only | Isolated malware scanner with bundled signatures |
+| Worker | `worker:8002/health`, internal only | Mock outbox delivery, scanning and combined health |
+
+Use `localhost` consistently; mixing it with `127.0.0.1` creates a different
+browser session. Published ports bind only to `127.0.0.1`.
+
+Kanban default login: `operator` / `local-operator-only`.
+MinIO default login: `dtf_local` / `local-storage-only`.
+These are public, disposable development values, not real credentials.
+Interactive API documentation is disabled; `/docs` and `/redoc` are not local
+operator interfaces.
+
+## Browser test order
+
+1. Open the Site. Choose **Arquivo por metro** and the manual/table-price path
+   (CDR/AI/PSD/TIFF). Select `local/fixtures/local-test.cdr`. This is deliberately
+   harmless text for upload testing, not a printable CDR file.
+2. Enter **1.01 metres**. The original calculation bills **1.10 m × R$19.90 =
+   R$21.89**, with grade 0 and pickup. You can also use your own non-sensitive
+   artwork with the original product controls, repetitions, and mixed cart.
+3. Use test CNPJ `11.222.333/0001-81`, phone `11999999999`, email
+   `local-test@example.test`, and pickup. For simulated freight, select delivery,
+   enter an eight-digit CEP, and click quote. The default mock price is R$15.00.
+4. Click the Site checkout button. Files upload directly to MinIO, remain
+   quarantined until ClamAV returns `clean`, and then become eligible for a quote.
+   The local banner displays a quote ID awaiting commercial review.
+5. Open Kanban and log in. In **Cotações**, download the original if needed,
+   confirm/correct total metres and grade, tick the manual confirmation, and
+   click **Aprovar cotação**. For the fixture keep 1.01 m and grade 0.
+6. Return to the Site, click **Atualizar pedido local**, inspect the authoritative
+   server total, then click **Criar pedido pago local**. No real payment occurs.
+7. Refresh Kanban. The paid order starts in **Arte recebida**. Move it using
+   the buttons or drag and drop to **Arte tratada**. Open **Arquivos de produção**,
+   select the manually prepared final files for every item, enter a review note,
+   tick the confirmation and click **Aprovar arquivos finais**. Use the harmless
+   fixture again only for this local test; no printable file is generated.
+   Continue through **Fila de impressão →
+   Imprimindo → Finalizado**. A move to **Correção** requires a reason; it can
+   return to **Arte recebida** or **Arte tratada**. Finalizado is terminal locally.
+8. Inspect **Histórico** and **Eventos locais de integração**. Worker receipts
+   should say recorded locally. Download links expire after five minutes;
+   request another link from Kanban when needed.
+
+The manual quote step is necessary to enforce the backend trust boundary while
+automatic pre-flight is deferred. Browser measurements and grade are proposals.
+Only an authenticated operator can approve them; the server derives prices and
+freight. Payment accepts a quote ID only. Approved quotes are immutable for
+24 hours, and repeated/concurrent payment requests return the same order.
+This local review step does not settle the future automated production checkout.
+Malware scanning is a separate safety gate and does not inspect dimensions,
+resolution, colors, printability, or any other pre-flight property.
+
+The existing client-side previews/analysis remain unchanged and advisory. No new
+automatic print pre-flight or artwork validation runs on upload; malware scanning
+is the separate gate described above. Original downloads and manually approved
+final files are separately labeled. Never print the local text fixture.
+
+## Customer accounts, corrections, and cart recovery
+
+Open **Minha conta** on the Site, or `/portal.html`. Create a local account with
+a test CNPJ, phone, email and password of at least 12 characters. Passwords use
+the current stronger scrypt format; legacy local hashes are verified and upgraded
+on successful login. Sessions are stored in PostgreSQL, expire after
+seven days, and are revoked on logout. Login/registration attempts are limited.
+No email service is used: email verification and password recovery remain absent.
+
+Registration associates only the current guest session's uploads, quotes and
+orders with the new account. Signing in from another browser restores access to
+that account's orders. Matching an email or CNPJ never grants access to an order.
+Guests can still order and track within their current session.
+
+In the portal, **Ver detalhes e arquivos** displays status history, correction
+reasons, files and expiry dates. When Kanban requests **Correção**, upload corrected
+files against the relevant item and add a note. The order stays in correction
+until the operator reviews it. The old final set is invalidated; the operator
+must upload/approve another complete set before re-entering the queue. Concurrent
+or repeated submissions with a stale order version are rejected.
+
+The browser saves unfinished cart items and their File blobs in IndexedDB for
+24 hours. Reloading restores them as cart rows; remove/replace a row to alter its
+artwork settings, or add more products. Existing calculations are preserved and
+delivery must be requoted. Payment clears the saved cart. Cart recovery is local
+to the browser, not a cross-device artwork library; very large files can exceed
+browser storage capacity, which is reported visibly. Expired browser entries are
+removed the next time the Site opens. Server retention does not erase downloaded
+files or copies stored by the browser/operating system.
+Logout also clears local checkout keys and IndexedDB File blobs across open Site
+tabs. Operator authentication uses an expiring, revocable HttpOnly session;
+browser storage never retains the operator password or a Basic-auth credential.
+
+## Automated checks
+
+Pricing parity requires Python 3.10+ and Node 22+ on the host, no package install:
+
+```bash
+python3 -m unittest local.test_pricing -v
+```
+
+This executes the real pricing constants/functions extracted from `dtf-site.html`
+and compares all four modes, 101 grades, and 11 lengths (4,444 cases) to backend
+pricing, plus assembly and invalid-input checks.
+
+With the stack healthy, run the integration test (Python standard library only):
+
+```bash
+python3 -m local.smoke_test
+python3 -m local.workflow_test
+```
+
+It checks direct two-part upload/resume, missing parts, session isolation,
+private downloads and byte identity, customer validation, tampered totals,
+operator corrections, all four products, freight, immutable quotes, concurrent
+payment retries, state rules, history, and durable fake integration receipts.
+Each run leaves one clearly labeled test order and an approximately 8 MiB object.
+The workflow test also checks guest-to-account migration, cross-session login,
+revoked sessions, ownership isolation, correction submissions, final revision
+invalidation, secure downloads, and mandatory final-file approval before queueing.
+
+Security and malware regressions are separate so an authorized harmless EICAR
+test is unmistakable:
+
+```bash
+python3 -m local.security_test
+python3 -m local.scanning_test
+docker compose exec -T api python -m local.runtime_security_test
+docker compose exec -T api python -m local.retention_test
+```
+
+`local.scanning_test` stores an EICAR fixture as `SECURITY-EICAR.cdr`; ClamAV
+must reject it, and quote/download gates must remain closed. The rejected fixture
+is retained for at most three days, so it temporarily appears in alert summaries.
+
+For an automated real-browser walkthrough, install Chrome and use Node 22+:
+
+```bash
+node local/browser_test.mjs
+```
+
+Set `CHROME_BIN` if Chrome is not at `/usr/bin/google-chrome-stable`. The test
+uses a separate temporary browser profile, walks through the actual Site and
+Kanban controls, and saves screenshots to `output/local/`. It leaves its local
+test order for inspection. Both integration scripts read `.env` automatically.
+
+## Configuration and storage
+
+`.env.example` lists local ports, database/MinIO values, operator login, adapter
+selection, mock freight amount, maximum file size (5 GiB), and multipart size
+(8 MiB by default). The malware scanner releases only files up to 128 MiB by
+default (`SCAN_MAX_BYTES`); larger uploads remain blocked even though the
+multipart transport supports 5 GiB. `S3_ENDPOINT=http://storage:9000`, database
+hostname `db`,
+and the internal service ports are fixed Compose wiring. The public S3 endpoint
+must resolve from the browser; keep `http://localhost:9000` for this stack.
+Parts use 15-minute presigned URLs and uploads must finish within one day.
+
+Only fake integration adapters and `s3-local` storage are accepted. Startup fails
+if a production adapter/environment or nonlocal S3 endpoint is selected.
+The API, worker, and database run on an internal Docker network; web gateways
+and MinIO also join a network that permits loopback port publishing.
+
+Objects are private, use UUID keys rather than filenames, and persist in a named
+volume. MinIO lifecycle rules expire objects after 30 days and abandon incomplete
+multipart uploads after one day; the API also blocks expired downloads. Order
+history remains in PostgreSQL. Completed files start in `pending`; unknown,
+scanner-error, over-limit, encrypted/unsafe, and malware results fail closed.
+Only `clean` files can cross quote, payment, download, final-approval, and queue
+gates. Rejected/error objects expire within three days. The worker deletes
+original bytes within seven days of
+manual final artwork approval, and final/correction bytes by 30 days from the
+order's first upload. Expired files remain visible as metadata but cannot be
+downloaded. Retention runs once a minute; MinIO lifecycle is a backstop.
+
+Upload retries resume completed parts. The saved cart retains files when browser
+storage permits; otherwise reselect them. Saved quote IDs also survive reloads.
+Clearing cookies loses a guest session, while registered customers can sign in
+again. The operator can still inspect order records.
+
+## Local backup and restore check
+
+```bash
+python3 -m local.backup create-and-verify
+```
+
+This writes a private four-file bundle in `backups/` (ignored by Git and Docker
+builds): a PostgreSQL custom-format dump, a gzip object archive, a JSON manifest,
+and the manifest's SHA-256 sidecar. The object archive includes only complete,
+unexpired files that ClamAV has marked `clean`; pending, rejected, errored, expired,
+and purged objects remain excluded. Run the command while uploads and retention are
+idle so the database dump and subsequent object selection describe the same local
+state.
+
+Verification restores the dump into a newly created UUID-named database and the
+object bytes under a UUID-named `originals/restore-verification/` MinIO prefix. It
+checks database counts, bundle hashes, every archived object's hash, and the bytes
+downloaded after restore, then removes only the temporary database and objects. It
+never restores over active data. Keep every bundle file private: it contains
+customer data, password hashes, and customer artwork. To verify it again, run
+`python3 -m local.backup verify` followed by the printed
+`backups/...manifest.json` path. Legacy database-only `.dump` backups remain
+verifiable. Scheduling, offsite copies, and a production restore runbook remain
+unfinished.
+
+After building the current image, test retention with synthetic files:
+
+```bash
+docker compose exec -T api python -m local.retention_test
+```
+
+This checks that expired bytes are removed while unexpired files survive. It
+cleans up its own synthetic object bytes and retains their metadata.
+
+## Operations and troubleshooting
+
+```bash
+docker compose logs --tail=100 api worker scanner
+docker compose restart api worker
+docker compose ps
+docker compose down
+```
+
+`down` stops the stack and preserves named database/storage volumes. Restart
+with `docker compose up -d --wait`. Do not add `--volumes` unless you intend to
+permanently erase all local orders and artwork. No reset is required for tests.
+
+Seven long-running services have Docker health checks; the database and storage
+initialization jobs must exit successfully. API health queries PostgreSQL and
+MinIO; worker health fails if its database-processing loop, scanning thread, or
+live ClamAV PING is unavailable. Both web gateways expose `/health`. MinIO
+exposes `/minio/health/ready`.
+
+Run the redacted local alert summary inside the API network namespace:
+
+```bash
+docker compose exec -T api python -m local.security_status
+```
+
+Exit status 1 means attention is required. Review blocked artwork, rate limits,
+failed logins, scanner availability, and signature age. Immediately after the
+security tests, expected synthetic login failures/rate limits and
+`SECURITY-EICAR.cdr` rejections will trigger the alert; confirm names and test
+timing before classifying them as expected. Unexpected failures, scanner errors,
+stale signatures (more than seven days), or non-test rejected files require
+investigation. Events are retained for 30 days. See `SECURITY_REPORT.md`.
+
+If a host port is occupied, change `SITE_PORT`, `KANBAN_PORT`, or `API_PORT` in
+`.env` and recreate the stack. MinIO currently reserves 9000/9001. If Docker says
+permission denied, fix Docker access or use your OS-approved Docker workflow.
+If image/package download fails, check development internet access. A warning
+about missing Buildx can fall back to Docker's classic builder; install your
+platform's Buildx plugin for the supported modern build path.
+
+If API is unhealthy, inspect its logs and check local adapter settings. Database
+and MinIO credentials initialize persistent volumes only once; changing values
+later also requires updating the existing database/storage account deliberately.
+If an upload fails, keep the session and click checkout again. If parts expired,
+reselect the file to start another upload. For a stale Kanban move, refresh;
+the backend rejects stale versions instead of overwriting another operator.
+
+The Site retains its existing remote visual assets (fonts/logo) and optional
+PDF.js 3.11.174 CDN dependency. Its known eval-based advisory is mitigated in
+the current call with `isEvalSupported:false`; the old CDN dependency is still
+an upgrade/vendor-pinning limitation, not a claim that it is current. The local
+API, manual fixture flow, storage, payment,
+freight, Kanban, and mock integrations work without these external services.
+Offline PDF previews may fall back to the prototype's manual/table-price path.
+
+## Dependency lock
+
+The API, worker, and database initializer install every Python dependency from
+`local/requirements.lock` with `--require-hashes`. `local/requirements.txt`
+remains the human-maintained direct dependency list. After deliberately changing
+a direct pin, regenerate the lock in the same Python 3.12 environment and rebuild:
+
+```bash
+./local/lock_dependencies.sh
+docker compose up --build -d --wait
+```
+
+The generator downloads public package metadata in a disposable container and
+does not modify the host Python environment. Review the resolved versions and run
+the exact-runtime audit plus regression suite before accepting an updated lock.
+
+## Staging readiness gate
+
+`compose.staging.yaml` is intentionally not a staging deployment. It runs only a
+network-disabled validator for non-secret decisions. After completing
+`PRODUCTION_INPUTS.md`, copy `staging/staging.env.example` to the ignored
+`staging/staging.env`, enter non-secret metadata, and run:
+
+```bash
+docker compose -f compose.staging.yaml run --rm readiness
+```
+
+A pass does not authorize deployment or prove provider access. The real staging
+composition and external secret injection still require approved provider
+contracts and owners. See `staging/README.md`.
+
+## Next production connection step
+
+Keep this stack local. Before connecting real services, confirm the production
+checkout trust workflow, complete `PRODUCTION_INPUTS.md`, and pass the isolated
+staging-readiness gate. Then implement the actual staging composition using the adapter contracts in
+`local/adapters.py`: start with private S3 staging storage, CORS, signed multipart
+contract tests and scoped credentials injected outside Git. Add provider sandbox
+adapters one at a time. Mercado Pago requires authenticated, signed, idempotent
+webhook handling before real payment is allowed; Tiny/Olist and WhatsApp need
+confirmed contracts and provider idempotency/delivery handling. Do not simply
+change the local fake flags or point this Compose file at production.
+
+## Production delivery package
+
+The repository now includes a separate Docker Swarm and Gitea Actions delivery
+package in `deploy/` and `.gitea/workflows/`. It is not used by this localhost
+Compose stack and does not alter its volumes. Production images run as
+unprivileged users and install the hash-locked Python runtime. Gitea publishes
+`latest` for the normal Portainer webhook plus the full commit SHA for rollback.
+The stack expects pre-provisioned external Swarm secrets and an external
+PostgreSQL volume. Only Site and Kanban ports are published for the existing TLS
+reverse proxy; API, database, worker, and scanner remain private.
+
+Run the source-only gate without credentials:
+
+```bash
+python3 deploy/production_preflight.py --source-only
+```
+
+It must remain blocked while `local/` supports only local fake adapters and does
+not load Docker secret `*_FILE` settings. Do not bypass or delete this check.
+After approved production implementations and inputs exist, follow
+`PORTAINER.md` and `deploy/PRODUCTION_CHECKLIST.md`; release and deployment
+still require the Gitea scan/test gates and explicit approvals.