dtf-system/docs/LOCAL_SETUP.md

# DTF local development

Read `CONTEXT.md` first. The service lives in `app/`; older portal,
Kanban, agent, requirements, and SQL files are historical prototypes.

## Start

Install Docker Engine/Desktop with Compose v2 or later. Docker must be running
and your account must have permission to use it. The initial build downloads
public container images and Python packages; running integrations are local.

From this repository directory:

```bash
docker compose -f compose.local.yaml up --build
```

`docker-compose.yml` is the production/R2 stack and will not start locally; the
localhost stack is always `compose.local.yaml`.

No `.env` is required: Compose has the same disposable defaults as `.env.example`.
To customize, copy `.env.example` to `.env` and edit it. Never use real credentials.
For a detached start with readiness verification:

```bash
docker compose -f compose.local.yaml up --build -d --wait
docker compose -f compose.local.yaml ps
```

| Component | Local URL / port | Purpose |
|---|---|---|
| Site DTF | http://localhost:8080 | Existing Site with local checkout bridge |
| Customer portal | http://localhost:8080/portal.html | Local account, order history, corrections and files |
| Kanban | http://localhost:8081 | Quote review, orders, movement, original downloads |
| API health | http://localhost:8000/health | Checks PostgreSQL and MinIO |
| MinIO S3 | http://localhost:9000 | Direct signed multipart uploads and downloads |
| MinIO console | http://localhost:9001 | Inspect private local storage |
| PostgreSQL | `db:5432`, internal only | Shared persistent data |
| ClamAV | `scanner:3310`, internal only | Isolated malware scanner with bundled signatures |
| Worker | `worker:8002/health`, internal only | Mock outbox delivery, scanning and combined health |

Use `localhost` consistently; mixing it with `127.0.0.1` creates a different
browser session. Published ports bind only to `127.0.0.1`.

Kanban default login: `operator@example.test` / `local-operator-only`.
MinIO default login: `dtf_local` / `local-storage-only`.
These are public, disposable development values, not real credentials.
Interactive API documentation is disabled; `/docs` and `/redoc` are not local
operator interfaces.

## Browser test order

1. Open the Site. Choose **Arquivo por metro** and the manual/table-price path
   (CDR/AI/PSD/TIFF). Select `tests/fixtures/local-test.cdr`. This is deliberately
   harmless text for upload testing, not a printable CDR file.
2. Enter **1.01 metres**. The original calculation bills **1.10 m × R$19.90 =
   R$21.89**, with grade 0 and pickup. You can also use your own non-sensitive
   artwork with the original product controls, repetitions, and mixed cart.
3. Use test CNPJ `11.222.333/0001-81`, phone `11999999999`, email
   `local-test@example.test`, and pickup. For simulated freight, select delivery,
   enter an eight-digit CEP, and click quote. The default mock price is R$15.00.
4. Click the Site checkout button. Files upload directly to MinIO, remain
   quarantined until ClamAV returns `clean`, and then become eligible for a quote.
   The local banner displays a quote ID awaiting commercial review.
5. Open Kanban and log in. In **Cotações**, download the original if needed,
   confirm/correct total metres and grade, tick the manual confirmation, and
   click **Aprovar cotação**. For the fixture keep 1.01 m and grade 0.
6. Return to the Site and click its checkout action again. Inspect the
   authoritative server total, then click **Criar pedido de teste**. No real
   payment occurs.
7. Refresh Kanban. The paid order starts in **Arte recebida**. Move it using
   the buttons or drag and drop to **Arte tratada**. Open **Arquivos de produção**,
   select the manually prepared final files for every item, enter a review note,
   tick the confirmation and click **Aprovar arquivos finais**. Use the harmless
   fixture again only for this local test. The text fixture is not an image, so
   its print file shows **preparar à mão**; with a PNG or JPEG artwork the
   generated PDF is preselected instead (see "Print files").
   Continue through **Fila de impressão →
   Imprimindo → Finalizado**. A move to **Correção** requires a reason; it can
   return to **Arte recebida** or **Arte tratada**. Finalizado is terminal locally.
8. Inspect **Histórico** and **Eventos locais de integração**. Worker receipts
   should say recorded locally. Download links expire after five minutes;
   request another link from Kanban when needed.

The manual quote step is necessary to enforce the backend trust boundary while
automatic pre-flight is deferred. Browser measurements and grade are proposals.
Only an authenticated operator can approve them; the server derives prices and
freight. Payment accepts a quote ID only. Approved quotes are immutable for
24 hours, and repeated/concurrent payment requests return the same order.
This local review step does not settle the future automated production checkout.
Malware scanning is a separate safety gate and does not inspect dimensions,
resolution, colors, printability, or any other pre-flight property.

The existing client-side previews/analysis remain unchanged and advisory. No new
automatic print pre-flight or artwork validation runs on upload; malware scanning
is the separate gate described above. Original downloads and manually approved
final files are separately labeled. Never print the local text fixture.

## Customer accounts, corrections, and cart recovery

Open **Minha conta** on the Site, or `/portal.html`. Create a local account with
a test CNPJ, phone, email and password of at least 12 characters. Passwords use
the current stronger scrypt format; legacy local hashes are verified and upgraded
on successful login. Sessions are stored in PostgreSQL, expire after
seven days, and are revoked on logout. Login/registration attempts are limited.
No email service is used: email verification and password recovery remain absent.

Registration associates only the current guest session's uploads, quotes and
orders with the new account. Signing in from another browser restores access to
that account's orders. Matching an email or CNPJ never grants access to an order.
Guests can still order and track within their current session.

In the portal, **Ver detalhes e arquivos** displays status history, correction
reasons, files and expiry dates. When Kanban requests **Correção**, upload corrected
files against the relevant item and add a note. The order stays in correction
until the operator reviews it. The old final set is invalidated; the operator
must upload/approve another complete set before re-entering the queue. Concurrent
or repeated submissions with a stale order version are rejected.

The browser saves unfinished cart items and their File blobs in IndexedDB for
24 hours. Reloading restores them as cart rows; remove/replace a row to alter its
artwork settings, or add more products. Existing calculations are preserved and
delivery must be requoted. Payment clears the saved cart. Cart recovery is local
to the browser, not a cross-device artwork library; very large files can exceed
browser storage capacity, which is reported visibly. Expired browser entries are
removed the next time the Site opens. Server retention does not erase downloaded
files or copies stored by the browser/operating system.
Logout also clears local checkout keys and IndexedDB File blobs across open Site
tabs. Operator authentication uses an expiring, revocable HttpOnly session;
browser storage never retains the operator password or a Basic-auth credential.

## Automated checks

Pricing parity requires Python 3.10+ and Node 22+ on the host, no package install:

```bash
python3 -m unittest local.test_pricing -v
```

This executes the real pricing constants/functions extracted from `web/site-config.js`
and compares all four modes, 101 grades, and 11 lengths (4,444 cases) to backend
pricing, plus assembly and invalid-input checks.

With the stack healthy, run the integration test (Python standard library only):

```bash
python3 -m tests.smoke_test
python3 -m tests.workflow_test
```

It checks direct two-part upload/resume, missing parts, session isolation,
private downloads and byte identity, customer validation, tampered totals,
operator corrections, all four products, freight, immutable quotes, concurrent
payment retries, state rules, history, and durable fake integration receipts.
Each run leaves one clearly labeled test order and an approximately 8 MiB object.
The workflow test also checks guest-to-account migration, cross-session login,
revoked sessions, ownership isolation, correction submissions, final revision
invalidation, secure downloads, and mandatory final-file approval before queueing.

Security and malware regressions are separate so an authorized harmless EICAR
test is unmistakable:

```bash
python3 -m tests.security_test
python3 -m tests.scanning_test
docker compose -f compose.local.yaml exec -T api python3 -m tests.runtime_security_test
docker compose -f compose.local.yaml exec -T api python3 -m tests.retention_test
```

`local.scanning_test` stores an EICAR fixture as `SECURITY-EICAR.cdr`; ClamAV
must reject it, and quote/download gates must remain closed. The rejected fixture
is retained for at most three days, so it temporarily appears in alert summaries.

For an automated real-browser walkthrough, install Chrome and use Node 22+:

```bash
node tests/browser_test.mjs
```

Set `CHROME_BIN` if Chrome is not at `/usr/bin/google-chrome-stable`. The test
uses a separate temporary browser profile, walks through the actual Site and
Kanban controls, and saves screenshots to `output/local/`. It leaves its local
test order for inspection. Both integration scripts read `.env` automatically.

## Configuration and storage

`.env.example` lists local ports, database/MinIO values, operator login, adapter
selection, mock freight amount, transport ceiling (5 GiB), and multipart size
(8 MiB by default). The API and customer picker admit only files within the
malware scanner's effective 128 MiB limit by default (`SCAN_MAX_BYTES`); the
larger-file path remains a Week 2 decision. `S3_ENDPOINT=http://storage:9000`, database
hostname `db`,
and the internal service ports are fixed Compose wiring. The public S3 endpoint
must resolve from the browser; keep `http://localhost:9000` for this stack.
Parts use 15-minute presigned URLs and unfinished reservations expire after
one hour. Clients can cancel a reservation through the upload DELETE endpoint.

Fake integration adapters and `s3-local` storage are the default. Mercado Pago
and Tiny can be selected for sandbox testing only (see "Provider sandboxes");
startup fails if their credentials are missing, if any other adapter is
selected, or if a production environment or nonlocal S3 endpoint is used.
The API, worker, and database run on an internal Docker network; web gateways
and MinIO also join a network that permits loopback port publishing.

Objects are private, use UUID keys rather than filenames, and persist in a named
volume. MinIO lifecycle rules expire objects after 30 days and abandon incomplete
multipart uploads after one day as a backstop; the API lease and worker release
unfinished reservations after one hour. The API also blocks expired downloads. Order
history remains in PostgreSQL. Completed files start in `pending`; unknown,
scanner-error, over-limit, encrypted/unsafe, and malware results fail closed.
Only `clean` files can cross quote, payment, download, final-approval, and queue
gates. Rejected/error objects expire within three days. The worker deletes
original bytes within seven days of
manual final artwork approval, and final/correction bytes by 30 days from the
order's first upload. Expired files remain visible as metadata but cannot be
downloaded. Retention runs once a minute; MinIO lifecycle is a backstop.

Upload retries resume completed parts. The saved cart retains files when browser
storage permits; otherwise reselect them. Saved quote IDs also survive reloads.
Clearing cookies loses a guest session, while registered customers can sign in
again. The operator can still inspect order records.

## Print files

Every paid order queues one print-file job per item. The worker renders a PDF
exactly as wide as the film and as long as the approved layout, placing each
copy at the position, rotation and mirror the customer reviewed. Each source
image is embedded once at its original resolution (JPEG bytes unchanged, PNG
transparency kept as a soft mask), so nothing is resampled. The Kanban card
shows the status per item, the page size and the lowest DPI, and offers
**Baixar PDF**. In **Arquivos de produção** the generated file is preselected
as the final file; untick it to upload one by hand instead.

Only JPEG, PNG, WebP and TIFF are generated. PDF, PSD, AI and CDR artwork, a
file whose proportions do not match the quoted size, a layout longer than was
billed, or an image above `PRINT_MAX_PIXELS` (250 Mpx by default) goes to
**preparar à mão** with the reason, and the operator prepares it as before.
**Gerar arquivos de impressão** retries those items and generates files for
orders paid before the generator existed. After a customer correction the
generated file is no longer offered: it reproduces the replaced artwork.

Layouts longer than about 5 m use the PDF `UserUnit` page scale instead of
being split into pages. Confirm on the factory's FlexiPRINT that such a file
imports at full length before relying on it for long orders.

```bash
docker compose -f compose.local.yaml exec -T api python -m unittest tests.test_printfile -v
docker compose -f compose.local.yaml exec -T api python -m tests.print_file_test
```

With PyMuPDF installed locally (`pip install pymupdf`), the unit suite also
draws each page and checks where every quadrant of the artwork lands.

## Provider sandboxes

`app/mercadopago.py` and `app/tiny.py` follow the providers' public API
documentation and pass their unit suites against a fake transport. They are not
verified integrations until they pass with the client's accounts.

The default local stack gives the API and worker no route to the internet, so
provider testing adds `compose.providers.yaml`, which does:

```bash
docker compose -f compose.local.yaml -f compose.providers.yaml up -d --wait
```

Put only **test** credentials in `.env`:

```bash
PAYMENT_ADAPTER=mercadopago
MP_ACCESS_TOKEN=TEST-...
MP_WEBHOOK_SECRET=...            # "Assinatura secreta" in the webhook settings
MP_NOTIFICATION_URL=https://<public tunnel>/api/payments/webhook
```

### Tiny (API v3)

Tiny uses OAuth2. In the client's Tiny (Construa plan or above, with the
"Gestão de Aplicativos" extension): **Configurações → Geral → Aplicativos →
+ novo aplicativo**, with the redirect URL set to this system's callback. That
gives a client ID and secret:

```bash
TINY_CLIENT_ID=...
TINY_CLIENT_SECRET=...
TINY_REDIRECT_URI=http://localhost:8081/api/operator/tiny/callback   # exactly as registered in Tiny
TINY_PRODUCT_TEXTIL_FOLHA=...            # Tiny product id for each Site product
TINY_PRODUCT_TEXTIL_AVULSA=...
TINY_PRODUCT_UV_FOLHA=...
TINY_PRODUCT_UV_AVULSA=...
```

With the client ID and secret set, the Kanban header shows **Conectar Tiny**.
Someone with a Tiny login approves access once; the tokens are stored in the
database (the refresh token rotates on every use) and the worker keeps the
connection alive. Only then set `TINY_ADAPTER=tiny`, which starts creating an
order in Tiny for every paid order: find or create the customer's contact by
CNPJ, then `POST /pedidos` with `numeroOrdemCompra = DTF-<order number>`. A
retry searches the customer's recent orders for that number first, so it does
not create a second one. **Tiny has no sandbox**: every test order is real, so
agree the test with the client and cancel the test orders afterwards.

`tests.tiny_oauth_test` checks the connection flow against the real database
with a fake token server; it saves and restores any existing connection.

With Mercado Pago selected, an approved quote shows **Pagar com PIX** on the
Site instead of the local test button. The order is created only by the signed
notification, after the payment is fetched from the Mercado Pago API and its
BRL amount matches the approved total. Mercado Pago must be able to reach the
webhook, so a local run needs a public HTTPS tunnel to the Site port. A paid
notification that cannot become an order, or a refund on an existing order,
appears under **Pagamentos que precisam de atenção** on the Kanban until an
operator records the resolution. Card payment needs the Mercado Pago public key
and its card form on the Site; that part is not built yet.

## Local backup and restore check

```bash
python3 -m ops.backup create-and-verify
```

This writes a private four-file bundle in `backups/` (ignored by Git and Docker
builds): a PostgreSQL custom-format dump, a gzip object archive, a JSON manifest,
and the manifest's SHA-256 sidecar. The object archive includes only complete,
unexpired files that ClamAV has marked `clean`; pending, rejected, errored, expired,
and purged objects remain excluded. Run the command while uploads and retention are
idle so the database dump and subsequent object selection describe the same local
state.

Verification restores the dump into a newly created UUID-named database and the
object bytes under a UUID-named `originals/restore-verification/` MinIO prefix. It
checks database counts, bundle hashes, every archived object's hash, and the bytes
downloaded after restore, then removes only the temporary database and objects. It
never restores over active data. Keep every bundle file private: it contains
customer data, password hashes, and customer artwork. To verify it again, run
`python3 -m ops.backup verify` followed by the printed
`backups/...manifest.json` path. Legacy database-only `.dump` backups remain
verifiable. Scheduling, offsite copies, and a production restore runbook remain
unfinished.

After building the current image, test retention with synthetic files:

```bash
docker compose -f compose.local.yaml exec -T api python3 -m tests.retention_test
```

This checks that expired bytes are removed while unexpired files survive. It
cleans up its own synthetic object bytes and retains their metadata.

## Operations and troubleshooting

```bash
docker compose -f compose.local.yaml logs --tail=100 api worker scanner
docker compose -f compose.local.yaml restart api worker
docker compose -f compose.local.yaml ps
docker compose -f compose.local.yaml down
```

`down` stops the stack and preserves named database/storage volumes. Restart
with `docker compose -f compose.local.yaml up -d --wait`. Do not add `--volumes` unless you intend to
permanently erase all local orders and artwork. No reset is required for tests.

Seven long-running services have Docker health checks; the database and storage
initialization jobs must exit successfully. API health queries PostgreSQL and
MinIO; worker health fails if its database-processing loop, scanning thread, or
live ClamAV PING is unavailable. Both web gateways expose `/health`. MinIO
exposes `/minio/health/ready`.

Run the redacted local alert summary inside the API network namespace:

```bash
docker compose -f compose.local.yaml exec -T api python3 -m ops.security_status
```

Exit status 1 means attention is required. Review blocked artwork, rate limits,
failed logins, scanner availability, and signature age. Immediately after the
security tests, expected synthetic login failures/rate limits and
`SECURITY-EICAR.cdr` rejections will trigger the alert; confirm names and test
timing before classifying them as expected. Unexpected failures, scanner errors,
stale signatures (more than seven days), or non-test rejected files require
investigation. Events are retained for 30 days. See `SECURITY_REPORT.md`.

If a host port is occupied, change `SITE_PORT`, `KANBAN_PORT`, or `API_PORT` in
`.env` and recreate the stack. MinIO currently reserves 9000/9001. If Docker says
permission denied, fix Docker access or use your OS-approved Docker workflow.
If image/package download fails, check development internet access. A warning
about missing Buildx can fall back to Docker's classic builder; install your
platform's Buildx plugin for the supported modern build path.

If API is unhealthy, inspect its logs and check local adapter settings. Database
and MinIO credentials initialize persistent volumes only once; changing values
later also requires updating the existing database/storage account deliberately.
If an upload fails, keep the session and click checkout again. If parts expired,
reselect the file to start another upload. For a stale Kanban move, refresh;
the backend rejects stale versions instead of overwriting another operator.

The Site retains its existing remote visual assets (fonts/logo) and optional
PDF.js 3.11.174 CDN dependency. Its known eval-based advisory is mitigated in
the current call with `isEvalSupported:false`; the old CDN dependency is still
an upgrade/vendor-pinning limitation, not a claim that it is current. The local
API, manual fixture flow, storage, payment,
freight, Kanban, and mock integrations work without these external services.
Offline PDF previews may fall back to the prototype's manual/table-price path.

## Dependency lock

The API, worker, and database initializer install every Python dependency from
`infra/requirements.lock` with `--require-hashes`. `infra/requirements.txt`
remains the human-maintained direct dependency list. After deliberately changing
a direct pin, regenerate the lock in the same Python 3.12 environment and rebuild:

```bash
./infra/lock_dependencies.sh
docker compose -f compose.local.yaml up --build -d --wait
```

The generator downloads public package metadata in a disposable container and
does not modify the host Python environment. Review the resolved versions and run
the exact-runtime audit plus regression suite before accepting an updated lock.

## Staging readiness gate

`compose.staging.yaml` is intentionally not a staging deployment. It runs only a
network-disabled validator for non-secret decisions. After completing
`PRODUCTION_INPUTS.md`, copy `staging/staging.env.example` to the ignored
`staging/staging.env`, enter non-secret metadata, and run:

```bash
docker compose -f compose.staging.yaml run --rm readiness
```

A pass does not authorize deployment or prove provider access. The real staging
composition and external secret injection still require approved provider
contracts and owners. See `staging/README.md`.

## Next production connection step

Keep this stack local. Before connecting real services, confirm the production
checkout trust workflow, complete `PRODUCTION_INPUTS.md`, and pass the isolated
staging-readiness gate. Then implement the actual staging composition using the adapter contracts in
`app/adapters.py`: start with private S3 staging storage, CORS, signed multipart
contract tests and scoped credentials injected outside Git. Add provider sandbox
adapters one at a time. Mercado Pago requires authenticated, signed, idempotent
webhook handling before real payment is allowed; Tiny/Olist and WhatsApp need
confirmed contracts and provider idempotency/delivery handling. Do not simply
change the local fake flags or point this Compose file at production.

## Production delivery package

The repository now includes a separate Docker Swarm and Gitea Actions delivery
package in `deploy/` and `.gitea/workflows/`. It is not used by this localhost
Compose stack and does not alter its volumes. Production images run as
unprivileged users and install the hash-locked Python runtime. Gitea publishes
`latest` for the normal Portainer webhook plus the full commit SHA for rollback.
The stack expects pre-provisioned external Swarm secrets and an external
PostgreSQL volume. Only Site and Kanban ports are published for the existing TLS
reverse proxy; API, database, worker, and scanner remain private.

Run the source-only gate without credentials:

```bash
python3 deploy/production_preflight.py --source-only
```

It must remain blocked while `app/` supports only local fake adapters and does
not load Docker secret `*_FILE` settings. Do not bypass or delete this check.
After approved production implementations and inputs exist, follow
`PORTAINER.md` and `deploy/PRODUCTION_CHECKLIST.md`; release and deployment
still require the Gitea scan/test gates and explicit approvals.